How Khan Academy Optimizes AI Tutoring with Experimentation

Khan Academy's experimentation journey, as detailed by Kelli Hill at Experimentation Island, highlights the organization's shift from intuition-based testing to sophisticated A/B testing on generative AI features, particularly with their AI tutor, Khanmigo. Initially relying on a homegrown platform for experiments, Khan Academy transitioned to self-hosting GrowthBook to maintain efficiency without compromising data integrity, allowing them to effectively manage a high volume of daily active users. The evolution of their AI evaluation methods from intuition-driven testing to structured manual testing and eventually to automated A/B testing underscores the challenges and complexities of measuring AI quality, especially with non-deterministic models like GPT-4. This structured approach facilitated a series of iterative improvements in AI functionalities, such as a math agent, ensuring enhancements in latency and accuracy. The broader cultural shift within Khan Academy, where experimentation transformed from a perceived hindrance to an essential safety net, further enabled teams to innovate confidently and efficiently, marking a significant cultural and operational transformation in their approach to AI development.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.