March 2026 Summaries

Khan Academy's experimentation journey, as detailed by Kelli Hill at Experimentation Island, highlights the organization's shift from intuition-based testing to sophisticated A/B testing on generative AI features, particularly with their AI tutor, Khanmigo. Initially relying on a homegrown platform for experiments, Khan Academy transitioned to self-hosting GrowthBook to maintain efficiency without compromising data integrity, allowing them to effectively manage a high volume of daily active users. The evolution of their AI evaluation methods from intuition-driven testing to structured manual testing and eventually to automated A/B testing underscores the challenges and complexities of measuring AI quality, especially with non-deterministic models like GPT-4. This structured approach facilitated a series of iterative improvements in AI functionalities, such as a math agent, ensuring enhancements in latency and accuracy. The broader cultural shift within Khan Academy, where experimentation transformed from a perceived hindrance to an essential safety net, further enabled teams to innovate confidently and efficiently, marking a significant cultural and operational transformation in their approach to AI development.