Lessons learned from Ronny Kohavi and Luke Sonnet: running trustworthy experiments
Blog post from GrowthBook
GrowthBook's webinar, featuring experts Ronny Kohavi and Luke Sonnet, focused on the challenges of ensuring A/B test results are reliable, emphasizing that while generating data is straightforward, obtaining trustworthy data is complex. Drawing insights from Kohavi's extensive experience at companies like Amazon and Microsoft, and Sonnet's leadership at GrowthBook, the webinar highlighted common pitfalls in A/B testing such as underpowered tests, misinterpreted p-values, and sample ratio mismatches. The discussion stressed the importance of rigorous setup practices like power calculations, setting realistic Minimum Detectable Effects (MDEs), and choosing appropriate significance thresholds based on the cost of errors. Post-experiment checks such as verifying sample ratios and computing false positive risks were recommended to enhance result reliability. The session underscored that A/B tests, while a gold standard in experimentation, require careful planning and analysis to avoid misleading conclusions, and GrowthBook offers tools to facilitate these best practices.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.