Company
Date Published
Author
Graham McNicoll
Word count
702
Language
English
Hacker News points
None

Summary

The text discusses the inherent tension between the rigorous statistical requirements of A/B testing and the fast-paced demands of modern product-driven companies. It highlights the challenges faced when companies must make decisions with less-than-perfect data due to time constraints and the need to iterate quickly, often resulting in decisions based on small sample sizes and noisy data. The text explains that while Frequentist approaches require predetermined sample sizes to avoid the peeking problem, Bayesian statistics allow for more flexibility by providing a measure of risk—known as potential loss—that can guide decision-making even when tests are incomplete. This risk measure, recently implemented by Growth Book with the help of Itamar Faran, allows teams to call tests earlier and be aware of the potential negative impact of their decisions, offering a way to balance speed and statistical accuracy by evaluating the chance to beat control, risk, and percent change confidence intervals.