Company
Date Published
Author
Tim Chan
Word count
1129
Language
English
Hacker News points
None

Summary

The blog post discusses a new methodology for calculating sample sizes in A/B testing, emphasizing transparency and education in the process. Unlike typical online calculators, this method can handle tests with imbalanced group proportions, such as 20/80 ratios, and allows users to provide their own standard deviation for non-Bernoulli metrics like time spent or average payment amount. It highlights the importance of determining the correct sample size to avoid Type I and Type II errors in experiments, ensuring that tests are properly powered. The post also explores the significance of setting a minimum detectable effect (MDE) and understanding the implications of different split ratios in test design. Additionally, the blog touches on cultural and infrastructural insights from industry experts, the evolution of web experience platforms, and personal experiences at Statsig, emphasizing the company's unique culture and the value of A/B testing in evidence gathering.