Company
Date Published
Author
Luke Sonnet
Word count
1028
Language
English
Hacker News points
None

Summary

Successful product teams often rely on rapid iteration and frequent experimentation to enhance their offerings, making it crucial to accurately measure the cumulative impact of these changes to guide product direction. However, assessing this impact can be challenging due to factors like external influences and the interaction of multiple experiments. GrowthBook offers a solution through holdouts, which maintain a control group that is not exposed to new features, allowing teams to accurately measure long-run cumulative effects. This method provides a more reliable assessment compared to summing individual experiment impacts, which can overstate results due to biases and interactions. By using holdouts, companies like Facebook and Twitter have been able to better understand the true effect of their product strategies over extended periods, ensuring that their development efforts are genuinely contributing to their goals. Holdouts involve diverting a small percentage of traffic away from new features, allowing for the comparison between the holdout group and the general population to derive meaningful insights about cumulative impact. This methodology helps teams allocate resources effectively and make informed decisions about future product developments.