Company
Date Published
Author
Luke Sonnet
Word count
1079
Language
English
Hacker News points
None

Summary

GrowthBook utilizes holdouts to accurately measure the cumulative impact of shipping new features by maintaining a small control group that is excluded from new functionalities and comparing it to the rest of the user base over time. This approach, termed "measure everything," captures the full scope of user experiences, including failed experiments and feature iterations, providing a realistic picture of what users encounter, unlike the "clean-room" method which only assesses the impact of successfully shipped features. By implementing holdouts at the project level, GrowthBook ensures that compliance is maintained without requiring individual reminders, with project-scoped holdouts inheriting the holdout by default for new features and experiments. The system encourages the use of long-range metrics instead of fixed conversion windows to account for the varied timing of user holdout exposure, and it emphasizes the importance of project-level organization for consistent and comparable assessment of impact. This comprehensive strategy allows for a clear understanding of the effects of all experimental variations, helping teams make more informed decisions about the direction of their programs.