September 2025 Summaries

GrowthBook 4.1 builds on the previous version by introducing key Enterprise features such as Holdouts and Experiment Dashboards, while enhancing AI integration and MCP server capabilities. Holdout experiments enable long-term impact analysis by maintaining a control group, and Experiment Dashboards allow for customized experiment views to facilitate insights sharing. The release enhances AI-driven functionalities, including automated result summarization and SQL assistance, and updates the MCP Server for seamless experiment creation within AI coding tools. Additionally, GrowthBook is now integrated natively in the Vercel Marketplace, offering streamlined feature flagging and A/B testing for Vercel projects. The update also supports pre-computed dimensions for efficient queries, introduces support for the open-source FerretDB as a MongoDB alternative, and offers integration with Sanity CMS for content variation testing. With over 150 commits, this release emphasizes user feedback and sets the stage for future developments.

Successful product teams often rely on rapid iteration and frequent experimentation to enhance their offerings, making it crucial to accurately measure the cumulative impact of these changes to guide product direction. However, assessing this impact can be challenging due to factors like external influences and the interaction of multiple experiments. GrowthBook offers a solution through holdouts, which maintain a control group that is not exposed to new features, allowing teams to accurately measure long-run cumulative effects. This method provides a more reliable assessment compared to summing individual experiment impacts, which can overstate results due to biases and interactions. By using holdouts, companies like Facebook and Twitter have been able to better understand the true effect of their product strategies over extended periods, ensuring that their development efforts are genuinely contributing to their goals. Holdouts involve diverting a small percentage of traffic away from new features, allowing for the comparison between the holdout group and the general population to derive meaningful insights about cumulative impact. This methodology helps teams allocate resources effectively and make informed decisions about future product developments.

GrowthBook utilizes holdouts to accurately measure the cumulative impact of shipping new features by maintaining a small control group that is excluded from new functionalities and comparing it to the rest of the user base over time. This approach, termed "measure everything," captures the full scope of user experiences, including failed experiments and feature iterations, providing a realistic picture of what users encounter, unlike the "clean-room" method which only assesses the impact of successfully shipped features. By implementing holdouts at the project level, GrowthBook ensures that compliance is maintained without requiring individual reminders, with project-scoped holdouts inheriting the holdout by default for new features and experiments. The system encourages the use of long-range metrics instead of fixed conversion windows to account for the varied timing of user holdout exposure, and it emphasizes the importance of project-level organization for consistent and comparable assessment of impact. This comprehensive strategy allows for a clear understanding of the effects of all experimental variations, helping teams make more informed decisions about the direction of their programs.