May 2024 Summaries

GrowthBook 3.0 introduces significant advancements, including new Edge SDKs for Cloudflare, Fastly, and Lambda, which facilitate seamless A/B testing by modifying HTML before it reaches users, thereby eliminating flickering and other common issues. The update also enhances Bayesian statistics with custom priors and introduces CUPED support, improving accuracy and reducing testing time. A long-awaited feature, Custom Roles, offers flexible permission configurations, while integration with the OpenFeature ecosystem and new alerting capabilities for Slack and Discord expand functionality. The JSON Feature Flag Editor is revamped to offer a more user-friendly experience, and updates to Next.js examples demonstrate new rendering strategies. Additionally, the release includes a new React Native SDK and improvements across multiple existing SDKs.

Traditional A/B testing focuses on comparing the mean effects of different treatments, which may not capture the impact on outliers or specific distribution points. GrowthBook introduces quantile testing for Pro and Enterprise customers, offering a way to evaluate how features affect different data percentiles, such as the 99th percentile latency (P99) rather than just the average. Quantile testing involves creating Fact Tables with relevant data like session IDs, user IDs, timestamps, and latency, and setting up quantile metrics for analysis. This method allows for a more nuanced understanding of feature impacts, such as measuring reductions in the worst-case latency scenarios or analyzing revenue changes across various user spending tiers. For instance, a treatment that reduces P99 latency significantly may not only improve the worst latencies but also provide insights into which user subgroups benefit the most, as seen with quantile metrics showing improvements among moderately high and high spenders. Quantile testing thus provides a comprehensive view of data distributions and feature effects, complementing traditional mean-based metrics.

The upcoming GrowthBook 3.0 release introduces significant enhancements to its Bayesian engine, allowing users to specify their own priors, integrate CUPED for variance reduction, and improve estimation accuracy in small sample sizes. These changes simplify the process by focusing on treatment effects rather than variation averages, making it easier for users to set priors and analyze experiments. While some existing experiment results may experience minor shifts, these are generally minimal and aim to enhance the power of the analysis engine. The new model facilitates improved computation of inferential statistics by employing techniques like the Delta method for variance and CUPED for faster experimentation, especially in handling small sample sizes more effectively. Although the previous model was not inaccurate, the new model offers a more streamlined setup and increased analytical power, making it an advantageous update for users.