July 2021 Summaries

Growth Book has released an updated version of its open-source A/B testing platform, introducing key features such as support for PrestoDB and TrinoDB as data sources, along with an enhanced user interface for displaying Bayesian risk metrics, which help assess the impact of choosing a variation under worst-case scenarios. The update also includes improvements in metric management by allowing users to tag and group metrics for easier organization and application in experiments, as well as the addition of guardrail metrics to monitor critical factors like page load time and error rates. Other improvements address logging, error messages, and bug fixes, including customization options for Snowflake roles and adjustments in SQL query handling. The team behind Growth Book encourages user feedback through multiple communication channels.

The text discusses the inherent tension between the rigorous statistical requirements of A/B testing and the fast-paced demands of modern product-driven companies. It highlights the challenges faced when companies must make decisions with less-than-perfect data due to time constraints and the need to iterate quickly, often resulting in decisions based on small sample sizes and noisy data. The text explains that while Frequentist approaches require predetermined sample sizes to avoid the peeking problem, Bayesian statistics allow for more flexibility by providing a measure of risk—known as potential loss—that can guide decision-making even when tests are incomplete. This risk measure, recently implemented by Growth Book with the help of Itamar Faran, allows teams to call tests earlier and be aware of the potential negative impact of their decisions, offering a way to balance speed and statistical accuracy by evaluating the chance to beat control, risk, and percent change confidence intervals.