March 2023 Summaries

GrowthBook, a warehouse-native feature flagging and experimentation platform, integrates with existing data infrastructure, eliminating the need to send data externally. It allows users to configure Metrics, Segments, and more via SQL, a task that can be daunting for many engineers, as indicated by Stack Overflow's 2022 Developer Survey, where 36% of respondents expressed discomfort with writing SQL. To streamline this process, GrowthBook introduced the Schema Browser, which provides a comprehensive view of available schemas, tables, and columns in connected data sources, reducing the need to switch between different platforms. With the launch of GrowthBook 2.0, new BigQuery and Postgres connections will automatically support the Schema Browser, while existing connections can easily generate an Information Schema View. The Information Schema is refreshed every 30 days by default, with an option for manual updates, and GrowthBook plans to support additional data source types soon.

CUPED, or Controlled Experiment Using Pre-Experiment Data, is a method for improving the speed and accuracy of experimentation by reducing variance through regression adjustment. Utilized within GrowthBook, CUPED allows experimenters to adjust post-exposure data using pre-experiment data, thereby minimizing uncertainty. This approach is particularly effective for metrics with high correlation between pre- and post-exposure data, such as engagement metrics, although it requires reliable historical data for optimal performance. GrowthBook offers customizable settings, including lookback and conversion windows, to tailor CUPED's application according to user behavior and experiment needs. While CUPED enhances experiment efficiency, it necessitates early data collection and does not address the "peeking" problem, which GrowthBook plans to tackle with upcoming sequential testing additions.

The announcement for the upcoming GrowthBook 2.0 release highlights several new features designed to enhance the experience for product managers and non-technical users. A major addition is the Visual Editor, which allows users to design, deploy, and analyze A/B tests directly in their browser without needing to code, marking a significant improvement from the previous Beta version. Additionally, a new Slack integration enables users to receive real-time alerts about important activities within GrowthBook, such as new feature creations or experiment terminations. To support effective experimentation, GrowthBook has also developed a comprehensive A/B Testing Best Practices Guide, which offers insights ranging from basic concepts to advanced strategies and is intended to be an evolving, open-source resource for users.

GrowthBook 2.0 introduces several enhancements focused on data and statistics to improve experimentation efficiency and user experience. The update incorporates CUPED for variance reduction, allowing users to speed up experimentation by effectively increasing the equivalent sample size, as evidenced by Microsoft achieving a 20% traffic boost for metrics. The new release also enhances the SQL editing experience with a schema browser, enabling users to directly access database schemas within GrowthBook, streamlining the construction of exposure and metric queries. SQL query execution speed has doubled thanks to internal optimizations and a new "Experiment Duration" attribution model, facilitating easier and more accurate analysis of experiment metrics. Additionally, GrowthBook 2.0 now supports native integration with Databricks, expanding its data source compatibility and simplifying the connection process for users.

GrowthBook 2.0 introduces several developer-centric enhancements designed to improve the user experience, including a fully typesafe feature flag SDK for Typescript that helps catch typos and bugs during compile time through an auto-generating GrowthBook CLI. The update also brings instant feature releases to GrowthBook Cloud, enabling rapid rollout of feature changes to all users in under a second, and expands the REST API with over 20 new endpoints, allowing for functionalities such as syncing experiment results to data warehouses. Additionally, improvements have been made to the example repository and documentation, with new examples for integrating GrowthBook, enhanced GoLang documentation, and support for Experiment CRUD events in webhooks. The update is set to cater further to data scientists and analysts in upcoming announcements.

GrowthBook has significantly improved the performance of its experiment analysis queries, achieving up to twice the speed by implementing three main changes. First, the system now handles situations where users are exposed to multiple variations or dimension values more efficiently, by excluding such users from the analysis and flagging experiments where this issue exceeds 1% of users. Second, a new attribution model called "Experiment Duration" has replaced the "Multiple Exposures" model, increasing the number of conversions included in analyses while enhancing performance. Lastly, the elimination of multiple JOINs and GROUP BYs from SQL queries, facilitated by a robust testing infrastructure, has further streamlined the process, reducing query length and complexity while maintaining accuracy. These updates represent only the beginning of GrowthBook's ongoing efforts to enhance performance.

GrowthBook 1.9 introduced a Proxy server to enable rapid feature rollouts for self-hosted instances, allowing changes in the GrowthBook UI to be propagated to users in under a second, a feature well-received by the self-hosted community. GrowthBook plans to extend this benefit to Pro and Enterprise accounts on GrowthBook Cloud, eliminating the need for users to manage their own Proxy servers. To ensure reliable, scalable, and instant feature flag deployment, GrowthBook Cloud utilizes a globally distributed CDN, short TTLs, and stale-while-revalidate headers, with Redis Pub/Sub and Server-Sent Events ensuring real-time responsiveness. This setup allows for features to be toggled and propagated quickly, maintaining uptime and speed. Currently, instant feature releases are supported for Javascript and React SDKs, with plans to expand support to other SDKs, inviting community contributions for further development.

In October, GrowthBook introduced its first REST API endpoint to list feature flags, marking the beginning of ongoing enhancements to their API offerings and documentation. The APIs are now fully documented using the OpenAPI 3.1 standard, with a new auto-generated documentation site that stays updated with each code change, enabling rapid iteration and the addition of new endpoints. The API facilitates new functionalities, such as syncing processed experiment results back to a data warehouse and automating feature management based on predefined guardrails, allowing users to programmatically disable features if certain metrics are not met. While most endpoints are currently read-only, GrowthBook plans to expand support for additional methods to enable more complex use cases.

GrowthBook has enhanced its client-side SDKs, originally written in 100% Typescript, by addressing previous limitations in type safety. While Typescript provided a solid foundation for type inference, it couldn't catch issues like typos in feature names or incorrect fallback value types. The latest update now allows for compile-time type checks, ensuring that errors such as a mismatched feature flag value type are caught early by defining all feature names and types when creating a GrowthBook instance. Additionally, GrowthBook introduced a command-line tool to automatically generate and maintain these feature types, simplifying the process and reducing human error. This tool can be integrated into development workflows to keep feature types up-to-date and is part of broader plans to expand type safety, including supporting strongly typed targeting attributes and integrating these capabilities across all SDK languages.