May 2026 Summaries

DoorDash's experimentation platform, led by senior engineering manager Ilya Izrailevsky, is designed to cater to three distinct user groups: consumers, dashers, and merchants, all of whom have unique and sometimes competing needs. The platform processed 12,000 experiments last year, handling up to 300 million feature flag evaluations per second. With a background in leading experimentation platforms at companies like Amazon and Uber, Ilya focuses on scaling experimentation across democratization, global expansion, vertical growth, and AI-powered development. DoorDash's CEO Tony Xu actively engages with the results of every experiment, fostering a company-wide culture of data-driven decision-making. The platform's sophisticated use of AI helps in mining institutional knowledge, configuring tests, and automating result summaries, though human judgment remains crucial. DoorDash's approach emphasizes the importance of balancing interests and the value of learning from every experiment, even those that don't achieve their initial goals. As DoorDash expands into new verticals and geographies, the emphasis remains on ensuring each experiment delivers tangible customer value, with experimentation serving as a vehicle for broader business impact.

GrowthBook 4.4 introduces a comprehensive update that enhances its platform's capabilities in experimentation, product analytics, and feature flagging, tailored for both cloud and self-hosted users. The release features a reconstructed API with extensive coverage, enabling teams to integrate with GrowthBook using agentic tools or through its UI. It introduces a conversational AI Data Analyst for self-serve analytics, allowing non-data teams to explore product metrics without SQL, and supports automated workflows through its API, facilitating a programmable experiment lifecycle. The update also simplifies the use of multi-armed bandits for continuous learning in AI applications and includes new safety and governance features for feature flagging, such as automated stale flag cleanup and monitored ramp schedules for controlled rollouts. These enhancements aim to provide a seamless, efficient workflow that maintains rigor and safety while enabling faster product development and deployment.

AI-driven development has significantly accelerated the pace at which new features are shipped, necessitating robust control mechanisms to prevent incidents in production environments. GrowthBook's Feature Flags provide a solution by allowing teams to manage rapid feature rollouts with safety and consistency via automated ramp schedules, configurable approval workflows, and enhanced stale feature flag detection. The latest version, GrowthBook 4.4, introduces new capabilities such as release plans with automated ramp schedules, configurable approval workflows, and improvements in stale feature flag detection, enabling a standardized and scalable feature flag management process. By integrating guardrails and approval gates into the rollout process, GrowthBook ensures that rapid development cycles remain secure and manageable, while audit logs and revision histories maintain transparency and control. The platform supports both human and AI-driven changes, ensuring all actions go through the same safety checks and governance, thus providing a comprehensive approach to modern software development challenges.

Andrew Willingham, Head of Legal and People Products at Atlassian, shares his journey from leading product development at Amazon to reinventing HR systems in the AI era. His experience underscores the importance of understanding the actual users of a product, as he learned when a feature-rich talent review system initially flopped due to its complexity for frontline users. Transitioning from Amazon's high-volume A/B testing environment to Atlassian's more qualitative research approach, Willingham emphasizes balancing efficiency and quality metrics in HR processes. He advocates for a strategic framework where durable truths are shipped without testing, while counterintuitive elements are thoroughly tested, stressing that the most valuable insights often come from unexpected failures. His approach to experimentation focuses on learning quickly by challenging assumptions, engaging directly with users, and adapting workflows that have long been industry standards, all while maintaining a mindset of iterative, small-scale changes that accumulate over time.

Feature flags, also known as feature toggles, are a crucial tool in modern software development, allowing teams to control the visibility and activation of features for specific user segments without deploying new code. They enable decoupling of deployment from release, faster iteration cycles, continuous delivery, and a robust experimentation infrastructure. However, scaling feature flags from a few to thousands introduces challenges such as technical debt, ownership ambiguity, performance issues, user experience complexity, and fragmented tooling. To manage these challenges effectively, it is essential to establish scalable architectures, governance, and ownership systems, as well as integrate feature flags into CI/CD pipelines for seamless management. GrowthBook, a feature flagging platform, addresses these challenges by offering local SDK-based evaluations, caching, comprehensive governance tools, and built-in experimentation capabilities, making it possible for organizations like Dropbox to scale feature flagging practices efficiently.

Dave Massey, who joined UPS in 2016, spearheaded a transformative approach to experimentation, which has since contributed over $500 million in incremental revenue for the company. Initially, UPS lacked a comprehensive testing program, but Massey's background in digital advertising and conversion rate optimization allowed him to demonstrate the impact of UX improvements on revenue, starting with a $35 million boost from a simple checkout navigation adjustment. Despite initial skepticism, the success of this pilot led to the integration of rigorous data analysis and UX research within UPS, fostering a culture of testing. Massey's team, known as JEDI, now supports numerous customer-facing applications and is regarded as a center of excellence within the company. Future plans include enhancing personalization and decentralizing testing capabilities to allow other business units to conduct their own experiments, with AI playing an integral role in improving efficiency. The team's commitment to unbiased results and learning from failures has solidified its reputation for rigor and honesty, ultimately proving the business value of experimentation.

Feature flagging tools are crucial in modern software development, offering control over code deployment and experimentation without redeploying applications. When evaluating open-source feature flagging platforms, considerations include predictable costs, avoiding vendor lock-in, data sovereignty, and transparency, which are significant advantages over commercial SaaS options. While some open-source platforms focus on simple flag management, others, like GrowthBook, integrate comprehensive experimentation capabilities, allowing for detailed statistical analysis and A/B testing directly tied to feature flags. Platforms like GrowthBook offer robust SDK support across various environments, ensuring compatibility with diverse tech stacks, while also providing community support and flexible deployment models. The choice of a platform depends on specific needs, such as governance, compliance, real-time streaming, or integration with existing analytics tools, with each platform offering unique features and limitations tailored to different organizational requirements.

Over the past decade, Fanatics has transformed its experimentation approach from a small conversion rate optimization team conducting 10 tests a month to a robust program running nearly 100 monthly experiments across its extensive e-commerce platform. This shift, underpinned by a strong data-driven culture and executive support, has resulted in significant annual growth, with experimentation contributing roughly 8% annually. A key factor in this evolution is the commitment to learning and institutionalizing knowledge through a structured wiki, which not only stores test results but also generates new experiments and informs decision-making. Meta-analysis of experiments further converts individual test outcomes into broader institutional insights, while the rigorous approach to metrics ensures true causality is verified before declaring success. Fanatics also prioritizes risk avoidance, recognizing the value in identifying and preventing detrimental changes, thereby safeguarding growth. The program's success is attributed to leadership buy-in, infrastructure that sustains learning, and a culture of humility and inquiry that encourages continuous improvement.

Feature flags are a valuable tool for controlling feature releases and minimizing deployment risks, but they can introduce significant issues if not managed properly. As feature flags accumulate, they can create a complex system within your infrastructure, leading to potential failures that are often silent and unnoticed until considerable damage occurs. Common mistakes include reusing feature flags, using client-side flags for security, and lacking proper testing of all flag states. These errors can be categorized into implementation, operational, and strategic mistakes, each contributing to technical debt and operational inefficiencies. Effective governance, observability, and management practices are essential to prevent feature flags from becoming unmanageable, as demonstrated by high-profile incidents like the 2021 Facebook outage. Solutions such as GrowthBook can help enforce best practices, ensuring that feature flags enhance rather than hinder your system's performance.

In the context of experimentation and treatment effects, the text explores the complexities of interpreting different metrics that result from randomized trials, emphasizing the importance of distinguishing between various treatment effects, such as Average Treatment Effect (ATE), Conditional Average Treatment Effect (CATE), Intention-to-Treat effect (ITT), Local Average Treatment Effect (LATE), and Average Treatment Effect on the Treated (ATT). Using a food delivery platform's free trial experiment as an example, the text illustrates how different analytical frameworks and assumptions can lead to varying interpretations of the same data, highlighting the potential for selection bias when comparing subgroups without proper randomization. By employing concepts from causal inference, the text underscores the necessity of understanding which metric accurately answers the business question at hand, and why randomization is crucial for isolating the true effect of an intervention. It also explains how different treatment effects provide insights into customer behavior, with LATE focusing on those who comply due to the assignment, and emphasizes that the correct interpretation of these effects is vital for making informed business decisions.

Experiment velocity, often used as a key performance indicator (KPI) in experimentation programs, can lead to a focus on quantity over quality, resulting in more tests but less valuable insights. While running frequent experiments may initially indicate a mature program, the emphasis on velocity alone can lead to trivial tests that yield superficial insights, ultimately hindering the learning process. The text argues that the real goal should be increasing the rate of meaningful learning by maintaining a balanced portfolio of easy, medium, and hard experiments, which provide deeper insights into user behavior and product performance. This approach prevents teams from optimizing for speed at the expense of valuable learning, ensuring that experimentation drives product and organizational growth. By reframing the focus from sheer velocity to the quality and impact of experiments, organizations can foster a culture that prioritizes understanding and strategic improvement over mere activity.