July 2026 Summaries

Kevin Yang's exploration of experimentation at JPMorgan Chase reveals a perspective that values the insights gained from failed experiments as much as, if not more than, the successes. While his team estimates that successful experiments have generated over a billion dollars, Yang emphasizes that the true value lies in the lessons learned from failures, which prevent potentially harmful changes from being implemented at scale. At Chase, where his team supports about 100 product teams running 300 experiments annually, the focus is on building a culture that appreciates the importance of control groups and pre-planned responses to failure, which help avoid confirmation bias and ensure informed decision-making. This approach is particularly crucial in the AI era, where rapid iteration and customization demand rigorous measurement to avoid compounding mistakes. Yang's insights underscore the importance of a balanced decision framework that values trust and customer satisfaction over raw engagement metrics, highlighting the necessity of planning for failure to foster innovation.

GrowthBook's Visual Editor is designed to address common shortcomings of traditional visual editors by offering an AI-first approach that allows users to make changes to a website for A/B testing without coding or relying on engineering teams. Unlike other editors that often require reverting to CSS or HTML for complex tasks, GrowthBook enables users to describe changes in natural language, which the AI then applies instantly on the page. It also incorporates image editing and generation, allowing users to create or modify images directly in the editor. The platform supports imports from Figma or static mockups, facilitating the seamless integration of designs into experiments. GrowthBook's editor is engineered to work with modern web platforms, using durable selectors to avoid issues with dynamically generated class names, ensuring variations persist through site updates. It includes features like a manual mode for precise control, global code editors for advanced customizations, and mechanisms to eliminate flicker, ensuring experiments run smoothly without bias. The interface supports multiple languages and offers a transparent change-tracking system, enhancing usability and control for diverse teams.

As software teams increasingly adopt AI-assisted code and face higher deployment frequencies, the complexity and risk of deployments have risen, prompting a shift towards using feature flags to mitigate these risks. Feature flags allow teams to decouple deployment from release, providing control over who sees what features and enabling quick rollbacks if issues arise, thereby reducing the risk associated with all-or-nothing deployments. Progressive rollouts and attribute-based targeting further minimize exposure to potential problems by gradually introducing features and selectively targeting users based on specific attributes. Tools like GrowthBook facilitate these processes by offering platforms that support feature flags, kill switches, and dark launches, allowing for more controlled and flexible deployment strategies. This approach not only helps manage the surface area for mistakes but also improves recovery times by enabling quicker responses to deployment failures, as evidenced by various case studies, including AWS and CrowdStrike. The emphasis on feature flags as a derisking mechanism reflects a broader industry trend towards more robust and resilient deployment pipelines in the face of increasing code volumes and complexity.

In a discussion on The Experimentation Edge podcast, Arun Bodapati, director of data science at Twitch, emphasizes the critical importance of preventing false negatives in experimentation, which can lead to potentially valuable ideas being disregarded and shelved for years. Unlike false positives, which typically get scrutinized and corrected, false negatives often go unnoticed and can have a lasting detrimental impact by institutionalizing incorrect conclusions. Bodapati advocates for rigorous preparation before running experiments, including clearly defining hypotheses in plain English, ensuring reliable enrollment triggers, and using broad "explore" experiments to avoid over-narrowing that could limit statistical power. He highlights the necessity of understanding the mechanisms behind positive results and warns against relying solely on numerical outcomes without a clear explanation of user behavior. A case study at Twitch regarding subscription pricing demonstrates how methodical experimentation and causal inference have shifted pricing strategy from being an untouchable area to a continuously adjustable lever, informed by reliable data and analysis.