August 2026 Summaries

Randomized experiments can produce misleading subgroup, funnel, and variance-reduction estimates when analysts condition on variables measured after treatment assignment that the treatment may have changed, creating “bad controls” and post-treatment bias. In a simulated reading-app experiment with a true uniform gain of two articles per user, engagement tiers recalculated during the experiment made every subgroup appear to benefit less than the overall population because treatment pushed users into higher tiers, causing treated and control users within each tier to represent different underlying populations; tiers fixed before launch correctly recovered the effect. This differs from Simpson’s paradox because the overall estimate remains reliable while the post-treatment subgroup analysis introduces the distortion. Similar problems arise when evaluating funnel steps only among users who reached an earlier step, or using step-to-step conversion rates with treatment-affected denominators, since these condition on users selected partly by the intervention; outcomes should instead generally be measured per assigned or exposed user. Variance-reduction methods such as CUPED, regression adjustment, and post-stratification are safe only with pre-experiment covariates, as adjusting for during-experiment behavior can remove part of the treatment effect itself. The central safeguard is to split, filter, or adjust only using variables known before the experiment begins, while using balance and sample-ratio checks as supporting diagnostics rather than definitive proof of validity.

Dimension splits estimate conditional treatment effects across user segments and can reveal meaningful differences hidden by an experiment’s overall average treatment effect, but smaller samples and repeated testing make such analyses vulnerable to noise, cherry-picking, and bias. Pre-specifying segments and hypotheses before an experiment improves credibility, though multiple-testing corrections remain necessary; family-wise error controls the risk of any false positive, while false discovery rate is often better suited to independent rollout decisions across many segments. Post-hoc analysis should be treated as hypothesis generation rather than confirmation, with all examined cuts disclosed, promising signals ranked and interpreted alongside uncertainty, and important findings tested in a pre-specified replication. Segment definitions must rely only on pre-treatment variables, since splitting on attributes affected by the treatment creates biased comparisons. Although exploratory slicing can uncover commercially important effects, particularly among high-value users, decisions should balance its risks with transparent reporting, replication, and potentially more systematic methods such as causal forests.

GrowthBook has launched Learnings, a feature intended to turn accumulated experiment results, user research, and other evidence into a shared organizational knowledge base for people and AI agents. Rather than treating A/B tests as isolated decisions, Learnings captures broader patterns across multiple studies, such as which approaches work for particular user segments or product areas, while linking conclusions to supporting and conflicting evidence and allowing them to be scoped, updated, or marked uncertain. The feature addresses the difficulty of finding relevant insights among large experiment archives by providing a compressed layer of decision-relevant context, accessible through GrowthBook APIs, MCP support, and agent-oriented Skills. GrowthBook can also use AI to identify candidate patterns across experiment histories, though teams retain responsibility for reviewing them. The company emphasizes that clear hypotheses, context, results, and conclusions are essential because better experiment documentation makes both individual tests and the broader knowledge base more useful over time.

Ben Schein, Director of Product Management at Clover, explains how experimentation differs when a product is a critical workplace tool for restaurants rather than a consumer app that can quietly expose users to A/B test variants. Because Clover serves more than 300,000 U.S. merchants and supports essential functions such as payments, ordering, menus, and handheld point-of-sale systems, its teams rely on structured pilots, ground-level validation, and rollout planning instead of production splits that could disrupt employees during service. Schein says the level of testing should be determined by uncertainty and potential downside, with high-risk changes to payment authorization or order-entry flows requiring extensive validation while established features such as Apple Pay may need less experimentation. He also emphasizes defining the strategic value of a test in advance, using counter metrics alongside primary measures, and seeking durable business learning rather than simple pass-fail results. Drawing on his experience at Shake Shack, he describes improving checkout funnels through contextual experiences, including clear preparation-time expectations and brand messaging during loading screens, while arguing that future AI-enabled experimentation tools will make testing more accessible but will not replace human judgment about risk, customer context, and meaningful opportunities.

Edd Saunders, product experimentation manager at JobLeads, argues that effective experimentation depends on validating customer problems rather than pursuing seemingly logical solutions, citing a pizza-ordering personalization feature that reduced checkout steps but had no effect because customers valued browsing and control. He trains experimenters through customer journey mapping, quantitative funnel analysis, qualitative behavior data, and user research, then prioritizes problems using a matrix based on evidence and potential impact. Since JobLeads began testing seriously in early 2025, it has increased its experiment launch rate from roughly 0.3 to 2.8 per month by standardizing workflows, sharing roadmaps, and maintaining a company-wide database of experiment learnings. Saunders emphasizes that unsuccessful tests remain valuable because they improve future decisions, and he aims to democratize experimentation by enabling non-specialists to run tests while using AI for low-cost MVP validation, documentation, and workflow automation.

Edd Saunders, product experimentation manager at JobLeads, describes how a seemingly obvious personalization feature for a pizza company—allowing returning customers to reorder in one click—produced no measurable effect because customers valued browsing and control despite often choosing the same meal. He argues that experimentation should begin with validated customer problems rather than proposed solutions, using detailed journey and funnel analysis, behavioral data such as heatmaps and session recordings, and qualitative research to identify and prioritize issues by evidence and potential impact. At JobLeads, standardized workflows, shared roadmaps, and a company-wide knowledge base helped increase experiment launches from roughly 0.3 to 2.8 per month in about a year, while emphasizing learning from unsuccessful tests as much as successful ones. Saunders aims to broaden experimentation beyond specialists by enabling marketers and content managers to run tests, supported by visible feedback on their contributions and AI tools for low-cost MVP validation, documentation, and workflow automation.

Feature flags and remote configuration both modify application behavior without redeployment, but they serve distinct purposes: flags determine whether users experience a feature, typically through boolean, targeted rollout controls, while remote configuration defines persistent values such as rate limits, UI layouts, or JSON settings. The discussion uses major outages attributed to configuration changes, including Google Cloud, CrowdStrike, and Cloudflare, to argue that configuration requires safeguards comparable to release controls, including validation, staged rollout, auditability, and kill switches. Although modern platforms increasingly combine both capabilities through string, numeric, and JSON flag values with targeting rules, teams should distinguish them by intent and lifecycle: feature flags are generally temporary and should be removed after releases, whereas configuration is designed to be updated in place over time. The text also warns that stale flags, interacting toggles, undocumented long-term settings, and client-exposed secrets can increase operational risk, recommending ownership, expiry dates, versioning, cleanup processes, secure server-side evaluation, and schema validation. GrowthBook is presented as a platform supporting both uses through typed feature values, rules, audit logs, stale detection, JSON schemas, dedicated configuration objects, and reusable constants.

Reliable A/B test results depend largely on decisions made before data collection, with experimental validity encompassing internal validity, external generalizability, appropriate metric construction, and sound statistical conclusions. The framework recommends pre-registering hypotheses, metrics, sample sizes, stopping rules, guardrails, and decision criteria; matching randomization and analysis units; re-randomizing users for new tests; ensuring treatment and control differ only in the intended change; and using cluster, geographic, or time-based designs when users influence one another. Teams should select one primary metric linked to the intended mechanism and longer-term organizational goals, calculate sample size through power analysis, set a sufficient runtime and conversion window, and account for concurrent experiments that may interact. Pre-launch checks such as A/A tests, balance assessments, instrumentation verification, overlap reviews, and runtime projections can identify design problems early, while monitoring for novelty effects, network interference, and treatment interactions remains important during and after testing. GrowthBook is presented as a platform that supports these practices through required checklists, power calculations, statistical health checks, sequential testing, and embedded decision frameworks.

Arie Polycarpou, Senior Manager of Test and Learn at Aspen Dental, describes building experimentation programs across Kohl’s, Marriott, Total Wine, and Aspen Dental by establishing data connections, testing processes, and dedicated teams. At Aspen Dental, where roughly 1,100 independently owned offices share one website, experimentation must assess not only appointment bookings but also downstream patient value, office quality, show rates, and return value, since directing demand to different locations can produce materially different business outcomes. Polycarpou views an approximately 25% win rate as a sign of a mature program, arguing that early high win rates often reflect easy opportunities while more advanced programs test harder, less certain ideas. He also advocates conservative reporting of annualized test value through “haircuts” that account for fading novelty, overlapping changes, limited test duration, and statistical uncertainty rather than simply adding measured lifts together. Aspen Dental’s developing program aims to increase testing volume, personalize experiences by office and service line, and cautiously adopt AI tools to support analysis and test development while retaining engineering oversight and methodological rigor.

Erika Dunn, assistant director of data science at Principal Financial Group, describes building an experimentation center of excellence within a 150-year-old, risk-averse financial services company by emphasizing controlled testing, shared learning, and practical use of existing data. Drawing on experience in quantitative psychology, marketing analytics, and experimentation at Amazon, she distinguishes informal attempts from rigorous experiments through the use of control groups and careful planning, while advocating for documented test failures to challenge outdated organizational assumptions. Her team develops metrics such as “looping,” which identifies customers repeatedly returning to the same web pages as a signal of frustration, and analyzes content characteristics to improve marketing messages. Principal has also created in-house synthetic digital audiences based on normalized customer data to rank content and campaign ideas before exposing them to live traffic, while retaining traditional A/B tests as the standard for validating outcomes. An initial synthetic-audience-selected variation won a controlled test, with additional tests planned, and Dunn sees future AI agents helping teams prioritize experimentation opportunities amid growing volumes of data.

Feature flags separate deployment, the technical act of placing code in production, from release, the business decision to expose functionality to users, allowing teams to deploy continuously while controlling when, where, and to whom features become available. By dark-launching code behind disabled flags, organizations can test production behavior before exposure and then use instant, targeted, percentage-based, or monitored gradual rollouts according to risk. The approach enables rapid rollback through a flag toggle rather than redeployment, supports product and QA ownership of release decisions, and can use guardrail metrics such as error rates, latency, and conversion to pause or reverse unsafe rollouts automatically. Effective adoption requires governance through approval workflows, access controls, audit trails, and human review, even when AI tools help create release plans. It also requires managing the complete flag lifecycle, including assigning owners and removal dates, monitoring releases against defined rollback criteria, distinguishing temporary flags from permanent configuration, and removing stale flags after stable full rollout to prevent technical debt.

Vijay Lal, Lead Product Manager for Experimentation at US Bank, describes how regulated organizations can scale customer testing while maintaining security, reliability, and rigorous oversight. Facing demand that exceeded the capacity of a central experimentation team, US Bank adopted a self-service platform supported by simplified tools, continuous training, user accountability, and guardrail metrics. A redesign of the bank’s secure login widget illustrated this approach: after testing revealed that slower connections could prevent some customers from loading the embedded widget, the team introduced a fallback experience to ensure no users were excluded and experiment coverage remained complete. Lal also emphasizes that experiments should begin with a clear hypothesis, which then determines primary, secondary, and guardrail metrics, rather than selecting KPIs after a decision has already been made. The discussion argues that effective experimentation requires organizations to democratize access responsibly, account for edge cases and performance differences, and measure behavior with metrics closely connected to the feature being tested.

Farfetch built an in-house experimentation platform, Fabs 2.0, after finding that its hybrid approach of internal tools and an external vendor created performance problems, inconsistent results, and additional analysis work. The platform routes every experiment through a deeply integrated feature-toggling system connected to customer segmentation, tracking, content, recommendations, and messaging, allowing technical and nontechnical teams to conduct tests without code injection. Its experimentation center of excellence focuses on enabling teams through shared hypothesis templates, peer-review clinics, open learning sessions, coaching, and a knowledge base rather than centrally executing tests. Farfetch measures success through a “learning rate” rather than a win rate, defining failure as poorly designed experiments while treating disproven hypotheses as useful outcomes. The company applied this approach to develop its Inspire recommendation engine, which initially underperformed against an external market leader but improved through hundreds of iterative tests over two years and ultimately replaced the vendor. The account argues that building an internal platform is most appropriate when experimentation technology and proprietary data are strategically important and an organization has the technical capacity to maintain it.

Sample ratio mismatch (SRM) occurs when an experiment’s observed traffic allocation differs significantly from its configured split, undermining the random assignment required to attribute outcomes reliably to a treatment. The issue can originate in assignment, exposure logging, analysis filters, data processing, or interference during an experiment, and may create selection bias when variations affect which users are recorded or retained. Diagnosis involves confirming that metrics use the same unit and identifier as randomization, checking for mid-run changes, determining whether the problem affects other experiments, examining its timing, severity, direction, and affected segments, comparing performance and engagement measures, and tracing unit counts through the data pipeline. Results can be recovered when correctly captured raw data was distorted only by a fixable processing or analysis issue, but experiments should be rerun with re-randomization when data was not recorded correctly, the treatment changed the population being measured, or the cause remains severe and unknown; in limited cases, an externally caused and time-bounded imbalance may support only documented directional insight. GrowthBook automates SRM detection and provides traffic, segmentation, pre-exposure bias, and multiple-exposure checks to help identify underlying causes.