September 2026 Summaries

In a discussion on The Experimentation Edge, PayPal Group Product Manager Gaurav Sethi described how rebuilding the company’s experimentation program transformed it from a system with an implausible 55% to 60% win rate into one that ran 2,587 experiments and measured roughly $180 million in annual impact. He traced the inflated win rate to assignment-based tracking, which counted users assigned to a test rather than users who actually saw it, creating diluted data and misleading statistical significance; replacing it with near-render exposure events improved result credibility. The team also standardized fragmented behavioral, transactional, and financial metrics across PayPal’s brands, enabling automated analysis and reducing experiment readout time from up to four weeks to 24 hours. Sethi highlighted using multi-armed bandits to reduce a projected 700-day homepage carousel test to a 51-day decision, and argued that failed tests create value through avoided development, maintenance, and customer costs. He emphasized deliberate hypotheses, predefined KPIs and guardrails, broad education for product and engineering teams, and feature flags for production AI agents, whose evaluation must account for task completion, consistency, user intent, and operating cost.

ServiceNow’s Ashraf Karim, Senior Vice President of Connected Customer Experiences and Technology, describes how the company acts as “Customer Zero” by testing new platform features internally before releasing them to customers, generating practical lessons about AI adoption in post-sale experiences such as support, onboarding, education, and developer services. She emphasizes that effective experimentation requires strong data foundations and instrumentation, particularly in historically underinvested post-sale functions, and focuses on customer outcomes including time to value, adoption speed, and fast issue resolution. Early agentic support tools successfully automated complex troubleshooting tasks but were initially abandoned because users found five-to-fifteen-minute waits unacceptable, leading the team to improve communication about background progress rather than simply changing the models. Users also preferred concise recommendations and next actions over visibility into every automated troubleshooting step, though the desired level of transparency can change as trust develops. ServiceNow evaluates nondeterministic AI outputs with an internal LLM judge against golden datasets while preserving customer control through follow-up questions and live-agent escalation, but it relies on real-user experimentation to measure satisfaction. Alongside business metrics such as revenue, operational savings, and agent time saved, the company tracks customer effort score, completion rates, repeat contacts, sentiment, and CSAT to distinguish technology performance from experience design. Karim argues that continual testing, learning from unsuccessful ideas, and separating technology variables from experience variables are essential as organizations deploy AI agents across diverse customer needs.

Wanli Lau, now leading global R&D analytics at Twilio after eleven years at Expedia Group, discusses how experimentation principles change between high-volume consumer marketplaces and lower-volume B2B developer platforms. At Expedia, an account-signup homepage takeover initially increased conversions but later proved harmful to return visits and engagement, prompting the team to make guardrail metrics, customer journey mapping, UX research, and formal trade-off escalation part of experiment design. Expedia’s program, which ran more than 1,500 A/B tests annually, also relied on an experimentation playbook that required predefined primary, secondary, and guardrail metrics to reduce cherry-picking and p-hacking. At Twilio, experimentation must account for smaller samples, account-level versus user-level randomization, and interference risks when multiple developers from the same company may see inconsistent experiences or prices. Lau argues that when statistical significance is difficult to reach, teams can still make disciplined decisions based on clearly defined hypotheses, signals, guardrails, and explicit conviction rather than treating significance as the only basis for action. She identifies self-service experimentation through Amplitude and the integration of web, financial, and account data as key opportunities to improve velocity and better understand customer journeys and friction points.

Feature flags enable teams to change production behavior without redeploying code, but can bypass conventional pull-request, CI, and approval controls, creating change-management risks under SOC 2, particularly Type II audits that assess controls over time. Relevant SOC 2 criteria include authorization, documentation, testing, approval, access restriction, monitoring, incident response, and vendor management; organizations must demonstrate evidence such as immutable logs of flag changes, environment-specific role-based access, segregation of duties, SSO and SCIM-based deprovisioning, scoped API keys, approval workflows, validation and rollout guardrails, SIEM exports, and documented emergency-change exceptions. Auditors commonly request samples of production changes showing before-and-after values, authors, approvals, affected environments, linked business context, access reviews, deprovisioning records, and retention policies. The text presents GrowthBook as a platform designed to provide these capabilities through draft-and-approval workflows, audit trails, RBAC, monitoring, rollback features, integrations, and SOC 2 Type II attestation, while noting that a vendor’s certification does not remove customers’ responsibility for their own controls.

GoPro digital product leader Will Guyeskey argues that experimentation programs should prioritize customer learning over win rate, since winning, losing, and flat A/B tests can all produce valuable insights when designed rigorously. Drawing on experience at Brooks Bell, Gap brands, and GoPro, he emphasizes that results are audience-specific: a recommendation module that increased Barnes & Noble order values on confirmation pages succeeded because book buyers readily made additional low-cost purchases, but failed with other retailers whose customers and products differed. At GoPro, a test of whether product-page configuration steps created excessive friction produced a flat result, revealing that buyers of high-consideration cameras valued guidance on subscriptions, SD cards, and accessories rather than viewing the choices as obstacles. Guyeskey advocates sharing both interim and final results across teams while explaining volatility and waiting for statistical significance, and he cautions that AI may accelerate test creation without replacing human judgment or the need to absorb lessons. His central view is that tests should be structured to yield meaningful learning regardless of outcome, because understanding customers ultimately improves experiences and performance.

GrowthBook 5.1, available to cloud and self-hosted users, expands the company’s experimentation platform with AI-assisted tools intended to support a continuous product-learning cycle. The release adds contextual bandits that personalize variant delivery and automatically adjust traffic, funnel metrics for assessing completion and drop-off across multistep experiences, and a Hosted MCP and OAuth server with more than 30 AI skills that can help agents create experiments and publish feature flags. It also introduces Learnings, a shared library for recording and retrieving experimental findings, along with a beta Slack app for querying GrowthBook data, interacting with its AI assistant, and receiving notifications or digests. Product Analytics enhancements include User Journeys for examining actions before and after key events, AI-generated shareable dashboards based on natural-language prompts, and a redesigned SQL Explorer that enables broader warehouse analysis without predefined metrics.

Feature flags allow teams to change production behavior without code deployments, but in regulated sectors they require the same governance as other production changes because auditors may require evidence of authorization, approvals, data handling, and accountability. Key compliance controls include immutable and exportable audit trails, role-based access and least-privilege permissions, separation of duties with approval workflows, careful handling of PII and PHI through local evaluation and data-residency options, and traceability for automated or AI-driven changes. The discussion connects these controls to SOC 2, HIPAA, GDPR, and SOX requirements, emphasizing evidence such as change histories, reviewer records, environment details, bypass logs, and SIEM integrations. It also advises organizations to assess vendors’ certifications, security practices, deployment options, data flows, encryption, identity management, and industry-specific contractual commitments. GrowthBook is presented as an example platform offering governance features, open-source code, warehouse-native data management, and self-hosting options intended to help organizations verify controls and meet strict residency requirements.

A strong experimentation culture treats testing as a default way to make decisions, prioritizing evidence over seniority, learning over always being right, and well-designed hypotheses over untested assumptions. Because relatively few experiments improve their target metrics, successful programs require sustained leadership support, psychological safety around negative outcomes, explicit and measurable hypotheses, trusted and transparent metrics, and accessible testing tools with appropriate governance. The text also emphasizes sharing results—including unsuccessful tests—through searchable repositories, cross-team communication, and recurring rituals such as reviews, prioritization sessions, office hours, retrospectives, and executive readouts. Examples from organizations including DoorDash, Microsoft, JPMorgan Chase, The Home Depot, and the Philadelphia Inquirer illustrate how leadership engagement, shared standards, and centralized learning can help experimentation scale across departments. It concludes that experimentation platforms can support this culture by reducing technical barriers, standardizing metrics and controls, and making results easier to access, while noting that organizational behavior and trust are more important than tooling alone.

Samsung Electronics America Lead Product Manager Anuradha Tempe discusses how experimentation differs between Samsung’s mature B2C storefront and its newer B2B platform, where lower traffic, highly variable bulk orders, longer approval processes, and cross-device buyer journeys require more careful interpretation of test results. Her team normalizes B2B order data to prevent large purchases from creating misleading outcomes and reserves full A/B tests for high-risk decisions, while using UAT feedback or pre/post analysis for simpler changes. An experiment that surfaced software and service add-ons alongside hardware purchases improved conversion and average order value, with add-on sales nearly doubling after the team recognized that business customers often research on mobile before completing approved purchases on desktop. Conversely, an early “Buy Now” option confused bulk buyers who wanted product details and configuration clarity before purchasing, reinforcing the principle that B2B customers prioritize confidence over premature speed. Tempe emphasizes setting a North Star metric before building, monitoring guardrails such as conversion or customer feedback, openly sharing unsuccessful results, and introducing experimentation practices collaboratively across teams and cultures. She is also developing an AI-assisted testing copilot to generate hypotheses and triage test needs, while maintaining human review to preserve judgment and prevent unreliable automated outputs.

Progressive delivery separates code deployment from user release by using feature flags, phased exposure, and observability to limit the impact of failures, an approach highlighted by OpenAI’s December 2024 outage after a telemetry service was deployed to all production clusters simultaneously. Teams can begin with targeted releases to employees or beta users, expand through percentage-based or monitored canary rollouts, or automate staged Safe Rollouts that advance only when guardrail metrics remain healthy and can pause or reverse when issues emerge. Feature flags also enable rapid rollback through kill switches, avoiding slower code redeployments, while streaming or polling configurations determine how quickly changes reach users. Effective rollout monitoring combines a small number of guardrail metrics, such as error rates, latency, and conversion rates, with secondary signal metrics and implementation health checks like sample ratio mismatches. When selecting a feature flag platform, organizations should assess automation, statistical measurement, warehouse-native metrics, approval workflows, auditability, SDK coverage, self-hosting options, and safeguards for AI-initiated changes.

Joe Yevoli, Director of Growth at Charlie Health, discusses on The Experimentation Edge how effective experimentation depends less on achieving immediate wins than on learning from unexpected outcomes across the full customer journey. He describes a landing-page intake-form test that increased funnel entry but reduced downstream conversion by admitting more poorly informed prospects, leading the team to move explanatory content earlier in the experience through a potential FAQ chatbot. Drawing on a previous role at Teachers Pay Teachers, he explains how narrowing the digital product Easel to a specific homework-assignment use case reduced its addressable audience but substantially improved retention, reinforcing the value of serving a focused segment rather than designing broadly for everyone. Yevoli also advocates premortems for high-risk experiments, in which teams assume a test has failed and identify potential causes before launch, helping surface concerns and add safeguards. His broader approach combines quantitative metrics, qualitative user research, full-funnel measurement, segmentation, and openness to failure, while cautioning that AI-enabled speed should not replace careful reasoning and awareness of unknowns.

Feature flags control which users receive a software change and enable rapid rollbacks, while A/B tests use randomized, persistent assignments and concurrent control groups to determine whether a change caused a measurable effect. A percentage rollout alone cannot establish causality because outcomes may be affected by timing, user mix, or other simultaneous changes. Converting a flagged release into an experiment requires a predefined hypothesis, treatment and control variations, randomized sticky assignment, primary and guardrail metrics, and statistical analysis that can detect significance, sample-ratio mismatches, and other data-quality issues. Percentage rollouts are suited to limiting exposure risk, safe rollouts automate staged releases and rollback based on guardrails, and experiments are appropriate when teams need evidence of impact. Recommended practices include logging exposures server-side, validating tracking with A/A tests, expanding experiment coverage without reshuffling existing users, and preventing interference between concurrent tests through coordination or namespaces. The text argues that integrated platforms such as GrowthBook can combine feature management, experimentation, warehouse-based metrics, and statistical reporting to reduce duplicated tools and improve testing workflows.

GrowthBook Cloud has introduced an EU data region for Managed Warehouses in AWS eu-west-1, allowing event data to be ingested and stored in an EU-based managed ClickHouse warehouse while retaining the convenience of hosted setup and maintenance. Event ingestion and storage remain in the EU, while the GrowthBook application, including login, interface, configuration metadata, and computed experiment results, remains hosted in the United States; customers using their own warehouse continue to keep event data entirely outside GrowthBook’s systems. The managed offering provides one-click provisioning, preconfigured experimentation tables and metrics, SDK and API-based tracking, near-real-time enriched event querying, and raw SQL access through SQL Explorer. Available on all GrowthBook Cloud plans, the EU region can be selected when creating a Managed Warehouse or Event Forwarder, with SDK setup configured to use the EU ingestion endpoint.

Mi Tian, Head of Applied Science at Zalando, describes how the e-commerce company manages more than 1,000 annual experiments by linking product decisions to a company-wide North Star goal through a KPI tree that connects high-level objectives, measurable proxy metrics, and controllable product inputs. Zalando’s centralized economics and experimentation organization, which reports under the chief economist and CFO, combines experimentation, causal inference, and economic analysis to select appropriate measurement methods and reinforce executive support for evidence-based decisions. Its self-service platform enables teams to launch and analyze tests, while risk tiering balances rapid, lower-risk experimentation with more rigorous evaluation of potentially disruptive initiatives. This approach supported the launch of Zalando’s personalized discovery feeds, where short-term A/B tests and long-term holdouts measured engagement, transaction effects, and gross merchandise value, ultimately indicating positive results. Looking ahead, the company is exploring LLM-based customer-behavior simulations to screen hypotheses and inform prioritization, while emphasizing that such tools are not replacements for live experiments.

Whitney Perez, Director of Product Management at Realtor.com, discusses with GrowthBook CMO Ashley Stirrup how effective experimentation depends less on simply running A/B tests than on establishing reliable data, shared testing knowledge, clear guardrails, and peer review processes. Drawing on experience at Citrix, GoDaddy, and Realtor.com, Perez advocates the “30/30/30” expectation that roughly equal portions of experiments will win, lose, or be inconclusive, helping teams value learning rather than chase only positive results. She illustrates the importance of secondary and guardrail metrics through a Realtor.com bundling test that appeared to increase attach rate by 300% but ultimately reduced total revenue because an added checkout step caused customer drop-off. Perez argues that experimentation skills are accessible to non-specialists who understand core concepts such as hypotheses, decision metrics, traffic requirements, confidence intervals, and confounding variables, while organizations should connect team-level metrics to broader North Star goals and use qualitative research before testing. She sees AI as a useful tool for analyzing data and uncovering opportunities, but emphasizes that human review remains necessary because AI can produce confident but incorrect conclusions. Her priorities for Realtor.com are dependable instrumentation, experimentation knowledge distributed across teams, and a culture that welcomes positive, negative, and inconclusive outcomes.

Rich Liebling, senior director of engineering at Learneo, describes how experimentation at Shop It To Me helped grow the service from about 50,000 to one million subscribers in nine months by making A/B testing a company-wide operational goal. A key practice emerged after a simplified page performed worse than the original while the opposite change—adding more explanatory text—won: teams should test the inverse of a focused hypothesis, since either result can reveal whether a variable meaningfully affects a target metric. Liebling emphasizes that small, isolated tests produce clearer learning than broad redesigns, while larger product bets may require repeated iteration, and successful variants should also be refined rather than assumed optimal. Shop It To Me translated its strategic subscriber target into a controllable goal of 300 tests, embedded experimentation into engineering onboarding by having new hires quickly ship their own tests, and encouraged ideas from all departments. At Course Hero, experimentation faces additional challenges, including teams with competing upload and subscription incentives, slow ad hoc analysis, concurrent-test interactions, and seasonal traffic patterns; the company addresses some of these through shared lifetime-value metrics and multi-armed bandit testing that shifts traffic toward stronger variations.

Trunk-based development relies on frequent merges into a single deployable main branch, a practice associated with stronger software delivery performance and reduced merge-conflict risk compared with long-lived branching models. Because substantial features and migrations often take longer than a day to build, feature flags can keep incomplete code deployed but inactive, separating deployment from user release and allowing teams to merge incremental work safely. A flag-first workflow involves creating a flag before coding, testing both enabled and disabled paths in CI, using environment-specific configurations, and conducting dark launches or gradual rollouts to selected users while monitoring results. For complex system replacements, branch by abstraction can allow legacy and new implementations to coexist behind a common interface. The discussion emphasizes that flags require lifecycle management, including owners, removal dates, stale-flag detection, and code cleanup, because unused flags multiply possible code paths and create technical debt. It also warns against adding flags only after code is live, using them to conceal long-lived branches, or failing to remove them after a feature is fully released, while presenting GrowthBook as a platform supporting local flag evaluation, environment controls, staged rollouts, monitoring, and cleanup tools.

Priya Singhee, VP of Enterprise Analytics & Data Science at Early Warning and former Wayfair analytics leader, argues that effective experimentation depends less on launching A/B tests than on determining whether they can be conducted validly. Her framework emphasizes clean randomization, alignment between randomization and measurement units, sufficient traffic and plausible effect sizes, reversible changes, and a specific falsifiable hypothesis. Because an estimated 85–90% of tests fail, she advises organizations to treat experimentation as a learning agenda rather than rewarding only positive results, since unusually high win rates may indicate flawed measurement or false positives. Singhee recommends pre-registering hypotheses, metrics, statistical methods, planned subgroup analyses, decision rules, and kill criteria to limit p-hacking and post-hoc interpretation, while documenting all experiments and lessons companywide. She maintains that testing prevents costly feature launches by identifying losing ideas before they harm revenue, making loss avoidance and organizational learning more valuable than maximizing apparent test wins.

Choosing whether to build or buy a feature-flagging system depends on an organization’s scale, technical requirements, governance needs, and willingness to maintain internal infrastructure. Building can suit small teams with simple server-side Boolean flags, a single service and language, engineer-only controls, and a need for tight integration or independence from vendors, but internally developed tools often become costly as requirements expand. Complex targeting, gradual rollouts, multi-language SDK consistency, audit logs, access controls, compliance, automated rollback, debugging, and stale-flag cleanup can require substantial ongoing engineering effort, on-call support, and technical-debt management. The text recommends a structured assessment of flag volume, release cadence, user roles, data residency, maintenance capacity, and cost predictability, suggesting that teams with broad, high-velocity, multi-platform needs may benefit from a platform. It also highlights evaluation criteria for purchased tools, including local flag evaluation, SDK coverage, governance features, lifecycle management, observability, warehouse analytics, self-hosting, open-source transparency, and standards that reduce vendor lock-in.

Feature flag governance is presented as a policy and automation framework for controlling flags throughout their lifecycle, reducing risks such as technical debt, untraceable production changes, outages, and compliance failures. Citing incidents including Knight Capital’s 2012 trading loss and CrowdStrike’s 2024 outage, the discussion emphasizes six core controls: clear flag categories and lifespans, standardized naming and ownership metadata, environment-specific role-based access, independent approval workflows for production changes, automated configuration validation, and proactive cleanup supported by audit trails. These measures become especially important as AI coding agents can create and modify flags through APIs, requiring their actions to follow the same permissions, validation, review, and attribution requirements as human users. Organizations are advised to adopt governance incrementally by first enforcing ownership and naming, then limiting production permissions, requiring reviews, adding validation, scheduling cleanup, and embedding enforcement in daily workflows. GrowthBook is highlighted as an open-source platform offering capabilities for these controls, including custom fields, environment-scoped permissions, approval policies, JSON Schema validation, stale-flag detection, code references, and auditable records suitable for self-hosted or regulated environments.

Scaling A/B testing requires more than increasing experiment volume; it depends on a company-wide operating model that preserves reliable measurement, shared learning, and quality controls across teams. The text identifies five core pillars: a scalable warehouse-native technical foundation with consistent metric definitions and A/A validation tests; rigorous prioritization frameworks that weigh expected impact against cost; an integrated experiment repository that records hypotheses, results, and reusable learnings; statistical safeguards such as pre-committed designs, automated health checks, multiple-testing corrections, variance reduction, and replication of surprising results; and self-service tools governed by required review workflows. Examples from organizations including DoorDash, Disney, Home Depot, Chess.com, Fyxer, the Philadelphia Inquirer, and Lingokids illustrate approaches to running experiments at different scales. It argues that programs should judge success not by test counts alone, but by validated product improvements, avoided harmful launches, and accumulated knowledge about users, while presenting GrowthBook as a platform intended to support these practices.

Supercell conducts relatively few A/B tests despite serving 300 million monthly players, prioritizing high-impact, hypothesis-driven experiments that preserve its decentralized creative culture and focus on player retention rather than revenue. Its central experimentation team provides testing infrastructure and promotes rigor while allowing independent game teams to make their own decisions, using experiments to validate or challenge creative ideas rather than simply follow dashboard metrics. To maintain player trust, Supercell publicly announces tests, explains their purpose, and offers make-up events to ensure players who receive less favorable test experiences are treated fairly. AI tools are now enabling more employees to independently analyze experiments, expanding access to data but raising concerns about analytical quality and overconfident errors. Supercell and its experimentation partners view templates, guardrails, and standardized defaults as essential for embedding expert-level rigor into self-service AI analysis, reinforcing the broader principle that trustworthy experimentation depends on transparency, fairness, and reliable systems.