September 2026 Summaries

Multiple comparisons in A/B testing arise when teams evaluate numerous metrics, variants, or audience segments, increasing the probability that at least one statistically significant result is a false positive even when individual tests use a 5% significance level. For example, 10 independent comparisons at a 5% alpha level produce roughly a 40% chance of at least one false positive, potentially leading teams to prioritize ineffective features, report gains that do not exist, and distort their understanding of what drives outcomes. Statsig addresses this issue through Bonferroni and Benjamini-Hochberg corrections: Bonferroni controls the probability of any false positive across a comparison family by applying a stricter threshold to each test, while Benjamini-Hochberg controls the expected proportion of false discoveries among declared wins and can retain more real effects. Statsig also offers preferential Bonferroni for emphasizing primary KPIs and selective Benjamini-Hochberg correction on primary metrics to preserve sensitivity for guardrail metrics, with the appropriate method depending on whether teams prioritize minimizing false positives or detecting more genuine effects.

Statsig outlines an AI strategy for its experimentation platform centered on trustworthy, high-impact assistance rather than adding AI broadly for speed alone. The company sees AI as a way to reduce friction in self-service experimentation by consolidating data, connecting signals, explaining findings, and freeing data scientists from routine analysis, while acknowledging technical users’ concerns about hallucinations and unverifiable reasoning. Its approach emphasizes grounding AI outputs in real data with guardrails and transparent evidence, providing conservative and customizable controls over autonomous actions, and making AI optional so it augments rather than disrupts established workflows. Statsig intends to prioritize AI for more time-consuming tasks such as generating code, analyzing results, prioritizing test ideas, and answering program-level questions across many experiments, with the broader goal of helping teams improve both the efficiency and quality of their experimentation.

Effective experimentation requires determining whether a change caused an outcome and selecting a design that fits the underlying research question rather than relying automatically on standard A/B tests. A/B testing depends on assumptions including independent observations, no interference between participants, predetermined sample sizes in frequentist testing, and alignment between randomization and measurement units; violations can produce misleading results. Experiments can also fail by answering the wrong question, such as optimizing lead volume rather than lead quality, making careful problem framing essential. Alternatives address common constraints: cluster randomization handles account-level spillovers, holdouts measure cumulative program impact, synthetic controls estimate market-level campaign effects, switchback tests address shared marketplace resources, and non-inferiority tests assess whether a change avoids material harm. Additional approaches include A/A tests for evaluating metric and pipeline behavior, elasticity and response-surface methods for optimizing levels or combinations of interventions, sequential testing for faster safety decisions, and multi-armed or contextual bandits for adapting traffic allocation and personalization. The central recommendation is to define the population, outcome, meaningful effect, and causal question first, then choose an appropriate experimental design and statistical method, potentially using advanced experimentation platforms when internal technical resources are limited.

Statsig’s August updates focused on improving cross-platform control, expanding AI capabilities, and addressing customer usability requests. New integrations and API enhancements include PagerDuty support for Topline Alerts, daily experiment-check retrieval, individual feature-gate override deletion, SCIM role aliases for custom identity-provider naming, query cost-attribution tags, and improved Athena pipeline visibility. AI-related changes added five MCP experiment-configuration fields, allowing coding agents to modify settings such as identifier mappings, non-production environments, links, and secondary metrics without using the console. Usability improvements include layers as a top-level experiment-targeting option to support mutually exclusive tests, expanded targeting multiselect options, and reusable display names across gates, experiments, and configurations. Statsig also indicated that larger releases are planned over the coming months, including progress toward an AI-powered behavioral and validation platform with Amplitude.