Multiple comparisons: More comparisons, more problems

Multiple comparisons in A/B testing arise when teams evaluate numerous metrics, variants, or audience segments, increasing the probability that at least one statistically significant result is a false positive even when individual tests use a 5% significance level. For example, 10 independent comparisons at a 5% alpha level produce roughly a 40% chance of at least one false positive, potentially leading teams to prioritize ineffective features, report gains that do not exist, and distort their understanding of what drives outcomes. Statsig addresses this issue through Bonferroni and Benjamini-Hochberg corrections: Bonferroni controls the probability of any false positive across a comparison family by applying a stricter threshold to each test, while Benjamini-Hochberg controls the expected proportion of false discoveries among declared wins and can retain more real effects. Statsig also offers preferential Bonferroni for emphasizing primary KPIs and selective Benjamini-Hochberg correction on primary metrics to preserve sensitivity for guardrail metrics, with the appropriate method depending on whether teams prioritize minimizing false positives or detecting more genuine effects.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.