Should you cap outliers? Winsorization in A/B testing

Winsorization in A/B testing caps values above a chosen percentile, such as replacing revenue above the 99th percentile of buyers with the threshold, reducing the influence of long-tailed outliers and often substantially lowering standard errors. Simulations of revenue experiments show that capping can provide precision comparable to much larger sample sizes and improve statistical power, particularly when a small group of high spenders contributes a large share of metric variance. However, it also changes the estimand by removing treatment effects among capped users: estimates remain effectively unbiased for effects distributed similarly across buyers but are biased downward when treatment effects scale with spending or are concentrated among top customers. Capping is therefore generally useful for ship-or-no-ship decisions when large buyers are not expected to respond disproportionately, while uncapped or carefully interpreted estimates may be preferable for forecasting and business cases that require the full treatment effect. Appropriate implementation matters, including excluding zero-spend users when defining buyer-based thresholds, recognizing the risks of deeper caps, and understanding that pooled thresholds reduce bias differently from per-arm thresholds, which can increase false-positive rates unless uncertainty is adjusted. CUPED, post-stratification, segment analysis, and comparisons across raw and differently capped metrics can complement winsorization by helping assess tail effects and reduce variance without the same bias.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.