Why is it OK to run a z-test on a non-normal distribution for A/B tests?

Z-tests and t-tests can generally be used for A/B-test metrics with non-normal raw distributions because they evaluate the difference between group means rather than individual observations. Under the central limit theorem, independently and identically distributed observations with finite variance yield sample means that become approximately normally distributed as sample sizes increase, and the difference between two such means is likewise approximately normal. Since online experiments commonly include thousands or millions of users and analyze user-level aggregates, this condition often holds for skewed metrics such as conversions, counts, and long-tailed revenue. The included R example demonstrates the principle by repeatedly resampling control and treatment data, plotting the resulting distribution of mean differences against a theoretical normal curve, and comparing simulated results with expected standard errors.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.