It’s normal not to be normal(ly distributed): what to do when data is not normally distributed

William Sealy Gosset, a brewer at Guinness, developed the t-test to address the challenge of estimating beer quality with small sample sizes, which later became a cornerstone of statistical analysis. While the t-test assumes normality of data distribution, this assumption is often violated in A/B testing, leading to concerns about the reliability of results. However, the Central Limit Theorem suggests that the t-test remains robust for large samples, even if the data isn't normally distributed. Despite this robustness, the t-test may not always be the best choice, as alternatives like non-parametric tests and bootstrapping can be more effective in cases of significant skewness or outliers. These alternatives offer advantages, such as not relying on distribution assumptions, but also come with drawbacks, including computational intensity and reduced interpretability. In some scenarios, using a parametric test tailored to the specific distribution of the Key Performance Indicator (KPI) might be more powerful. While the t-test remains a reliable tool, data analysts should consider the data's distribution and context to select the most appropriate method.