Introducing stratified sampling

Statsig has introduced stratified sampling to enhance the reliability of experimental results by addressing pre-existing group differences in experiments, particularly beneficial for B2B tests or those influenced by a few power users. This feature helps reduce false positive and discovery rates, leading to more consistent results. Stratified sampling works by using multiple "salts" to randomize subjects and selecting the best one for balanced group distribution, ensuring results are consistent and trustworthy across repeated experiments. Despite existing tools like Winsorization and CUPED, stratified sampling addresses unique challenges such as extreme bias and imbalance among small-unit-count experiments. This method is integrated into Statsig's workflow by allowing users to configure metrics or attributes to balance, streamlining the process and enhancing experiment confidence levels without increased data collection time.