Controlling your type I errors: Bonferroni and Benjamini-Hochberg

The Benjamini-Hochberg Procedure and the Bonferroni Correction are statistical methods employed to reduce false positives in multiple hypothesis testing, with the former now available on Statsig. The Benjamini-Hochberg Procedure is particularly useful when testing a large number of hypotheses, as it controls the False Discovery Rate (FDR), allowing for a moderate reduction in false positives compared to the more conservative Bonferroni Correction, which controls the Family-Wise Error Rate (FWER) and is optimal for a smaller number of hypotheses. Choosing between these methods involves balancing the risk of Type I errors (false alarms) and Type II errors (missed detections) based on the experiment's objectives and the resources available. Experimenters can apply these corrections per variant, per metric, or both, with the decision driven by the need to penalize distinct hypotheses. Statsig allows users to configure these settings within their platform, offering flexibility in managing false positives in experimental results.