Informed bayesian A/B testing: Two approaches

Bayesian methods in A/B testing offer an alternative to traditional frequentist approaches like null-hypothesis significance testing (NHST), promising intuitive metrics and flexible monitoring. "Informed Bayesian" methods incorporate prior knowledge to potentially speed up decision-making and provide more accurate estimates, but they come with challenges such as the need for careful calibration of priors to avoid skewed results and the potential for increased resource demands. The two main types of informed Bayesian adjustments include shifting the point estimate, which can risk data manipulation, and tightening the credible interval, which can slow down decision-making by requiring larger sample sizes. While Bayesian methods can complement frequentist techniques like False Discovery Rate (FDR) adjustments, their effectiveness depends on thoughtful implementation, transparent communication, and alignment with organizational goals to maintain a robust experimentation culture. By adopting practices such as empirical-Bayes pipelines and transparent reporting, teams can leverage Bayesian methods' benefits without compromising their testing speed or integrity.