When being "good enough" is enough: Understanding non-inferiority tests

The concept of "do no harm" versus "do good" significantly influences the choice of testing methods, such as superiority and non-inferiority tests, used in A/B testing scenarios. While superiority tests aim to identify a clear winner by showing that a new version outperforms an old one, non-inferiority tests are designed to confirm that a new version's performance does not significantly decline beyond an acceptable margin, often used in contexts where changes are necessary, such as compliance or design updates. These tests are particularly relevant in cases like branding updates, algorithm improvements, or regulatory compliance, where the goal is to ensure no significant degradation of key performance metrics. The key to a successful non-inferiority test lies in setting an appropriate non-inferiority margin, balancing statistical power, and business considerations, and recognizing that a non-significant result does not equate to non-inferiority. Implementing non-inferiority tests requires a cultural shift within organizations to avoid misinterpretation of results and overuse in scenarios where improvement is the primary goal, emphasizing the importance of clearly defining test objectives upfront.