What are guardrail metrics in A/B tests?

Launching new features in a product can often lead to unintended side effects, making it crucial to use guardrail metrics in A/B testing to ensure that improvements in primary metrics do not compromise overall system health and user experience. Guardrail metrics serve as a safety net by monitoring aspects such as system performance, user churn rates, or technical issues like page load times, thus providing a holistic view of the impact of tests. These metrics, distinct from primary metrics, help maintain balance by ensuring that gains in one area do not cause losses in another, a strategy employed by companies like Airbnb, Netflix, and Uber. However, selecting effective guardrail metrics requires a deep understanding of your product, users, and business objectives, and they should be meaningful and statistically powerful without causing noise or data overload. Implementing guardrail metrics in platforms like Statsig involves setting them up as secondary metrics and using features like Alerts++ for bug detection or adopting Sequential Testing for continuous monitoring to maintain product integrity and business health.