Designing controlled experiments to test correlated metrics

Correlated metrics refer to the relationship between different variables where changes in one can predict changes in another, often measured by the Pearson Correlation Coefficient. In an experimental setting, these metrics can be categorized into metric families, surrogate metrics, and intrinsic metrics, each serving distinct roles. Metric families help explain the underlying causes of changes in user behavior by analyzing related metrics like total spend per user and purchase frequency. Surrogate metrics are used to predict long-term business outcomes based on early experimental data, although they require careful accounting for prediction errors. Intrinsic metrics act as guardrails against false positives by highlighting inconsistencies in experimental results, such as when unexpected changes in related metrics suggest a misleading outcome. Correlated metrics pose challenges in multiple comparison corrections, as many traditional methods assume metric independence, leading to potential loss of statistical power when metrics are correlated.