Running faster tests: Modifying metrics (Part 2)

Experiments that appear to require months can sometimes be shortened by changing how outcomes are measured, but such adjustments introduce assumptions and tradeoffs that must be defined before testing to avoid p-hacking. Reducing metric variance can lower required sample sizes through approaches such as removing or capping outliers, converting continuous measures to binary outcomes, or applying transformations like logarithms, though each method can alter the meaning of the metric and potentially bias results. Teams can also choose proxy metrics that are more directly affected by a treatment or are less variable than the ultimate business KPI, such as recommendation clicks instead of total spending, while monitoring the original goal as a secondary measure. Time-windowed metrics add unavoidable observation periods after exposure, which can extend test duration, while CUPED uses correlated pre-experiment user behavior to reduce variance and improve A/B test sensitivity, although it is less useful for new users without historical data. The broader series also considers changes to sample-size assumptions, user targeting, and experiment design when metric adjustments alone cannot make a test practical.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.