What does a holdout test actually measure?

Holdout testing is an experimental method used to measure the overall impact of multiple features released over a period by comparing a group of users who did not receive the updates against those who did. This method addresses the inflation of individual A/B test results due to the winner's curse and potential feature interactions that might cancel each other out. The text examines four configurations of holdout testing, each providing a slightly different perspective on the cumulative effect of recent product changes: full-terminal, full-incremental, split-incremental, and split-terminal, with a fifth, reverse holdout, as an after-the-fact option. These configurations differ based on whether they focus on the long-term settled value of features or the real-time experience of users, how they handle novelty effects, and their approach to accounting for interactions between features. The right configuration depends on the desired outcomes, such as understanding the net effect of all features or identifying which features or teams drive more impact. The choice between configurations considers factors like interaction weight, novelty effects, and the power cost of maintaining separate test groups.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.