Bad controls: why splitting your experiment on a post-treatment variable backfires

Randomized experiments can produce misleading subgroup, funnel, and variance-reduction estimates when analysts condition on variables measured after treatment assignment that the treatment may have changed, creating “bad controls” and post-treatment bias. In a simulated reading-app experiment with a true uniform gain of two articles per user, engagement tiers recalculated during the experiment made every subgroup appear to benefit less than the overall population because treatment pushed users into higher tiers, causing treated and control users within each tier to represent different underlying populations; tiers fixed before launch correctly recovered the effect. This differs from Simpson’s paradox because the overall estimate remains reliable while the post-treatment subgroup analysis introduces the distortion. Similar problems arise when evaluating funnel steps only among users who reached an earlier step, or using step-to-step conversion rates with treatment-affected denominators, since these condition on users selected partly by the intervention; outcomes should instead generally be measured per assigned or exposed user. Variance-reduction methods such as CUPED, regression adjustment, and post-stratification are safe only with pre-experiment covariates, as adjusting for during-experiment behavior can remove part of the treatment effect itself. The central safeguard is to split, filter, or adjust only using variables known before the experiment begins, while using balance and sample-ratio checks as supporting diagnostics rather than definitive proof of validity.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.