When allocation point and exposure point differ

A/B testing can be complicated by discrepancies between allocation and exposure points, as demonstrated in scenarios where users are assigned to a test group but do not interact with the new feature being tested. This misalignment can dilute the test's statistical power, making it harder to detect true effects, because unexposed users in the test group behave like control users. The issue can arise due to technical challenges in tagging exposure events or the use of offline allocation, where group assignments are made before the experiment and some users do not engage during the test phase. Although post-hoc solutions, such as filtering out non-relevant users, are available, the best strategy is to ensure accurate tagging of exposure points or adjust test planning by considering the proportion of non-relevant data. Proactive preparation can prevent these issues, ensuring that A/B testing remains sensitive enough to detect meaningful differences, thereby avoiding incorrect conclusions about a new feature's impact.