Company
Date Published
Author
Maggie Stewart
Word count
1198
Language
English
Hacker News points
None

Summary

A/B testing often faces challenges when tracking customers transitioning from logged-out to logged-in states, particularly when testing features like account creation flows. Proper selection and logging of Unit IDs, such as User ID or Stable Device ID, are critical for obtaining valid experimental results. User IDs typically represent individual accounts and are used when available, while Stable Device IDs are useful alternatives in scenarios where User IDs are unavailable, such as client-side experiments targeting anonymous users. These device-level experiments allow for comprehensive tracking of user engagement both before and after account creation, although they have limitations, such as inability to track across multiple devices or distinguish users sharing a device. For long-term studies, User IDs are more reliable due to their consistency across devices. The text also touches on the benefits of CUPED for reducing bias in experiments and highlights insights from experts on building a robust experimentation culture.