A quick guide to sample ratio mismatch (SRM)

The text discusses the concept of Sample Ratio Mismatch (SRM) in experimental design, particularly in the context of random assignment issues, using coin flip experiments as a metaphor to explain the statistical anomaly. It describes how SRM can occur due to various factors such as non-random assignment, data processing issues, or differences in exposure between test and control groups. The text highlights the importance of checking for SRM using tools like chi-squared tests and emphasizes the role of the p-value in identifying significant deviations from expected results. It further explores how tools like Statsig can automate the detection of SRM and suggests strategies to address and debug SRM, such as re-running experiments or employing methods like CUPED to control for pre-experiment bias. The text underscores the necessity of understanding the root cause of SRM to make informed decisions about experiment results and emphasizes the importance of rigorous methodologies to ensure reliable experimental outcomes.