Debugging sample ratio mismatch: Custom dimensions in Statsig
Blog post from Statsig
Statsig, a platform used by companies like Vista for running experiments, has introduced new debugging capabilities to address Sample Ratio Mismatch (SRM) issues that can lead to skewed and unreliable experiment results. SRM occurs when there's an uneven distribution of users between control and test groups, often due to technical issues like website crashes. Previously, Statsig's debugging tools were limited to a preset list of dimensions, but the recent update allows customers to define custom dimensions for more precise analysis. By enabling the analysis of these custom dimensions, Statsig offers deeper insights into the root causes of SRM, helping customers with complex setups to diagnose and resolve these issues effectively, ensuring the validity of their experiment findings.