Calculate exact relative metric deltas with Fieller intervals
Blog post from Statsig
Fieller's method, now available in Statsig, offers a more precise way to calculate confidence intervals for relative metric deltas compared to the Delta Method, which uses an approximation for variance. This approach is particularly useful when dealing with noisy data or small denominators, providing an exact solution for confidence intervals, which are typically asymmetrical, unlike the symmetrical intervals from the Delta Method. Fieller intervals are recommended for new experiments as they offer more accurate and reliable results, especially in edge cases where the control group's mean isn't statistically distinguishable from zero, potentially leading to unbounded confidence intervals. While existing Statsig users may notice changes in notation and reporting style due to the switch to Fieller intervals, the increased precision justifies these adjustments, and the platform encourages adopting this methodology for more dependable experimental analysis.