August 2026 Summaries

Experiments that appear to require months can sometimes be shortened by changing how outcomes are measured, but such adjustments introduce assumptions and tradeoffs that must be defined before testing to avoid p-hacking. Reducing metric variance can lower required sample sizes through approaches such as removing or capping outliers, converting continuous measures to binary outcomes, or applying transformations like logarithms, though each method can alter the meaning of the metric and potentially bias results. Teams can also choose proxy metrics that are more directly affected by a treatment or are less variable than the ultimate business KPI, such as recommendation clicks instead of total spending, while monitoring the original goal as a secondary measure. Time-windowed metrics add unavoidable observation periods after exposure, which can extend test duration, while CUPED uses correlated pre-experiment user behavior to reduce variance and improve A/B test sensitivity, although it is less useful for new users without historical data. The broader series also considers changes to sample-size assumptions, user targeting, and experiment design when metric adjustments alone cannot make a test practical.

Product managers from Statsig and Amplitude discuss how they prioritize experimentation, beginning with a naming test that temporarily freed the term “Journeys” for a journey-orchestration product before that initiative was deprioritized. They argue that most meaningful feature changes should be tested or placed behind feature gates, particularly when revenue, user behavior, or operational safety could be affected, while acknowledging that time, engineering capacity, and limited B2B traffic can constrain statistically significant A/B tests. Feature gates can still support safer releases, regression detection, observability, and directional learning even when experiments are underpowered. The group recommends prioritizing tests based on expected customer and business impact, implementation effort, measurable outcomes, available traffic, and the urgency of the information needed, including qualitative feedback for customer-specific features. They also contend that AI is reducing the cost of building variants and accelerating development, making experimentation culture increasingly important for keeping rapid product changes reversible, measurable, and low risk.