Heap is a product analytics tool that captures large-scale web and mobile user behavior data, storing billions of events daily in a distributed Postgres cluster. This vast scale creates unique challenges in optimizing SQL queries, as customers can use Heap's query builder to ask diverse questions involving filters, groupings, and comparisons. These query variations can significantly impact performance, especially when slight changes prevent efficient operations like index-only scans. To address this, Heap conducts performance experiments by constructing 95% confidence intervals to gauge the variability in query performance and uses a controlled testing method that alternates between experimental and control SQL statements to mitigate caching effects. This approach ensures that any observed performance improvements are genuine, even though it requires careful management of system loads and sample sizes.