We have built a performance testing system to understand the performance of our software, detect performance changes early, and aggressively redress the root cause. However, this data was overwhelming and needed advanced statistical techniques to process it. We realized we were trying to solve the wrong problem by comparing measured performance between two versions of software; instead, we focused on detecting changes in a measured value over time using change point detection. This technique is used to identify when changes in values occurred in time in the presence of noise or other confounding variables. We implemented this technique and integrated it with our production system, which now runs hundreds of distinct performance tests generating over 100k distinct results per software commit. Our system is good at detecting regressions and our engineers are good at fixing them. We share our data and learnings with the community to enable better research and take advantage of that research, aligning it with our needs.