Heap has developed a system called "shadow prod" to safely test database changes before deploying them to production. This system mirrors production environments, allowing engineers to evaluate changes under actual workload conditions without impacting live operations. Traditional benchmarking methods, such as using synthetic isolated benchmarks, often failed to predict real-world performance, leading to costly errors. Shadow prod addresses this by using Postgres backup and restore functionalities to create a duplicate machine with the experimental configuration. It mirrors reads and writes from production to ensure the experimental environment experiences the same workload, allowing for accurate comparisons between the experimental and control machines. This setup enables Heap to validate performance changes by programmatically analyzing query execution data and ensuring any observed differences are due to the intended modifications rather than extraneous factors. The system has proven valuable in identifying potential issues, such as configuration discrepancies or unoptimized query designs, before changes affect customers, ultimately allowing for faster and safer implementation of database optimizations.