Transparency in benchmarking

Database benchmarks, though imperfect, are valuable tools that help users understand tradeoffs and make informed decisions about infrastructure. However, many cloud vendors have restrictions known as "DeWitt clauses" that limit comparative benchmarking, a practice counter to the legacy of database researcher David DeWitt, who advanced the industry by publishing real system measurements. PlanetScale is moving away from such restrictions by updating their Acceptable Use Policy (AUP) to allow and encourage fair and transparent benchmarking, provided it adheres to best practices. They stress that benchmarks should be conducted in good faith, avoiding misrepresentation or interference, and they advocate for a new standard that emphasizes depth, technical soundness, and reproducibility. PlanetScale invites other vendors to adopt similar standards in their AUPs, encouraging transparency and allowing customers to effectively compare systems.