Benchmarks and Obscurantism: A “red” line that should not be crossed

At a recent Databricks Data and AI Summit, Databricks showcased a benchmark for its new low-latency compute product, Reyden, claiming superior performance over ClickHouse, which allegedly crashed during the test. However, the benchmark's lack of transparency regarding hardware, configuration, and methodology raised concerns about its validity. ClickHouse attempted to replicate the benchmark using available details but found no crash, achieving 15,000 QPS with proper sizing and no optimizations. The situation highlights the importance of open, reproducible benchmarks that provide meaningful insights and facilitate fair competition. Benchmark transparency is essential for ensuring credibility and fostering innovation, as demonstrated by ClickHouse's approach of openly sharing configurations and results for public scrutiny and verification.

No tracked trend matches for this post yet.