Introducing CostBench: an open benchmark for data warehouse cost-performance

CostBench is an open benchmark designed to evaluate the cost-performance of cloud data warehouses, focusing on performance-per-dollar rather than just speed to help teams choose the most efficient system for real-time analytical workloads. It emphasizes that speed alone is insufficient because costs vary based on how much compute power a system requires to complete tasks, with different platforms using various units to measure this. CostBench specifically measures both read-side cost-performance, which assesses query efficiency per dollar, and write-side cost-performance, which evaluates the cost-effectiveness of processing fresh data for queries. The initial release highlights read-side performance across platforms like ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift, revealing ClickHouse Cloud as the only option maintaining fast and low-cost performance as data scales. By publishing its methodologies, scripts, and results openly, CostBench allows users to inspect, verify, and potentially improve the configurations, removing the opacity often associated with cost-performance assessments.