CostBenchのご紹介: データウェアハウスのコストパフォーマンスを測るオープンベンチマーク

CostBench is an open benchmark designed to measure the cost performance of cloud data warehouses in terms of "performance per dollar" rather than just speed, assisting users in selecting systems that offer the highest performance for real-time analytics workloads. While traditional benchmarks focus on query execution speed, CostBench emphasizes the inseparability of speed and cost, highlighting that operational costs can significantly affect system performance comparisons. The benchmark addresses the complexity of comparing cloud platforms due to their varied cost presentation models, such as credits and compute units, by focusing on how much computational resource is needed to complete workloads and the associated costs. CostBench evaluates cost performance on two fronts: read-side, measuring query performance per dollar, and write-side, assessing the efficiency of converting fresh ingest into queryable data. The initial release focuses on read-side performance, comparing major cloud data warehouses like ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift, using anonymized datasets and real-world queries. ClickHouse Cloud stands out as maintaining "fast and low-cost" performance even as data scales, outperforming competitors by a significant margin. CostBench's open and reproducible design allows users to validate claims and explore results, ensuring transparency in cost performance assessments.