clickhouse-local vs DuckDB on Two Billion Rows of Costs

In a comparison of clickhouse-local and DuckDB for analyzing over two billion rows of cloud infrastructure spending data, clickhouse-local demonstrated a significant speed advantage, being three times faster than DuckDB for both reading and writing operations. Despite its performance edge, clickhouse-local's user experience was noted to be less favorable, particularly in terms of saving tables locally and preserving query states across sessions. The analysis was conducted on a 16 GB MacBook Pro using gzipped CSV files, with data eventually loaded into each tool for querying. While DuckDB offers a more user-friendly experience, clickhouse-local's raw speed makes it appealing for handling large datasets, though the need to load data from disk every session can be impractical. The piece highlights the rapid innovation in database technologies like clickhouse-local and DuckDB, which provide valuable tools for data engineering tasks such as generating complex reports and visualizations.