March 2024 Summaries

The March ClickHouse newsletter highlights the latest developments in real-time data warehouses. Key updates include the release of version 24.2 with improved data ingestion features, integration with Rill dashboards for faster operational reporting, and a new challenge to query one trillion rows of data. Additionally, chDB has joined the ClickHouse family as an embedded SQL OLAP engine, and upcoming events are listed for community engagement.

The article discusses the shift from schema-less technologies to structured data processing methods, driven by the need for efficiency and organization in managing databases. It highlights the challenges faced with schema management and introduces Database Schema-as-Code tools like Atlas as a solution. These tools simplify schema operations, offering a declarative API that allows users to define their desired database state and automatically generate migration plans. The author demonstrates how Atlas works in practice using ClickHouse as an example.

ClickHouse version 24.2 introduces 18 new features, 18 performance optimizations, and 49 bug fixes. Among these updates are automatic detection of file format, pretty format for large numbers, security enhancements for views, vectorized distance functions, and adaptive asynchronous inserts. These improvements aim to provide better user experience, increased efficiency, and more accurate results when using ClickHouse.

Today, we are excited to announce that chDB, an embedded SQL OLAP engine powered by ClickHouse, is now part of ClickHouse. ChDB was created with the goal of using ClickHouse as an "out of the box" Python module and allows users to perform in-memory analysis of data from various formats and sources supported by ClickHouse. With chDB's creator and main contributor, Auxten, joining forces with us, we will focus on evolving chDB and integrating it more closely with the ClickHouse ecosystem. This collaboration will result in better performance, integration with more programming languages and data sources, and a seamless experience for users analyzing data locally or remotely.

The text discusses the successful completion of a challenge set by Gunnar Morling from Decodable, which required users to write a Java program to compute each city's minimum, average, and maximum temperatures from a text file containing 1 billion measurements. Using ClickHouse, the response time was around 19 seconds using the exact hardware profile stipulated by the rules. The challenge was later extended to 1 trillion rows by Dask, prompting the authors to attempt querying this larger dataset. They achieved this in under 3 minutes for $0.56 using spot instances in AWS and a ClickHouse cluster. The text also provides details on how they optimized their settings and queries to achieve these results.