September 2024 Summaries

ClickHouse, a high-performance columnar database management system, combined with Estuary Flow, a robust data integration platform, forms a real-time analytics platform that can ingest and transform data from hundreds of sources. The integration is facilitated by Estuary Flow's Dekaf, which allows consumers to read data from Estuary Flow collections as if they were Kafka topics. This enables ClickHouse users to directly ingest data from various real-time sources supported by Estuary Flow without the need for intermediate systems. The combined technologies offer simplified data ingestion, delivery guarantees, scalability, reduced complexity, and broader source support.

ClickHouse is a robust enterprise security program that involves systematic collection, storage, and analysis of log data from various systems, applications, and devices within an organization's IT environment. Choosing ClickHouse as the underlying technology for security information and event management (SIEM) was an obvious choice due to its ability to support any data format from any source, provide alerts and visualizations through the official ClickHouse Grafana Plugin, and effectively compress logs while preserving fast query access times. RunReveal is a security data platform designed to collect raw logs from SaaS and Cloud Services, ingest them into ClickHouse Cloud, and automatically detect anomalies and compromises in an organization's environment. It simplifies the process of adding new data sources, integrates tightly with log providers, and allows users to view and edit SQL queries that make up detections. The security applications of ClickHouse have become crucial components of organizations' security programs, enabling quick and efficient responses to security incidents.

Epilot, an energy sector SaaS platform, faced challenges with its existing data infrastructure as it scaled. The company initially used AWS Redshift and S3 for data storage and processing but experienced limitations in performance and real-time analytics. To address these issues, epilot switched to ClickHouse as their central data storage layer, offering improved performance, scalability, cost efficiency, and flexibility. The implementation of ClickHouse led to significant improvements in query performance, real-time analytics, and data consolidation, enabling better service delivery and customer insights. Looking ahead, epilot plans to expand its use of ClickHouse further, leveraging it for AI and machine learning applications as part of their future growth strategy.

ClickHouse has introduced API endpoints that allow developers to expose secure HTTP endpoints which consume parameters and use them to populate and execute predefined SQL queries. This feature simplifies interfaces, adds separation of concerns, and makes it easier for teams to expose analytics without needing to write SQL or interact directly with a ClickHouse database owned by different teams. The text provides an example of how API endpoints can be used in conjunction with the ClickPy application to add new GitHub analytics features in just a few minutes.

The September ClickHouse newsletter highlights various updates in real-time data warehouses. Key points include the introduction of a JSON data type, the first ClickHouse research paper presented at VLDB 2024, and a private preview of BYOC on AWS. Additionally, the newsletter features upcoming events, user stories, and quick reads related to ClickHouse applications and improvements.

The blog post discusses how ClickHouse's internal data warehouse (DWH) has evolved over the past year to support a more diverse set of users, data sources, and access points. It highlights the use of ClickHouse and dbt as primary components in the stack that have enabled DWH to support real-time data processing into regular batch reporting. The post also covers how the architecture of DWH has been configured with nineteen raw data sources, handling 6 billion rows and 50 TBs of data daily. It mentions how dbt centralized transformation logic related to batch reporting in one place, making it easier to manage growing complexity as SQL became the way business logic was encoded. The post also discusses incorporating more real-time data into DWH and configuring additional access points for users. Finally, it outlines future plans for scaling DWH by decentralizing compute resources and exploring AI features.

The first ClickHouse research paper has been accepted and published at the prestigious VLDB conference, which focuses on data management. This year's conference marked its 50th anniversary and featured over 250 paper presentations and accompanying workshops. The ClickHouse paper provides a comprehensive overview of the database's architecture, storage layer, query processing layer, integration layer, benchmarks, and future plans. Alexey Milovidov, CTO and creator of ClickHouse, presented the paper at VLDB 2024 in Guangzhou, China, followed by a Q&A session.

ClickHouse version 24.8 introduces several new features, including a revamped JSON type for storing semi-structured data, a table engine for time-series data, exactly-once processing of Kafka messages, and join improvements. The release is an LTS (Long Term Support) one, which means it will be supported for 12 months after its release. Additionally, the ClickHouse community welcomes new contributors to the project.