April 2025 Summaries

APIs are a critical component of modern digital ecosystems, but they also create vulnerabilities that attackers can exploit. Wallarm, a company founded in 2016 to help enterprises prevent and protect against API breaches, faced challenges scaling its infrastructure to deliver real-time threat detection. After migrating from a self-hosted ClickHouse deployment to ClickHouse Cloud, Wallarm was able to overcome latency issues and improve performance, delivering near-instant results and allowing customers to respond to threats without delay. The migration also enabled the company to reduce costs and focus on innovation rather than day-to-day maintenance. With the growing threat landscape driven by AI adoption, Wallarm is well-positioned to tackle today's threats while staying ahead of new risks that come with agentic AI adoption.

Flock Safety, a company that provides automated license plate readers, video cameras, and gunfire detection systems to help deter crime and aid criminal investigations, has transitioned its data architecture from Amazon Redshift to ClickHouse Cloud. The new architecture allows for real-time analytics and data updates, eliminating the need for daily refreshes and reducing downtime. With ClickHouse's AggregatingMergeTree tables and materialized views, Flock Safety can now process large volumes of data quickly and efficiently, providing customers with 24/7 access to the latest data. The transition has also enabled long-term data retention, giving customers richer insights and deeper trend analysis. By switching to ClickHouse Cloud, Flock Safety aims to provide its customers with real-time insights that help them make smarter, data-driven decisions, ultimately contributing to safer communities.

AgentHouse is an interactive demo environment that showcases the powerful combination of ClickHouse's real-time analytics capabilities with large language models. It was built as a testing ground for users to explore how ClickHouse can serve as an ideal backend for LLM applications, featuring Anthropic's Sonnet model and LibreChat UI project. AgentHouse provides access to multiple public datasets, allowing users to delve into various use cases through a simple conversational interface. The demo environment includes features such as test natural language queries, experience real-time analytics, try interactive data exploration, view automated visualizations, and more, making it an ideal platform for users to interact with ClickHouse and large language models seamlessly.

ClickHouse Cloud has introduced several new features and improvements to enhance its capabilities, including Bring Your Own Cloud (BYOC) on AWS, native Postgres Change Data Capture (CDC), a native MySQL CDC connector in private preview, Slack notifications for Cloud events, a Resource Utilization dashboard, a new region: AWS Middle East (UAE), Cross-Region Private Link in public beta, ClickPipes with AWS PrivateLink support, a Usage cost API endpoint, expanded user roles, PCI and HIPAA compliance, and plans to continue delivering more powerful features as summer approaches.

REA, a real estate analytics company, struggled with its MongoDB-based architecture due to large datasets and performance issues. They turned to ClickHouse, which offered improved query performance, faster table refreshes, and more flexibility in handling large-scale data updates. REA redesigned their pipeline to pull data directly into ClickHouse, reducing intermediate processing layers and improving efficiency. While ClickHouse has solved many of REA's challenges, there are some gaps in its support for historical dates and spatial data, as well as a lack of native support for automatically incrementing IDs. Despite these challenges, REA has seen significant improvements in query performance and pipeline efficiency with ClickHouse, enabling them to provide real-time insights to customers and scale their operations.

In this post, we’ll walk through how lazy materialization works and how it fits into ClickHouse’s broader I/O optimization stack. To give a complete picture, we’ll also briefly demonstrate the other key building blocks of I/O efficiency in ClickHouse, highlighting not just what lazy materialization does, but how it differs from and complements the techniques already in place.` This sentence provides an objective summary of the text, covering the main topic of the post (lazy materialization) and its relationship with other optimization techniques in ClickHouse. It highlights that the post aims to provide a complete understanding of lazy materialization by briefly introducing other key building blocks of I/O efficiency in ClickHouse.

This April 2025 ClickHouse newsletter highlights several key developments and updates, including the release of version 25.3, which introduces a query condition cache for improved performance in dashboarding and observability use cases. The newsletter also shares insights from CloudQuery's six-month experience with ClickHouse, highlighting its speed and scalability benefits. Additionally, ClickHouse has acquired HyperDX, a fully open-source observability platform, to strengthen its offerings. Other notable mentions include the integration of Rust into the ClickHouse codebase, improvements in scaling mechanics for ClickHouse Cloud, and various community-driven projects such as QuerySight and Anthropic's Model Context Protocol. The newsletter also features a spotlight on Julian LaNeve, CTO at Astronomer, who shares his experience with ClickHouse and its benefits for data observability.

Ruby developers can now access free gem analytics via sql.clickhouse.com in partnership with Ruby Central. The dataset includes over 180 billion rows of download logs since 2017, allowing for exploration of trends and adoption patterns across the Ruby ecosystem. The data is hosted on ClickHouse, a column-store database that enables efficient querying of large datasets. Users can query the data directly via SQL, or use pre-aggregated daily statistics provided by Ruby Central. Additionally, materialized views are available to accelerate common queries, such as finding the most downloaded gems or downloads over time for a specific gem version. The service is free and intended to support researchers, maintainers, and curious developers in understanding the Ruby ecosystem.

At 3 a.m., PagerDuty alerts trigger a flurry of activity as DevOps engineers scramble to diagnose issues, with logs, metrics, and traces scattered across multiple tools. To solve the fragmentation problem, Last9 built an observability platform that centralizes telemetry data, making it easier, faster, and more cost-effective to analyze. The company's telemetry control plane provides users with control over their data, allowing them to filter, reshape, and drop data in real-time without instrumentation or code changes. ClickHouse, a high-performance columnar database, is at the heart of Last9's platform, enabling fast query performance, scalability, and accessibility. By combining ClickHouse with AI-powered observability tools, Last9 aims to redefine observability and give teams real control over their telemetry data, leading to less guesswork, faster insights, and fewer 3 a.m. firefights.

The MySQL Change Data Capture (CDC) connector is now available in private preview for ClickPipes, allowing users to replicate their MySQL databases to ClickHouse Cloud with blazing-fast analytics. This connector supports continuous replication and one-time migration from MySQL, both on-premises and in the cloud, without requiring external ETL tools. The experience is natively integrated into ClickHouse Cloud through ClickPipes, eliminating the need for significant overhead to set up and manage external tools. The connector is powered by PeerDB, a fully open-source database replication project by ClickHouse, ensuring no lock-in and extending value to self-hosted ClickHouse users as well. Key features include blazing-fast initial loads, continuous replication, table and column-level filtering, schema changes, flexible replication modes, native data type support, MySQL version compatibility, in-place editing, enterprise-grade security, built-in metrics and monitoring, and a fully managed solution with high availability. Users can sign up for the private preview by filling out a form on the ClickHouse Cloud website, which will provide early access to this feature without any cost or delay.

The founders of Dash0, a company that provides an OpenTelemetry-native observability platform, have chosen ClickHouse as their primary storage solution. This decision was influenced by their experience with Instana, another company they founded, and their need for greater efficiency and scalability in handling large volumes of telemetry data. ClickHouse's columnar architecture, compression capabilities, and flexible schema design make it an ideal choice for storing high-cardinality telemetry data. The team has optimized ClickHouse's operational aspects to balance maintainability, performance, and cost efficiency. They have also implemented a hybrid storage approach combining AWS S3 storage with temporary local storage to manage large customer accounts and reduce storage costs. The ClickHouse database is used as the main storage for all OpenTelemetry data, with PostgreSQL handling customer-specific settings and related data. The team has carefully designed table schemas, including primary keys, aggregation strategies, and indexing techniques, to achieve both speed and storage efficiency. They have also implemented sampling strategies to improve query performance and have optimized JOINs to reduce memory consumption. Overall, the implementation of ClickHouse as the foundational storage layer for Dash0 demonstrates the power of a thoughtful database architecture in supporting high data volume and cardinality while maintaining performance at scale.

The creation of ClickHouse Cloud's MultiSTS approach, which allows for efficient scaling vertically, was driven by the need to balance velocity with stability in technical decision-making. The initial choice of using StatefulSets to control server pods provided advantages such as persistent volume claims and rolling restarts, but also introduced limitations that made it difficult to scale replicas dynamically. To address this, a new approach called Make Before Break (MBB) was developed, which involves adding capacity to the existing cluster before removing old replicas. This approach requires custom Kubernetes controllers and Temporal-based automation systems to manage the complex migration process. The implementation involved significant challenges, including issues with system tables, attachments, and partial maintenance mode, but ultimately resulted in a more elastic and resilient cloud infrastructure for ClickHouse users. With MBB, customers can now take advantage of vertical scaling without disrupting their production workloads, significantly improving scaling times and capacity availability.

Rust has been successfully integrated into the ClickHouse codebase, allowing for the development of new system components in Rust. The integration was done by rewriting some parts of the existing C++ code to use Rust and adding a few small Rust libraries initially. The first library added was the BLAKE3 hash function, which is implemented only in Rust at that time. After that, PRQL, another query language alternative to SQL, was integrated into ClickHouse. Delta Lake, an open data lake format from Databricks, was also integrated using a Rust library. The integration faced several challenges, including supply chain issues, complex wrappers for Rust and C++ interop, panics in Rust, sanitizers, cross-compilation, library linking, symbol sizes, composability, build profiling and caching, and dependency management. Despite these challenges, the integration was successful, and ClickHouse now has a growing community of Rust developers contributing to it.