October 2025 Summaries

This blog post delves into the application of log clustering with Drain3 and ClickHouse UDFs to transform raw application logs into structured data, achieving significant compression while maintaining queryability and reconstructability. The process involves identifying log templates to extract key fields into columns, demonstrated by achieving nearly 50x compression on application logs and over 170x on Nginx logs. The challenge with application logs lies in their inconsistent structure compared to the predictable formats of third-party systems like Nginx. The technique of log clustering is highlighted as a powerful method for detecting patterns in unstructured logs, which can then be stored efficiently in a columnar database. The post details implementing this process in ClickHouse, where user-defined functions (UDFs) allow running custom code, including Python-based log template miners like Drain3, directly within the database environment. Structured logs enhance troubleshooting by grouping similar events and detecting unusual patterns early. The blog acknowledges that while the compression gain with structured logs is modest compared to raw logs, the approach offers improved query flexibility by extracting key fields into columns. The experiment suggests further exploration into optimizing data types and sorting keys for different services to enhance compression ratios, pointing towards a potential future feature in ClickStack for automating log clustering at scale.

QBit is a new data type added to ClickHouse to enhance vector search performance by storing floats as bit planes, allowing users to adjust precision and speed at query time without altering the original data. This innovation addresses the limitations of traditional vector search methods, such as brute force and Approximate Nearest Neighbour (ANN) techniques like HNSW, which often require upfront parameter decisions affecting recall and performance. QBit enables dynamic exploration of the balance between accuracy and speed by reading only the required subcolumns to reconstruct data to the user-specified precision, thus reducing I/O and computation time while maintaining high accuracy. This flexibility is particularly beneficial in vector search applications, such as music recommendations and retrieval-augmented generation (RAG) for large language models, where traditional vector stores are not always ideal for handling structured data. Through examples and benchmark tests, QBit has demonstrated its ability to significantly speed up vector searches without compromising accuracy, even when float values are reduced to as few as five bits.

The blog post explores the integration of ClickHouse MCP Server with 12 popular AI agent frameworks, revealing that large language models vary significantly in handling ambiguous instructions. The author uses the OpenAI agents library for its superior tracing capabilities to analyze decision-making processes, particularly in determining the most popular GitHub projects monthly in 2025. By configuring an OpenAI agent with ClickHouse's SQL Playground, the study demonstrates tracing data loading into ClickHouse and visualization with HyperDX. The post delves into the tracing mechanism, from capturing and exporting traces to executing SQL scripts that populate ClickHouse tables, ultimately enabling detailed analysis of AI agent workflows. The article highlights the importance of observability in AI systems, providing insights into agents' reasoning and decision-making processes, which are crucial for debugging and benchmarking models. This comprehensive approach emphasizes the growing need for observability tools in AI, paralleling their established role in microservices.

Netflix processes an enormous 5 petabytes of logs daily using ClickHouse, managing 10.6 million events per second with sub-second query times, thanks to three key optimizations: faster log fingerprinting with generated lexers, improved serialization with a custom native protocol, and efficient sharding of tag maps. This system supports over 40,000 microservices, retaining logs from two weeks to two years, and handles 500 to 1,000 queries per second for debugging and monitoring. The architecture combines ClickHouse for speedy access to recent data and Apache Iceberg for cost-effective long-term storage, enabling near-instantaneous log searchability, often within 20 seconds. Achieving this required overcoming significant bottlenecks in ingestion, serialization, and querying, ultimately resulting in a highly interactive and scalable logging system. This allows Netflix to maintain smooth operations across 190 countries for over 300 million subscribers.

Columnar storage presents an efficient method for compressing and storing logs while maintaining fast and flexible queries, achieving up to 178x compression by structuring raw logs into columns, using optimized data types, and clustering similar values. Logs, unlike traces and metrics, are often unstructured and do not compress well, yet they contain vital historical data essential for debugging. By transforming logs into structured data, identifying variable message parts, and using suitable data types, logs can be compressed efficiently. Experiments with Nginx access logs demonstrated that even by structuring logs and using column-based storage, compression ratios improved significantly, reaching up to 92x. Further optimization through ordering data on disk based on selected columns with low cardinality and skewed distribution resulted in a compression ratio of 178x. However, when typical scenarios like time-based querying are considered, compression efficiency decreases, highlighting the impact of ordering keys on compression outcomes. Ultimately, columnar databases like ClickHouse offer a viable solution for achieving high compression rates, thereby improving I/O efficiency, query speed, and reducing storage costs, albeit with potential trade-offs in query performance.

ClickStack's new text-to-chart feature simplifies the process of visualizing observability data by allowing users to create charts through natural language descriptions, eliminating the need for menus or dropdowns. This innovation leverages large language models to transform text prompts into queries, generating immediate visual representations of logs, traces, and metrics. By integrating this feature into its high-performance, open-source observability stack that utilizes ClickHouse, ClickStack aims to democratize access and enhance user experience. While currently requiring an Anthropic API key for operation, future updates plan to support additional language model providers like OpenAI. The feature, designed for quick and intuitive use, is part of a broader effort to increase productivity and ease of use in data exploration, with more enhancements anticipated in the future.

GitLab's transition to ClickHouse for analytics has significantly enhanced its ability to handle massive data scales and deliver near-instant insights, transforming its analytics infrastructure into a real-time, scalable platform. Initially using Postgres for both transactional and analytical workloads, GitLab recognized the need for a specialized analytics solution as the demand for real-time, scalable user-facing analytics grew. ClickHouse emerged as the optimal choice due to its superior performance, scalability, and ease of deployment, supporting GitLab’s architectural requirements and handling large data volumes efficiently. This shift allowed GitLab to overcome previous performance bottlenecks, with queries that once took 30-40 seconds now resolving in under a second. By integrating ClickHouse, GitLab has improved features such as Contribution Analytics and SDLC trends, enabling advanced insights into engineering outcomes and AI adoption. The transition has also been strategically aligned with GitLab's broader move towards an event-driven, analytics-first platform, leveraging ClickHouse's capabilities for a seamless and scalable analytics experience across its SaaS, on-premise, and self-managed environments.

Following the successful ClickHouse Open House conference in San Francisco, the event took its roadshow to New York on October 7th, where ClickHouse customers like Capital One, Ramp, and Modal shared insights into their use cases and successes with ClickHouse technology. The conference highlighted four main use cases for ClickHouse: real-time analytics, observability, data warehousing, and AI/ML infrastructure, emphasizing its role in delivering interactive data experiences, efficient data management, and supporting large-scale analytics workloads. Capital One showcased how they achieved an 80% improvement in response times and halved infrastructure costs using ClickHouse. Ramp demonstrated the profound impact of migrating to ClickHouse for observability, resulting in significantly faster analytics queries. Modal Labs illustrated how they transformed their logging infrastructure to handle immense data volumes with ClickHouse, achieving instant search capabilities across half a trillion logs. The roadshow aims to continue connecting with data teams globally, with further stops planned in Sydney, Bangalore, and Amsterdam, and recorded sessions from the New York event have been made available for those unable to attend.

Lakehouses utilizing open table formats like Apache Iceberg and Delta Lake are transforming data management by combining the scalability and low cost of object storage with database-like semantics, making them viable for observability workloads. These formats provide structured, queryable data storage solutions that reduce duplication and vendor lock-in, while enabling schema evolution, snapshots, and catalogs. Although challenges persist, such as partitioning tradeoffs, metadata growth, and limitations of the Parquet file format, innovations like liquid clustering and the introduction of new file formats like Lance are addressing these issues. As open table formats mature, they promise to offer a cost-effective, scalable solution for massive telemetry datasets, particularly when integrated with systems like ClickHouse that provide fast ingestion and low-latency analytics. This evolution suggests a future where the strengths of databases and lakehouses converge, delivering open, efficient storage with the performance and manageability of specialized analytical engines.

The October 2025 ClickHouse newsletter highlights several key developments and events, including a newly redesigned text index, enhanced scaling request logging capabilities, and the introduction of the DataLakeCatalog engine in ClickHouse Cloud. The newsletter features Mayank Joshi, Co-Founder and CTO at Auditzy, who achieved significant performance improvements by migrating from PostgreSQL to ClickHouse. It also outlines various upcoming events such as the Amsterdam User Conference and training sessions on real-time analytics and observability at scale with ClickStack. Technical insights include Geocodio's migration from MariaDB to ClickHouse for handling billions of geocoding requests and Laravel Nightwatch's use of Amazon MSK and ClickHouse Cloud for high-performance observability. Additionally, the newsletter discusses modeling messy data for OLAP performance, building ClickHouse-powered analytics APIs, and various quick reads and video content showcasing ClickHouse's capabilities in different contexts.

ClickHouse, a SaaS database company, has introduced a new feature allowing users to export backups of their ClickHouse Cloud services to their own cloud storage, enhancing user data sovereignty and control. Users can now manage their own backup lifecycle, decide on retention policies, and store backups in various regions for added security and cost efficiency, though they must handle the deletion of backups themselves to avoid excessive storage costs. This feature is available on AWS, GCP, and Azure, requiring users to configure roles and permissions for backup operations. Unlike ClickHouse's default incremental backups, external backups are full backups, minimizing the risk of data loss but requiring more resources and time. The company is also planning to introduce cross-cloud backup capabilities, enabling backups across different cloud providers, which could offer additional economic and strategic benefits.

mpathic, a startup founded in 2021, aims to enhance patient safety and compliance in drug trials by leveraging AI to analyze therapy sessions, identifying issues before they impact the trials. Initially using Elasticsearch to manage its machine learning (ML) pipelines, the company faced challenges with inefficiencies and the inability to perform complex data operations, which hampered innovation and increased costs. Transitioning to ClickHouse improved mpathic's data processing capabilities significantly, allowing for faster and more efficient ML workflows by enabling in-database tasks that previously required additional compute resources. This switch not only accelerated data processing and reduced infrastructure management burdens but also enhanced data analysis, enabling the team to focus more on refining their ML models and less on system administration. As mpathic continues to evolve, it plans to expand its data analysis capabilities, using ClickHouse to further improve clinical trial outcomes and maintain model accuracy as therapy data patterns change.

The Model Context Protocol (MCP) is an open protocol standardizing interactions between applications and large language models (LLMs), which has become the standard for integrating systems with LLMs since its launch in late 2024. MCP servers, similar to REST APIs, define tools and resources for LLMs to interact with, allowing them to perform actions like running SQL queries or listing databases. There are now at least 12 major agent SDKs supporting MCP, each catering to different needs: Claude Agent SDK is security-focused, OpenAI Agents SDK emphasizes agent handoffs, CrewAI handles multi-agent workflows, LangChain offers ecosystem breadth, and Agno prioritizes minimal code and performance, among others. These SDKs facilitate the integration of MCP servers into applications, often handling complexities like authentication and tool discovery. MCP servers can be local or remote, with remote servers addressing operational concerns like rate limiting and authorization. The choice of agent SDK and LLM depends on the application's requirements and the developer's ecosystem preferences.

ClickHouse has expanded its capabilities to query Iceberg and Delta Lake tables directly through the DataLakeCatalog engine, enabling seamless integration with catalogs such as AWS Glue Catalog and Databricks Unity Catalog. This advancement allows users to connect ClickHouse to these catalogs, automatically detecting table formats and querying them instantly, enhancing ClickHouse's role as a high-performance lakehouse query engine. Through the ClickHouse Cloud platform, features like a highly parallel native Parquet reader, distributed cache layer, and stateless compute nodes provide fast, scalable querying across both native and external data sources. The DataLakeCatalog engine facilitates querying lakehouse data as native tables, supporting full Iceberg and Delta Lake compatibility, including schema evolution and catalog integration. Demonstrations with AWS Glue and Unity Catalogs highlight ClickHouse's ability to perform federated queries across different data sources, showcasing its potential for integrating and analyzing diverse datasets in a unified analytics environment. Future developments include support for Iceberg V3, write operations, and optimized file handling, positioning ClickHouse Cloud as a comprehensive solution for lakehouse data management and analysis.

Auditzy, a company founded in Mumbai in 2022, focuses on converting user interaction data from websites into actionable insights for various business teams, addressing both performance optimization and business impact. Initially, Auditzy's architecture using Postgres struggled with scalability and performance issues, such as query latency and complex maintenance tasks. In response, they transitioned to ClickHouse in 2023, following a successful proof of concept that demonstrated significant improvements in query speed, storage efficiency, and reduced technical debt. This shift allowed Auditzy to streamline operations and focus on new features, such as the Auditzy Copilot, which uses ClickHouse and LLMs to provide real-time, conversational analytics for users across an organization. The move to ClickHouse has not only enhanced the user experience and developer efficiency but also positioned Auditzy to innovate further in delivering dynamic and accessible analytics solutions.

Laravel Nightwatch, a first-party observability platform launched by the popular web framework Laravel, offers developers real-time insights into application performance, processing over one billion events daily with sub-second query latency. Built on AWS managed services and ClickHouse Cloud, Nightwatch integrates Amazon Managed Streaming for Apache Kafka and AWS Lambda to provide scalable, low-latency monitoring while preserving Laravel's signature simplicity and user experience. The platform's architecture enables it to ingest millions of JSON events per second and deliver real-time dashboards with analytical workloads managed by ClickHouse Cloud's advanced compression and query capabilities. With features like VPC peering and global server load balancing, the platform efficiently handles multi-region infrastructure challenges. Laravel Nightwatch's launch saw 5,300 user registrations and 500 million events processed on its first day, logging 760,000 exceptions in real time with an average dashboard request latency of 97 milliseconds. Future plans include expanding to more regions, enhancing data collection, achieving SOC 2 certification, and offering advanced monitoring to meet diverse compliance and insight needs, all while supporting a wide range of application sizes, from hobbyists to enterprises.

ClickHouse, a prominent player in real-time analytics and data warehousing, has announced an extension of its Series C financing, bringing new investors such as Citi Ventures and Insight Partners onboard, along with continued support from existing backers. The company has experienced rapid growth, surpassing 2,000 customers and significantly increasing its annual recurring revenue, with notable new clients including Hewlett Packard Enterprise and Meta. ClickHouse's popularity is especially strong among AI companies, and it was named to the 2025 Forbes Cloud 100 list, highlighting its innovation and growth in the private cloud sector. The company recently hosted its first user conference, OpenHouse, and continues to expand its leadership team and platform capabilities, with new products like ClickStack and ClickHouse Private, which cater to various data processing needs and regulated environments. CEO Aaron Katz emphasized the importance of the extended financing in accelerating ClickHouse's mission to provide fast analytics solutions, supported by endorsements from investors and customers who recognize its transformative impact on data-driven AI initiatives.

ClickHouse v25.8 introduces significant improvements to its JSON data handling, enhancing performance for analytics over JSON data. The update builds on the JSON data type introduced in v24.8, addressing inefficiencies when dealing with large numbers of unique JSON paths. Two new serialization formats, bucketed shared data and advanced shared data, aim to improve query performance by efficiently managing how JSON paths are stored and accessed. The bucketed format divides shared data into manageable buckets, while the advanced format uses a complex structure to allow selective reads of JSON subcolumns without unnecessary data loading. Although these formats enhance performance, especially for selective reads, they necessitate a trade-off with increased storage requirements due to the need to store data in both the new and original formats. Benchmark tests demonstrate substantial gains in query speed and memory usage, particularly for datasets with tens of thousands of unique paths, making ClickHouse a more robust choice for handling large-scale JSON data.

Datavations, a New York-based company founded in 2020, has developed a comprehensive data and optimization platform tailored for the home improvement industry, aiding manufacturers in inventory and pricing decisions to secure retail shelf space. The platform relies on a proprietary web aggregation system that collects extensive store-level data, which is then processed by machine learning algorithms to align closely with actual sales metrics across numerous categories. Initially, Datavations used a standard data stack involving S3, Databricks, and Tableau, but faced challenges with cost and performance as the company expanded. To address these issues, they transitioned to a ClickHouse-powered architecture that streams data through Amazon Kinesis Firehose and separates processing and analytics tasks across dedicated ClickHouse instances, significantly reducing costs and improving speed and data quality. This new setup not only optimizes data storage and query performance but also enables real-time analytics and exploratory data analysis without disrupting production workflows. Looking ahead, Datavations plans to migrate to ClickHouse Cloud for enhanced parallel processing capabilities and is exploring AI-native workflows using ClickHouse's MCP server, aiming to further innovate and enhance their data-driven services for clients.

ClickGems is a newly launched analytics platform developed by the Ruby Central team, in collaboration with ClickHouse and Metabase, to provide detailed insights into the download statistics of over 200,000 Ruby gems from 2017 onwards, totaling over 200 billion downloads. Built on the same code base as ClickPy, ClickGems allows Ruby developers to explore global statistics and detailed data visualizations, such as downloads over time and by country, using SQL queries powered by Materialized Views in ClickHouse for fast execution. The platform also supports shareable charts, enabling users to embed analytics into websites or documentation, and operates on a unified ClickHouse Cloud infrastructure that separates compute resources to ensure efficient handling of large data volumes and queries without impacting the performance of other services like ClickPy. This initiative demonstrates a collaborative effort to enhance open-source analytics by providing a user-friendly, scalable tool for the Ruby community.

ClickHouse version 25.9 introduces several new features, performance optimizations, and bug fixes, including automatic global join reordering, streaming for secondary indices, and a new text index. The global join reordering feature allows for more efficient execution of complex join operations by optimizing the order of joins based on the size of the tables involved, significantly improving query runtime and reducing memory usage. Additionally, the release enhances the performance of secondary indices by enabling streaming, which concurrently checks index entries while reading data, reducing startup delays and inefficiencies. The update also includes a new experimental text index for more efficient full-text search and several other improvements such as data lake support, a new arrayExcept function, and the ability to define boolean settings without specifying arguments. These advancements contribute to making ClickHouse more efficient and versatile for complex data processing tasks.