August 2025 Summaries

ClickHouse's CI/CD system relies heavily on AWS Lambda functions, with a preference for deploying them as ZIP archives due to their lighter weight and ease of automation. Since late 2023, Terraform/OpenTofu has been used to manage their CI/CD infrastructure, raising questions about artifact management. Key challenges encountered include maintaining consistent file orders within ZIP archives, handling file timestamps to prevent unnecessary redeployments, excluding Python byte-code files from the archives, and dealing with OS-dependent variations in ZIP format. To address these issues, ClickHouse employs Python's zipfile module for consistent ZIP creation and builds virtual environments aligned with AWS Lambda's Python version, ensuring dependency management and reducing unnecessary files. The solutions have been integrated into their Terraform/OpenTofu configuration for efficient Lambda function deployment.

ClickHouse has been recognized with the AWS Advertising and Marketing Technology competency, highlighting its expertise in managing high-performance analytics for advertising and marketing platforms. The company addresses the industry's challenges such as processing millions of events per second and executing complex queries on vast datasets by using its columnar storage, parallel execution engine, and domain-specific SQL functions. Companies like Braze and Cognitiv utilize ClickHouse for significant enhancements in their real-time analytics and machine learning infrastructure, respectively, achieving notable improvements in performance and operational efficiency. Braze, for instance, reported an 8x performance gain by transitioning to ClickHouse Cloud, while Cognitiv relies on it for optimizing real-time bidding algorithms. ClickHouse's capabilities also empower marketing automation platforms like Klaviyo, enabling sophisticated customer journey tracking and real-time segmentation. This recognition underscores ClickHouse's continued commitment to innovation in the marketing and advertising technology ecosystem, helping organizations turn data into a competitive advantage.

The text discusses how Sierra, a platform for enhancing customer experiences with AI, is integrating observability and analytics through the use of ClickHouse, a database known for its real-time ingestion and speed. Traditionally, observability and analytics have been separate, with different tools and priorities; observability focuses on system health and real-time responses, while analytics delves into long-term trends and user behavior. Arup Malakar, a software engineer at Sierra, argues for a unified approach that treats all data as one "event stream," bridging the gap with ClickHouse’s capabilities like columnar storage and vectorized query execution. This integration allows Sierra to retain full data context without sacrificing speed, enabling a comprehensive view that informs both technical and business decisions. Sierra’s use of ClickHouse facilitates real-time alerts, connecting performance metrics with customer interactions, and is transforming how teams understand and utilize data, ultimately aiming to dissolve the divide between observability and analytics.

The text outlines an innovative approach to business intelligence (BI) that integrates a chat-native BI layer within Slack, utilizing ClickHouse and MCP to streamline data analysis and visualization directly in conversation threads. This method shifts the traditional BI focus from dashboards to conversations, allowing users to ask questions in natural language and receive both visualizations and SQL queries in response. The process leverages a Python bot to generate Vega-Lite visualizations, making it easy to create and share mini-reports within Slack without requiring users to learn complex BI tools or SQL syntax. This approach is designed to complement rather than replace traditional BI dashboards, offering a more fluid and rapid investigative tool for exploratory data analysis. The system's architecture benefits from ClickHouse's speed and scalability, while MCP ensures robustness and portability, allowing for seamless integration across different platforms. Despite limitations in complex visualizations, this practical solution enhances productivity by reducing friction in data-driven discussions, emphasizing its utility in exploratory and conversational analytics.

ClickHouse Cloud's ClickPipes has introduced new scaling options to enhance flexibility and efficiency in managing diverse data ingestion workloads, addressing customer requests for more adaptable configurations. The platform now supports both vertical and horizontal scaling for streaming ClickPipes, allowing users to adjust the number and size of replicas deployed, thereby optimizing for cost and performance. Each replica operates as a consumer for Kafka or Kinesis streaming data sources, enabling high-throughput and scalable data ingestion with fault tolerance and efficient load balancing. Enhanced resource monitoring displays CPU and memory usage, aiding in informed scaling decisions. The new default setting for replicas is Extra Small, with a flexible pricing model starting at $0.0125 per hour, reflecting the adjusted cost strategy aligned with replica size and number. This advancement allows customers to better manage data ingestion workloads, maintaining performance while controlling expenses.

MooseStack by 514 is an open-source developer toolkit designed to enhance the developer experience (DX) for building data infrastructure with ClickHouse, addressing the convergence of data and software engineering. Traditionally, data infrastructure catered primarily to analysts with SQL-first workflows, but MooseStack shifts this approach to empower both data and software developers. It draws inspiration from modern web development practices, emphasizing a local-first, git-native approach with integrated CI/CD, and the use of native programming languages like TypeScript and Python instead of configuration files like YAML. This toolkit provides abstractions for infrastructure boilerplate, modular integration, and open-source flexibility, supporting the development of real-time analytics and AI-powered features. By offering a robust DX layer, MooseStack bridges the UX and DX gaps in data systems, enabling engineers to create efficient, scalable analytics applications with confidence and agility, akin to modern software development lifecycles.

Laminar, an open-source observability and evaluation platform founded by Robert Kim and Din Mailibay in 2024, focuses on enhancing the management and debugging of browser-based AI agents by providing a unique type of observability known as browser agent observability. The platform captures the full visual traces of AI agents' sessions by recording DOM diffs, allowing developers to see exactly what the agents see and do without the burden of slow, raw video recording. Laminar uses ClickHouse Cloud as its real-time data warehouse, chosen for its speed, flexibility, and capability to handle high-throughput data without latency, allowing Laminar to process over 500,000 browser events daily and more than a billion events in total since its launch. The platform's use of ClickHouse ensures a smooth user experience with rapid data retrieval and efficient storage, as evidenced by their ability to quickly replay traces and manage large data volumes. With the growing prevalence of browser-based agents in AI and automation, Laminar aims to become the default observability layer, leveraging ClickHouse's scalability and their optimized Rust backend to support an expanding user base and increasing data demands.

Character.AI, a rapidly growing AI platform, overcame its fragmented logging infrastructure by adopting a centralized observability stack built on ClickHouse and ClickStack, enabling efficient monitoring of its massive data scale. Mustafa Yildirim, the first Site Reliability Engineer at Character.AI, spearheaded the transition, which involved architectural decisions, schema optimizations, and ingestion strategies to handle over 450PB of log data monthly. This shift resulted in faster log search times, improved visibility, and significantly reduced costs. The implementation of ClickStack, following ClickHouse's acquisition of HyperDX, provided a modern user interface, fast query performance, and efficient data compression, allowing Character.AI to process 10 times more data while spending half the previous cost. By leveraging features like real-time log tailing, denoise, and pattern-based event grouping, the team enhanced root cause analysis and issue resolution. Looking forward, Character.AI plans to streamline its observability pipeline further by introducing a centralized gateway for log processing and integrating metrics into the same platform to enable comprehensive correlation and alerting capabilities.

ClickStack has introduced several updates to enhance its open-source observability stack for ClickHouse, aiming to improve user experience, performance, and efficiency. The latest release includes the availability of the HyperDX component in ClickHouse Cloud, which simplifies adoption for cloud users and integrates with ClickHouse Cloud's passwordless authentication. New features such as smarter search, dynamic visualizations, custom chart aliases, pinned fields, and support for the "any" aggregation function aim to streamline data analysis and visualization processes. Additionally, improvements in query efficiency for time-based primary keys and the introduction of auto-correlated sources facilitate better data management and exploration. Significant advancements in ClickHouse's inverted indices promise to enhance full-text search capabilities, while the newly configurable search limit offers greater flexibility in handling large datasets. These developments are part of ClickStack's vision to unify observability, real-time analytics, and data warehousing into a single platform.

Artly, a Seattle-based startup, is revolutionizing the coffee industry by developing robotic baristas capable of delivering high-quality coffee with the precision and consistency of industrial automation. These fully autonomous machines, trained by renowned barista Joe Yang, can create a wide range of custom drinks, including specialty espresso beverages and latte art, while addressing the challenge of scalability that independent coffee shops face. Key to their operation is a sophisticated data infrastructure that processes telemetry and sales data in real time, allowing Artly to maintain consistent quality across multiple locations. Initially using Elasticsearch, the company transitioned to ClickHouse for its superior flexibility, speed, and cost-effectiveness. ClickHouse's capabilities enable Artly to efficiently analyze data, track customer preferences, and optimize operations, supporting their expansion efforts and enhancing customer personalization without sacrificing the artisanal quality of their coffee.

ClickPipes, part of the ClickHouse team, is developing high-performance connectors to transfer data from various sources, including Delta Lake, to ClickHouse. The team has built Change Data Capture (CDC) connectors for databases like Postgres, MySQL, and MongoDB and is now focusing on implementing CDC from data lakes, starting with Delta Lake. A reference implementation of this process is available as open-source and uses Python to move data from Delta Lake to ClickHouse. Delta Lake provides a transactional storage layer on object storage, effective for handling large volumes of data, while ClickHouse is optimized for fast analytical queries. The combination of these technologies allows for efficient data replication and real-time data access. The process involves using Delta Lake's Change Data Feed (CDF) to capture changes and ClickHouse's ReplacingMergeTree table engine to model data changes. While the current implementation is not production-ready, it lays the groundwork for a robust CDC pipeline, with plans to integrate it into ClickPipes in the future. The implementation demonstrates the potential for real-time analytics by utilizing Delta Lake's CDF and ClickHouse's capabilities, although challenges remain, such as handling schema evolution and supporting delete operations.

The discussion centers on the potential benefits and challenges of applying Object-Relational Mapper (ORM) principles to Online Analytical Processing (OLAP) databases, particularly ClickHouse. While transactional ORMs like Prisma and SQLAlchemy have been successful in the Online Transaction Processing (OLTP) realm, their extension to OLAP is fraught with difficulties due to fundamental differences between the two database types, such as data storage methods and query complexity. Moose OLAP, part of the open-source MooseStack, seeks to address these challenges by offering an ORM-like interface specifically tailored for OLAP workloads. This involves creating OLAP-native semantics that reflect the unique characteristics of OLAP databases, such as column-orientation and append-only operations, while borrowing beneficial ORM concepts like schema as code for version control and developer ergonomics. The article argues for a distinct OLAP modeling API, rather than extending existing OLTP ORMs, and emphasizes the need for a developer experience that balances simplicity with access to the full power of OLAP databases. Moose OLAP aims to provide a more developer-friendly experience for managing complex analytical queries and schema migrations, without compromising the capabilities that make OLAP databases valuable.

ClickHouse has been completely re-engineered to offer a faster, leaner, and fully integrated full-text search (FTS) feature, optimized for its columnar database design. By transitioning from a Bloom-filter-based approach to a native inverted index, ClickHouse now supports high-performance search capabilities directly within the database. The new design includes advanced data structures like Finite State Transducers (FSTs) for space-efficient dictionaries and Roaring bitmaps for fast, compressed posting lists. These innovations enable efficient querying without reading the text column, significantly improving speed and reducing resource usage. Recent updates have introduced a more intuitive API, new tokenizer options, improved compression techniques, and enhanced compatibility with ClickHouse Cloud. Additionally, the system now supports fast row-level searches using the index alone, resulting in considerable performance gains, especially for frequent search terms. The revamped full-text search is available for a private preview, inviting early adopters to test and provide feedback.

ClickHouse and PostgreSQL, two leading open-source databases, showcase distinct strengths in update performance due to their differing architectures. ClickHouse, primarily an OLAP database, demonstrates remarkable speed in bulk updates—up to 4,000 times faster than PostgreSQL—thanks to its columnar design and parallel processing, making it ideal for large-scale data manipulations. In contrast, PostgreSQL, an OLTP-oriented database, excels in transactional integrity and performs well on single-row updates due to its B-tree indexing, offering a more traditional approach to data durability and transaction handling. Despite their differences, both databases utilize Multi-Version Concurrency Control (MVCC) to ensure consistent data snapshots without blocking concurrent operations. While ClickHouse lacks the full transactional capabilities of PostgreSQL, its recent advancements in SQL-standard updates position it as a formidable option for analytical workloads and even certain OLTP scenarios. Additionally, ClickHouse's ability to quickly apply changes and support immediate analytical queries highlights its potential in environments where fast data processing and retrieval are crucial.

AWS Glue, Amazon's serverless data integration service, now offers an official ClickHouse Connector, available in the AWS Marketplace, which simplifies the process of working with ClickHouse using Apache Spark-based ETL jobs. This connector allows users to work with PySpark or Scala within AWS Glue environments by eliminating the need for manual installation and management of the ClickHouse Spark connector. The connector is designed to work with AWS Glue version 4, supporting Spark 3.3, Scala 2, and Python 3, and can be configured with various job parameters for different environments. In addition to writing and reading data between Spark DataFrames and ClickHouse, the connector supports executing DDL operations within Spark SQL, enabling users to create and manage database tables seamlessly. The blog also discusses setting up IAM roles, configuring job parameters, and optimizing Glue jobs for production use, along with providing insights into the connector's future roadmap, such as supporting AWS Glue's no-code interface and enhanced IAM role integration.

The text outlines an experiment conducted by ClickHouse to evaluate whether AI-powered observability, specifically through large language models (LLMs), can effectively replace Site Reliability Engineers (SREs) in performing root cause analysis (RCA). Despite the potential of LLMs like Claude Sonnet 4, OpenAI's GPT models, and Gemini 2.5 Pro, the study concluded that these models are not yet capable of autonomously identifying root causes in complex, real-world scenarios without guidance, even though they can assist with documentation tasks such as drafting RCA reports. The experiment revealed that while models could sometimes pinpoint issues, their performance was inconsistent, largely due to a lack of context and domain specialization. Furthermore, the unpredictability in token usage and cost presents challenges for integrating LLMs into automated observability workflows. The study suggests that the current best approach is a collaborative one that combines human engineers with fast, scalable observability tools and LLMs for supportive tasks, allowing for more efficient and accurate incident resolution.

The August 2025 ClickHouse newsletter highlights several advancements and community activities, including the release of ClickHouse 25.7, which offers faster SQL UPDATE and DELETE operations and AI-powered SQL generation. Gene Makarov is featured for his innovative approach to rendering vector tiles from ClickHouse for mapping engines. Upcoming events cover virtual and in-person training sessions across various regions, focusing on topics like observability and query optimization. A SQL analytics bake-off compares the performance of Claude Opus and Gemini 2.5 Pro models in generating SQL queries from plain English. The newsletter also discusses technical insights such as ClickHouse's solutions for fast row-level updates, high-performance dashboard analytics, and building LLM observability with ClickStack, OpenTelemetry, and MCP. Additionally, it includes quick reads on topics like generating Parquet files, optimizing ClickHouse queries, and constructing real-time analytics pipelines.

The announcement introduces the private preview of the MongoDB Change Data Capture (CDC) connector in ClickPipes, enabling seamless replication of MongoDB collections to ClickHouse Cloud for enhanced analytics. This integration, powered by the open-source PeerDB, supports both continuous replication and one-time loads from MongoDB, whether on Atlas or self-hosted instances, without the need for external ETL tools. The connector is tailored to handle MongoDB's document-oriented architecture, efficiently mapping JSON data into ClickHouse's analytics engine, and offers features like advanced JSON support, real-time change streams, and built-in monitoring. This collaboration between MongoDB and ClickHouse aims to balance operational flexibility and analytical performance, allowing MongoDB to manage transactional workloads while ClickHouse handles complex analytical queries. Users interested in early access can sign up for the private preview, which is free and includes $300 in credits for a trial period, with options for continued use based on different pricing plans.

As open-source LLM chat applications like LibreChat, AnythingLLM, Open WebUI, and Chainlit evolve, the adoption of the Model Context Protocol (MCP) is becoming crucial for developers aiming to create data-rich, tool-augmented AI experiences. LibreChat stands out with its enterprise-grade features and full native MCP integration, making it ideal for complex multi-user environments. AnythingLLM excels in document-based Retrieval-Augmented Generation (RAG) workflows due to its straightforward setup and organization capabilities. Open WebUI, though lacking native MCP support, uses a proxy approach to convert MCP servers into REST API endpoints, maintaining adaptability with OpenAI's API standards. Chainlit offers the most flexibility for developers with its code-first approach, requiring Python programming to build custom applications. Each platform provides varying degrees of usability, flexibility, and MCP support, allowing users to integrate real-time data workflows effectively, with ClickHouse MCP Server as a practical example of connecting chat interfaces to live data sources.

ClickHouse version 25.6 introduces a host of updates, including 27 new features, 26 performance optimizations, and 98 bug fixes, while welcoming new contributors to the community. Notable enhancements include the introduction of lightweight SQL UPDATE and DELETE statements that use a patch-part mechanism, significantly boosting performance by applying updates instantly without impacting query performance. The update also features AI-powered SQL generation, improved aggregation processes, and optimizations for JOIN operations, all aimed at enhancing speed and efficiency. Additionally, native support for Geo Parquet types and new geospatial functions have been added, alongside security enhancements that allow for more flexible user provisioning and granular permission settings. These improvements collectively enhance ClickHouse's functionality and efficiency, maintaining its position as a popular choice for data processing and analytics.

ClickStack, an open-source observability stack built on ClickHouse, is now available in a private preview on ClickHouse Cloud, offering a streamlined, integrated experience for users. Launched to replace fragmented and costly observability stacks, ClickStack is designed to deliver high performance and efficiency through its components: OpenTelemetry Collector for data ingestion, ClickHouse for data storage, and HyperDX as the user interface. The integration into ClickHouse Cloud simplifies access by eliminating the need for separate infrastructure and authentication, while allowing users to manage observability data alongside business analytics seamlessly. This marks a significant step towards the unification of real-time analytics, data warehousing, and observability, presenting a future where these domains are integrated into a single platform. The move aims to address the limitations of proprietary SaaS solutions by enabling high-cardinality joins and SQL-powered investigations, ultimately positioning ClickHouse as a comprehensive solution for data-driven insights. The release is a precursor to a multi-tenant, fully managed version of ClickStack, promising cost-efficiency, performance, and openness without the operational overhead.

ClickHouse, a columnar database known for its high performance in analytical queries, has introduced standard SQL UPDATE capabilities that leverage lightweight patch-part updates to significantly enhance speed, achieving up to 1,000× faster performance than traditional mutations. This innovation is part of a series exploring fast updates in ClickHouse, detailing how the system efficiently implements row-level changes using engines like ReplacingMergeTree and CoalescingMergeTree, and now, through declarative SQL-style UPDATEs that utilize patch parts. Extensive benchmarking reveals ClickHouse's new SQL UPDATEs not only match PostgreSQL in point updates but can be up to 4,000× faster in bulk changes. By storing only the differences in small, quickly applied patches, these updates maintain query speed and are highly suitable for high-frequency, small-scale changes, contrasting with the slower, full-column modifications of classic mutations. The benchmarks, conducted on a large-scale dataset with reproducible scripts, demonstrate ClickHouse's capability to handle update patterns traditionally seen in row-store databases, all while maintaining its core analytical strengths.

Klaviyo, a B2C CRM platform used by major brands like Glossier and Vans, is revolutionizing customer engagement through advanced real-time segmentation and personalization, facilitated by the adoption of ClickHouse. Initially, Klaviyo faced challenges with their Python-based segmentation engine, which was slow and costly for large datasets due to its reliance on MySQL and Cassandra. By transitioning to ClickHouse, they significantly reduced processing times from hours to seconds, allowing efficient handling of billions of updates and segment changes daily. The new system employs bi-level sharding and materialized views to optimize data processing, enabling targeted marketing strategies to be executed swiftly. This transformation not only saved infrastructure costs but also enhanced scalability and performance. Additionally, Klaviyo sees future potential in ClickHouse Cloud for more flexible scaling and in new features like the Model Context Protocol for AI-driven insights, aiming to make customer data more actionable.

ClickHouse has announced the general availability of HIPAA and PCI self-service deployments within ClickHouse Cloud, providing customers with enhanced data security management capabilities. This development allows users to handle electronic protected health information (ePHI) and payment card data securely and efficiently, adhering to HIPAA and PCI DSS standards. ClickHouse's compliance program incorporates the HIPAA Security Rule and PCI DSS Level 1 Service Provider requirements, ensuring robust administrative, technical safeguards, and stringent data protection. The service includes specialized environments with enhanced isolation, fortified access controls, and granular auditing, supported by an independent audit confirming SOC 2 Type II + HIPAA and PCI compliance. Customers can easily access Business Associate Agreements and activate PCI compliance through the console, streamlining the deployment of secure services across certified cloud providers and regions.