July 2025 Summaries

ClickPipes is a solution designed to manage real-time data ingestion into ClickHouse Cloud, supporting various data sizes and topologies through object storage, message brokers, and databases. It has introduced new scaling options to enhance flexibility in configuring streaming workloads, allowing users to control both horizontal and vertical scaling by selecting the number of replicas and replica sizes. The system utilizes replicas that act as consumers of Kafka or Kinesis data sources, which can be scaled to match workload demands, enabling high-throughput, scalable ingestion with efficient load distribution. Users can configure these settings through a UI, OpenAPI, or Terraform, and monitor resource usage via enhanced monitoring tools to optimize the cost-performance ratio. Pricing now varies based on replica size and count, starting at $0.0125 per hour, with additional data ingestion costs, enabling users to better manage expenses according to their specific needs.

Blacksmith, founded in 2024, aims to enhance Continuous Integration (CI) processes by offering a faster and more observable alternative to GitHub Actions, serving over 600 organizations and handling 9 million monthly jobs. By providing features such as step-by-step visibility, caching insights, and global log search, Blacksmith helps developers diagnose and resolve CI issues more effectively. To support its advanced analytics and observability needs, Blacksmith transitioned from Postgres to ClickHouse Cloud, leveraging tools like ClickPipes and the Postgres CDC connector for real-time data streaming without complex ETL processes. This setup enables fast and flexible analytics, powering both engineering and go-to-market strategies. Blacksmith also benefits from ClickHouse Cloud's innovative features like Ask AI, which simplifies SQL querying, and the Model Context Protocol (MCP) server, enhancing accessibility and intelligent workflow development. As Blacksmith continues to grow, ClickHouse remains integral to its operations, offering a robust solution for modern companies looking to optimize CI and data analytics workflows.

SEON, a company focused on fraud prevention and anti-money laundering, collaborated with AWS Partner ClickHouse to optimize its database performance and scalability amid rapid growth. Co-headquartered in the US and Hungary, SEON provides services to sectors like financial services, ecommerce, and iGaming, using real-time digital footprint analysis and AI-driven rules to detect and prevent fraud. As transaction volumes increased, SEON faced challenges in maintaining efficient processing times, particularly for enterprise clients with complex rule sets. To address this, SEON explored ClickHouse's open-source database and implemented it as a managed service on AWS, resulting in significant performance improvements, including up to 80 percent faster database processing times and 30 percent reduced storage costs. The collaboration not only enhanced the company's fraud detection capabilities but also impressed clients with lower latencies and efficient handling of massive web traffic scenarios, ultimately reinforcing SEON's confidence in using AWS's advanced technology and customer service.

The text discusses the evolution of SQL-style UPDATE operations in ClickHouse, a columnar database, which traditionally faced challenges in efficiently handling updates due to its architecture prioritizing read speed over write operations. Initially, ClickHouse employed a mutation model that was reliable but heavy on resources, involving full-column rewrites and delayed visibility of changes. To address these limitations, ClickHouse introduced on-the-fly updates for immediate visibility and later developed "patch parts," a novel mechanism allowing lightweight, efficient updates by treating updates as inserts. Patch parts apply changes during background merges, leveraging the existing high-speed insert and merge capabilities of ClickHouse, ensuring minimal overhead and instant visibility of updates. This approach not only maintains the database's performance but also integrates seamlessly with SQL syntax, providing users with the ability to perform updates without sacrificing speed or scalability. The improved update mechanisms significantly outperform traditional methods, with benchmarks showing up to a 1,000× increase in speed, marking a substantial advancement in handling updates within columnar storage systems.

Property Finder, a prominent real estate platform in the Middle East and North Africa, significantly enhanced its database performance and reduced operating costs by migrating to a central database solution using ClickHouse and Amazon Web Services (AWS). Headquartered in the UAE, the company faced challenges with decentralized systems affecting analytics performance and overprovisioned computing resources. After conducting a competitive proof of concept, Property Finder selected ClickHouse, which offered a blend of improved performance and cost savings, reducing operating expenses by 50% and improving database performance fivefold. The migration enabled the company to establish a single source of truth, enhancing user experience and providing faster database query responses, with some queries seeing a 96% improvement in speed. Utilizing AWS capabilities such as Amazon EC2, Amazon S3, and AWS PrivateLink, Property Finder optimized its infrastructure to meet data residency requirements and scale seamlessly with demand, while also leveraging AI tools like Amazon Bedrock and Amazon SageMaker to enhance data processing and predictive modeling.

ClickHouse addresses the challenges of row-level updates in columnar databases by treating updates as insert operations, utilizing engines like ReplacingMergeTree, CoalescingMergeTree, and CollapsingMergeTree. These engines leverage ClickHouse's high insert throughput and background merging processes to handle updates and deletes efficiently without modifying existing rows, thereby maintaining performance. This approach allows the database to manage high-ingest and rapidly changing workloads typical in IoT, e-commerce, finance, gaming, and CRM/HR applications. By transforming updates into fast inserts, ClickHouse circumvents the traditional update bottlenecks found in column stores, where each column is stored separately. The series explores the mechanics of these purpose-built engines, their role in fast SQL-style updates through patch parts, and benchmarks their performance against systems like PostgreSQL, demonstrating substantial speed advantages, especially in bulk update scenarios. The ReplacingMergeTree engine, for instance, retains the latest version of a row during merges, while CoalescingMergeTree supports partial updates, and CollapsingMergeTree allows for deletes by inserting canceling rows. Though efficient, these methods require users to understand ClickHouse’s merging logic and engine-specific behaviors, prompting the development of SQL-style updates in Part 2 of the series, which aim to offer familiar syntax without sacrificing speed.

ClickPipes has launched the public beta of its MySQL Change Data Capture (CDC) connector, enabling users to stream MySQL data directly into ClickHouse Cloud with ease. The release follows a successful private preview where users provided valuable feedback, allowing for enhancements such as reliable real-time replication, parallel snapshotting for faster initial loads, and intelligent change data capture that maintains transactional consistency. Built on the open-source CDC engine PeerDB, the connector supports a wide range of MySQL versions and includes features like user-facing alerts and secure connections. During the beta period, the connector is free, with pricing details to be announced as it approaches General Availability. The tool is designed to simplify data pipelines, reduce maintenance costs, and enhance real-time analytics capabilities by eliminating complex infrastructure requirements.

The text explores the potential for using ClickHouse as a business intelligence (BI) tool and suggests a novel approach for BI in 2025 by developing a self-service analytics agent. This agent, implemented as a Slack bot, directly interacts with a ClickHouse data warehouse to process natural language queries and generate SQL queries through the integration of PydanticAI and the Anthropic API. The bot, which is connected to ClickHouse via the ClickHouse MCP server, aims to streamline the BI process by eliminating the multi-step, often inefficient, traditional BI approach. It allows users to interact with data directly through Slack, providing real-time answers and simplifying data exploration. The article guides the reader through setting up the Slack bot, configuring environment variables, and handling message interactions, while emphasizing the simplicity and potential for future enhancements, such as producing visual representations or integrating with external systems.

OpenAI handles an immense volume of log data daily, comparable to billions of iPhone photos, necessitating a robust observability infrastructure to manage rapid data influx and ensure system reliability across its diverse operations like model research, ChatGPT, and enterprise APIs. To meet these demands, OpenAI uses ClickHouse, an open-source database solution chosen for its high performance, scalability, and flexibility, allowing them to manage complex queries and data spikes efficiently. During a significant traffic surge following the release of GPT-4o's image generation feature, OpenAI's team faced a critical challenge when CPU usage spiked, leading to performance issues. They resolved this by optimizing their Bloom filter operations, which reduced CPU usage by 40%, demonstrating the benefits of ClickHouse’s open-source nature and cloud-native design. Moving forward, OpenAI continues to enhance its observability system, focusing on improving query planning and developing more autonomous observability stacks, with the ultimate aim of integrating AI agents into incident response workflows.

Tesla, renowned for its expansive operations, has developed an advanced observability platform named "Comet" to manage its immense data requirements, leveraging ClickHouse for its scalability, speed, and durability. Faced with the limitations of existing tools like Prometheus, Tesla's software engineering team, led by Alon Tal, opted for ClickHouse due to its ability to handle Tesla's vast data volumes in real-time, maintain long-term data retention, and support complex queries with flexibility. Comet integrates seamlessly with PromQL, allowing Tesla's engineers to continue using familiar query languages while benefiting from ClickHouse's performance. The system has successfully managed a load of over 1,000 trillion rows without issues, demonstrating exceptional reliability and efficiency. Tesla is considering open-sourcing Comet, reflecting its commitment to innovation and collaboration in the tech community.

Anthropic has established itself as a leader in developing large-scale language models like the Claude series, emphasizing safety and responsibility. The company's infrastructure is deeply integrated with observability, which plays a crucial role in both performance and protection. In response to the rapid increase in Claude's usage, particularly after the release of Claude 3.5, Anthropic faced challenges in scaling its observability systems to handle vast amounts of telemetry and logs. Seeking a better database solution, they chose ClickHouse for its ability to support real-time data ingestion, fast analysis, and scalable deployment within their secure environment. Although ClickHouse Cloud offered dynamic scaling and cost-effective storage, Anthropic required a custom deployment to meet its strict security standards, leading to a hybrid approach with ClickHouse Cloud's architecture adapted for their infrastructure. This setup, orchestrated via Kubernetes and monitored by Prometheus, enables Anthropic to maintain their security protocols while allowing engineers to focus on developing tools and models without being burdened by database maintenance. ClickHouse also supports Anthropic's move toward agent-driven analytics, facilitating advanced model training and enabling programmatic querying of metrics. This foundation allows Anthropic to continue advancing AI capabilities while maintaining rigorous safety measures.

ClickHouse Cloud has introduced a new feature called the remote Model Context Protocol (MCP) Server, which is now available in the "AI Agents and Tools" category on the AWS Marketplace. This feature facilitates seamless integration of high-performance analytics into AI applications by providing a secure, managed connection between analytical data and MCP-compatible clients without requiring infrastructure deployment or maintenance. The remote MCP Server utilizes OAuth authentication for secure access and is built on the existing ClickHouse Cloud Query API Endpoints, ensuring compatibility with various AI tools. While currently in private preview, the remote MCP Server is designed to be read-only by default to enhance security and prevent unintended actions, such as prompt injections. ClickHouse encourages developers to explore these capabilities by offering a public demonstration and $300 in free credits for new users to experience the potential of conversational data exploration in real-time.

ClickHouse's new CoalescingMergeTree table engine, introduced in version 25.6, is designed to efficiently consolidate sparse updates, making it ideal for scenarios where data is incrementally updated, such as IoT device telemetry. Unlike traditional methods that require full row overwrites, CoalescingMergeTree gradually merges sparse records over time, preserving the most recent non-null values for each entity without sacrificing data fidelity. This approach is particularly beneficial in high-throughput environments, where it reduces data redundancy and improves query performance by pre-aggregating data on disk. The engine is optimized for append-only models, making it well-suited for handling fragmented updates from multiple subsystems, such as those found in modern connected vehicles. It offers a more efficient alternative to frequent single-row updates by allowing background merges to handle data consolidation, significantly reducing the number of rows and data scanned during queries. CoalescingMergeTree can also be applied in various other contexts, including user profile enrichment and security audit trails, where it helps reduce storage costs and query latency while maintaining data completeness.

The July 2025 ClickHouse newsletter covers a diverse range of topics, from detailed performance comparisons of ClickHouse against Databricks and Snowflake to reflections on obtaining the ClickHouse Certified Developer credential. Highlighting the achievements of Amos Bird, a top contributor and software engineer at Tencent, the newsletter also includes upcoming events, such as community meetups and virtual training sessions. Notable features of the ClickHouse 25.6 release, such as consistent snapshots and Bloom filter optimization, are discussed. Additionally, the newsletter explores various applications of ClickHouse, including real-time sports analytics and scaling observability platforms beyond 100 petabytes, along with insights from different companies' journeys to optimize their analytics infrastructure. The newsletter also delves into PostgreSQL integration, observability advancements, and practical advice for developers transitioning to ClickHouse, while offering educational resources like blog posts and webinars to enhance user understanding and engagement.

Lansweeper has evolved significantly from its origins as a basic inventory tool to become a sophisticated technology asset intelligence platform, capable of handling IT, OT, and IoT environments. Initially relying on MongoDB and later implementing Elasticsearch for enhanced analytical capabilities, Lansweeper faced challenges with cost and scalability as customer demands for real-time, actionable insights grew. The company subsequently transitioned to ClickHouse, which offered superior performance and cost efficiency, allowing them to manage vast amounts of data and complex queries. This shift enabled Lansweeper to enhance its offerings, such as customizable views and interactive dashboards, transforming it into a comprehensive insights platform. The adoption of ClickHouse Cloud further streamlined their infrastructure management, reducing costs and improving performance, while empowering users with greater control over their data through innovative features like Luzmo-powered interactive dashboards.

Tesla has developed Comet, a powerful observability platform built on ClickHouse, to manage the massive scale of data generated by its global operations, including Gigafactories and connected vehicles. The system was designed to handle enormous volumes of data in real-time, ensuring availability, durability, and speed, while retaining compatibility with Tesla's existing PromQL query language and tools like Grafana. ClickHouse was chosen for its exceptional performance, enabling Comet to ingest tens of millions of rows per second and scale far beyond Tesla's requirements, with a successful test reaching over one quadrillion rows. Comet's architecture includes an ingest pipeline using Kafka and OpenTelemetry, and a transpiler that converts PromQL into ClickHouse SQL, allowing engineers to maintain their workflows seamlessly. With Comet functioning smoothly, Tesla is exploring additional capabilities like distributed tracing and considering open-sourcing the platform for broader use.

LLM observability has become increasingly crucial for understanding and optimizing AI-driven applications, with tools like ClickStack providing robust solutions for monitoring performance, usage, and costs. This discussion focuses on using ClickStack, an open-source observability stack built on ClickHouse, to instrument LibreChat, an AI chat platform that integrates models from providers such as OpenAI and Google, and the ClickHouse MCP Server. By leveraging OpenTelemetry for capturing traces and metrics with minimal code changes, developers can gain insights into interactions between LibreChat, LLMs, and MCP services, allowing for real-time database queries and natural conversation interfaces. The implementation demonstrates how to deploy and configure ClickStack, enabling teams to monitor token consumption, usage patterns, and performance bottlenecks effectively. This observability framework not only enhances reliability and efficiency at scale but also facilitates proactive alerting on key properties to prevent unexpected cost surges or latency issues, ensuring a scalable, flexible, and cost-effective AI application management.

The text discusses a challenging bug encountered by the ClickPipes team related to logical replication slot creation on Postgres read replicas, causing significant customer frustration due to a query that could not be terminated. The issue, initially overlooked, resurfaced and was traced back to a Postgres bug that made certain backend processes unkillable when creating replication slots under specific conditions. The investigation revealed that the bug stemmed from an inefficient loop in the Postgres code, which failed to handle interrupts correctly on read replicas. A patch was submitted and accepted to address this issue, enhancing the robustness of Postgres in handling replication slots on hot standbys. This experience highlighted the complexities of modern database systems and the importance of collaboration with the open-source community to resolve such edge cases, ultimately benefiting both ClickPipes and the broader Postgres user base.

Anthropic, known for developing the Claude series of language models, faced challenges in managing vast telemetry, metrics, and logs as their AI models grew more sophisticated and usage soared. To address these challenges, they chose ClickHouse to enhance their observability infrastructure due to its ability to handle real-time data ingestion, provide fast analytics, and scale cost-effectively within a secure environment. This involved deploying a custom, air-gapped version of ClickHouse Cloud architecture internally, allowing them to maintain stringent security standards while improving database performance and reducing operational burdens on engineers. The implementation resulted in faster, reliable queries over large datasets, enabling the team to focus on developing advanced models and explore new possibilities like agentic analytics, which allows models to query metrics programmatically. This strategic move supports Anthropic's goal of building better AI tools and models while ensuring data security and operational efficiency.

ClickStack, an open-source observability platform built on ClickHouse, integrates logs, metrics, traces, and session replays using the same high-performance engine trusted by companies like Netflix and eBay. By incorporating OpenTelemetry for data collection and HyperDX as the UI, it offers a comprehensive observability experience from data ingestion to visual exploration. A significant update includes beta support for the native JSON column type, which enhances scalability, performance, and compression, offering up to 9x faster queries over ClickStack's previous Map type. This change allows for the ingestion and storage of semi-structured data without rigid schemas, preserving data types and improving query efficiency. HyperDX benefits from this update by enabling smarter filtering and more responsive visualizations. Additional improvements include Helm chart enhancements, a CSV download feature, an Elastic Stack Migration Guide, and optimizations in Docker image sizes, all aimed at improving user experience and performance.

ClickHouse, a prominent player in real-time analytics and data management, has appointed Kevin Egan as Chief Revenue Officer, capitalizing on his extensive experience with leading tech companies such as Atlassian, Slack, Dropbox, and Salesforce. Egan's role will involve overseeing global go-to-market functions as the company experiences increasing demand for its services. His appointment follows ClickHouse's recent $350 million Series C funding round, which has elevated the company's valuation to $6.35 billion and supports the expansion of its ClickHouse Cloud service. This service, which experienced over 300% growth in the past year, now caters to more than 2,000 customers across various industries. ClickHouse recently hosted its first user conference, featuring major organizations like Tesla and OpenAI, further showcasing its influence in large-scale production environments. CEO Aaron Katz expressed enthusiasm about Egan joining the leadership team, emphasizing his proven ability to scale enterprise sales and build strong customer relationships, which are crucial as ClickHouse continues its expansion. Known for its high-performance, open-source columnar database, ClickHouse supports real-time data processing and analytics, making it a trusted platform for companies seeking instant insights from vast data volumes.

The 2025 Roland Garros final between Jannik Sinner and Carlos Alcaraz highlighted the fine margins in tennis, as Sinner, despite being on the brink of victory with three championship points, ultimately lost the match. This scenario was reminiscent of a previous analysis of Alcaraz's performance at Wimbledon, which suggested he was never truly in danger of losing his first-round match, despite the game extending to five sets. The analysis used ClickHouse, a database management system, to build a function that computes how many points a player needs to win the game, illustrating the complexity of tennis scoring. The author initially faced challenges with AI coding assistants in understanding tennis's unique scoring system, leading to the choice of ClickHouse for the analysis. The study involved creating tests for user-defined functions and storing tennis data in ClickHouse to determine players' proximity to victory. The project culminated in a Streamlit and plot.ly application to visualize match data, underscoring the technical and analytical effort involved in understanding tennis matches through data.

ClickHouse Cloud has achieved a fully stateless compute architecture by introducing a new in-memory database engine powered by a Shared Catalog, effectively removing the dependency on local disks. This transformation began with the decoupling of data storage from compute using the SharedMergeTree table engine, which utilizes shared object storage for data, and the adoption of a distributed cache to mitigate object storage latency. The final step involved decoupling metadata from local disk storage through the Shared Catalog, enabling stateless compute nodes to operate without disk dependencies. This architecture supports cloud-native elasticity, resilience, and correctness, with capabilities such as atomic DDL operations and seamless integration with external data formats like Iceberg and Delta Lake. The system now allows fast, diskless provisioning and promises a future of serverless computing, enhancing both performance and scalability.

ClickHouse version 25.6 introduces a range of enhancements, including 25 new features, 32 performance optimizations, and 105 bug fixes, with a focus on ensuring consistent data snapshots across all subqueries and table references within a query. Notably, this release allows filtering by multiple projections and introduces JSON support in Parquet, Time, and Time64 data types for enhanced compatibility with other SQL DBMS. The update also includes new system tables like codecs and iceberg_history for improved data management and insights, alongside a significant optimization in the Bloom filter index that resulted in a 40% reduction in CPU usage during high traffic spikes, as experienced by OpenAI. Additionally, the release features a new command-line tool, chdig, for real-time monitoring and diagnostics, providing detailed views of query performance and server behavior.