June 2025 Summaries

Real-time tick data applications, exemplifying real-time analytics, involve processing high-frequency event streams with minimal delay, crucial in financial markets where even slight lags can affect trades. ClickHouse is well-suited for handling such workloads, offering efficient data compression, time-based queries, and real-time data processing capabilities. This guide details building a real-time tick data application using Polygon.io for market data access and ClickHouse for data storage and querying, employing NodeJS for backend operations and React for live visualization. The text explains the differentiation between quotes and trades, the importance of WebSockets for data streaming, and efficient data modeling in ClickHouse. It further highlights the significance of optimizing ingestion methods, whether synchronous or asynchronous, and offers practical strategies for scaling, latency monitoring, and using materialized views for pre-aggregating data to optimize query performance. Essential SQL queries for real-time data visualization are provided, focusing on constructing live trading tables and candlestick charts, with an emphasis on maintaining system performance and reliability amid growing data volumes.

OpenAI faces the formidable challenge of managing massive volumes of log data, equivalent to 500 Libraries of Congress daily, as it supports various high-growth AI systems like model research, ChatGPT, and enterprise APIs. To handle the rapid growth and complexity of their observability needs, OpenAI selected ClickHouse, an open-source, cloud-native database, for its flexibility, scalability, and ease of integration into AI workflows. When a surge in user activity overwhelmed their systems during the launch of image generation in GPT-4o, the team optimized performance by reducing CPU usage through a simple code change in Bloom filter indexing, showcasing ClickHouse's adaptability. OpenAI continues to refine its observability infrastructure, focusing on smarter query planning and integrating AI agents for autonomous incident management, all while contributing to the broader ClickHouse community through shared improvements.

Braze, a multi-channel engagement platform used by major brands such as PayPal and Canva, has significantly enhanced its real-time analytics capabilities by transitioning from a dual-system architecture to a streamlined solution using ClickHouse Cloud. Previously, Braze managed a complex setup with MongoDB and Snowflake to handle the massive volume of data generated by trillions of messages and billions of events per hour, but faced challenges with latency, complexity, and scalability. The new architecture allows data to flow seamlessly from Kafka into ClickHouse, enabling fast, reliable, and unified analytics with reduced operational burdens. This transition not only improves query performance and consistency but also supports future expansions into advanced analytics and observability, highlighting ClickHouse's potential to replace older, costlier systems.

Replacing traditional joins with in-memory dictionaries in ClickHouse Cloud can significantly enhance query performance and reduce costs with minimal SQL modifications. By utilizing ClickHouse-native dictionaries for dimension table lookups, users can achieve up to 6.6 times faster query execution and over 60% cost savings, as demonstrated in benchmark tests involving 1.4 billion rows. This approach leverages the high-speed, low-latency advantages of in-memory key-value structures, making it particularly effective for foreign key joins on small dimension tables in star or snowflake schemas. The transition from joins to dictionaries is seamless, requiring no data reloading or schema changes, and scales efficiently across multiple compute nodes. This method not only optimizes execution times for complex queries but also highlights the adaptability and power of the ClickHouse engine, encouraging users to consider its cloud services for enhanced performance and cost efficiency.

A public benchmark originally designed to test join-heavy SQL queries on Databricks and Snowflake was used to evaluate the performance of ClickHouse Cloud, revealing that ClickHouse was faster and more cost-effective across various scales, from 721 million to 7.2 billion rows. The benchmark involved running 17 SQL queries, most of which included joins, without any special tuning or rewriting, on similar instance sizes. The results demonstrated that ClickHouse consistently outperformed the alternatives, showing significant improvements in speed and cost, even as the dataset size increased. ClickHouse's enhancements over the past six months have resulted in dramatically better join performance, and future tests, including a full TPC-H benchmark, aim to further showcase ClickHouse's capabilities in handling complex joins. The benchmark setup and results are fully reproducible, with resources available on GitHub for those interested in testing ClickHouse Cloud against other platforms.

Langfuse, an open-source LLM engineering platform, initially launched with a simple Postgres-based architecture that allowed rapid deployment and proved popular with enterprises needing data observability without external data transfers. However, as demand grew and more companies integrated Langfuse, the platform faced scalability challenges with billions of rows and complex queries. To address these issues, Langfuse re-architected its system around ClickHouse, a columnar OLAP database, which offered better performance and scalability. This transition was carefully communicated to their self-hosted user community through open discussions and guided migrations, resulting in a successful rollout. Langfuse now provides a range of deployment options, including Docker Compose, Helm charts, and ClickHouse Cloud, catering to both small-scale setups and large enterprises. This strategic shift has allowed Langfuse to maintain ease of adoption while supporting enterprise-scale workloads, positioning it for future growth.

The new Terraform provider, clickhousedbops, has been launched to facilitate the management of database users, roles, and privileges for ClickHouse instances, whether on ClickHouse Cloud or self-hosted. This provider enhances existing Terraform functionalities by allowing users to programmatically automate configuration and management of their ClickHouse services, which simplifies maintaining the state across different environments. Users can define connections to ClickHouse instances, create databases, and assign users and roles with specific privileges. A practical example demonstrates setting up a ClickHouse Cloud service for log analytics and managing user access through Terraform. The provider also supports self-hosted ClickHouse deployments, allowing resource creation across multiple replicas in a cluster. Feedback is encouraged through GitHub repositories to improve the offering.

Over the past year, LogHouse, an internal logging platform initially designed to monitor ClickHouse Cloud, has undergone substantial growth and transformation, handling over 100 petabytes of data across nearly 500 trillion rows. This expansion necessitated significant architectural changes and the development of new tools, such as the System Tables Exporter (SysEx), to address the inefficiencies of OpenTelemetry (OTel) in managing high-throughput, high-fidelity system logs. SysEx has enabled a dramatic increase in event processing efficiency, achieving a 20-fold surge in data handling with only a tenth of the previous CPU resource usage. The shift from a one-size-fits-all approach to specialized tooling, combined with the integration of HyperDX, a ClickHouse-native UI, has not only improved data management but also fostered a cultural shift towards high-cardinality, wide-event-based observability. HyperDX facilitates seamless log exploration and correlation, supporting both standardized OTel formats and specialized data from SysEx, thus providing a unified user experience. As LogHouse continues to evolve, future enhancements, including zero-impact scraping and potential migration to JSON, are anticipated to further refine the platform's capabilities and efficiency.

ClickHouse Cloud Dashboards have officially moved from beta to general availability, offering users an intuitive way to create, share, and visualize data without the need for SQL proficiency. Initially launched to address the challenge faced by non-technical users who were deterred by SQL-based data access, the dashboards allow users to generate visual insights from SELECT queries, leveraging the Vizhouse charting library for diverse visualization types. Over the six-month beta period, user feedback and telemetry data helped refine the feature, leading to enhanced interactivity and sharing capabilities. Users can now track key performance indicators (KPIs) and monitor system performance more efficiently, thanks to ClickHouse's powerful data processing capabilities, which allow for rapid querying of large datasets. Looking forward, ClickHouse plans to expand dashboard functionalities with enhanced visualizations, AI integration for SQL analysis, and options for embedding dashboards into applications, aiming to further simplify data interaction and broaden accessibility.

Cloud CIRCUS, a Tokyo-based SaaS company, aims to streamline marketing efforts for businesses in Japan by offering digital tools like BlueMonkey, a CMS platform hosted on AWS. BlueMonkey supports over 2,000 customer websites, each generating extensive access logs, which were initially analyzed using Amazon Athena. However, this approach was slow, costly, and difficult to scale due to the need for thousands of separate queries and unpredictable expenses. To address these challenges, Cloud CIRCUS transitioned to ClickHouse, an open-source, self-managed analytics pipeline that offered faster, more efficient log analysis with fixed costs. This migration has resulted in significant improvements in query speed and cost management, consolidating all logs into a single table and automating data ingestion. With ClickHouse, Cloud CIRCUS now efficiently handles large-scale log analysis, enhancing its ability to support the growing demands of BlueMonkey and its numerous customer websites.

The June 2025 ClickHouse newsletter announces the launch of ClickStack, an open-source observability solution that combines ClickHouse's database technology with a user interface from HyperDX, aiming to enhance the observability ecosystem. The newsletter highlights community member Joe Karlsson, a Senior Developer Advocate at CloudQuery, known for his contributions to data engineering content, including his work on a full-text search engine using ClickHouse's ngrambf_v1 Bloom filter indices. Various events and workshops, both virtual and in-person, are scheduled globally to provide training and networking opportunities for ClickHouse users. Additionally, the newsletter covers advancements like the ClickHouse 25.5 release, which includes improvements in vector similarity indexing and Hive metastore catalog support, and explores diverse use cases such as analyzing Financial Information eXchange (FIX) data, building agentic applications, and developing scalable user segmentation pipelines. Through detailed blog posts, the newsletter provides insights into leveraging ClickHouse for tasks like efficient query optimizations, constructing conversational analytics experiences, and executing K-Means clustering for user segmentation.

Cloud CIRCUS, a SaaS company based in Tokyo, enhances business marketing efficiency in Japan with digital tools, including the CMS platform BlueMonkey, which hosts over 2,000 websites on AWS. Due to the increasing volume of access logs generated by CloudFront distributions, the company faced challenges with Amazon Athena's slow query performance and high costs. To address this, infrastructure engineer Zhu Jiulin presented a solution at the 2025 ClickHouse Tokyo meetup, where Cloud CIRCUS transitioned to ClickHouse for log analysis. The self-managed, open-source ClickHouse system enabled faster, more cost-effective data processing by consolidating logs into a single table and automating the import pipeline with Amazon EventBridge and ECS. This switch resulted in significant performance improvements, reducing query times from 16 seconds with Athena to 0.043 seconds with ClickHouse, while also lowering monthly costs to approximately $300. The transition allowed Cloud CIRCUS to support thousands of customer websites efficiently and scale its operations to meet growing traffic demands, offering a streamlined and cost-predictable analytics solution.

LINE Digital Frontier, a subsidiary of WEBTOON Entertainment, manages LINE Manga, a leading digital comic platform with around 150 million monthly active users, leveraging data analytics to optimize recommendations and sales tracking. Despite its success, the platform faced challenges with its extensive MySQL database network, which hindered large-scale data analysis due to complex and time-consuming processes. To address this, LINE Manga integrated ClickHouse, enabling real-time querying of MySQL without data ingestion, thus improving efficiency by directly querying MySQL with virtual tables. This method allows seamless integration of product and user transaction data from different servers, supporting real-time SQL responses. The use of ClickHouse has streamlined operations, making query execution significantly faster and easier to debug while maintaining MySQL as the primary record system. The solution serves as a model for other developers, showcasing how to enhance data processing without overhauling existing systems.

The LINE MANGA team, operating a popular digital comic service, faced challenges with their MySQL database architecture as their user base grew. They struggled to analyze reader behavior and optimize recommendations in real-time due to the complexity of joining data from multiple servers. To overcome this, they adopted ClickHouse, which allowed them to query MySQL directly without duplicating data or moving it into a separate system. This enabled real-time analysis, aggregation, and debugging capabilities, making their development process faster and more efficient. The team uses ClickHouse as a real-time query layer, leveraging its MergeTree table engine and condition pushdown feature to unify horizontally sharded data and aggregate sales in real time. By keeping things simple and using tools like clickhouse-local for local development, they've opened the door to real-time analytics without adding unnecessary complexity.

The text discusses the concept of Agentic applications, which are intelligent, interactive user experiences powered by Large Language Models (LLMs). These applications aim to provide a seamless and natural experience for users by leveraging AI capabilities. The article focuses on building an Agentic application using ClickHouse MCP Server and CopilotKit. ClickHouse is used as a real-time analytics database that supports complex analytical tasks and high-frequency interactive querying, ensuring stable performance during chat-based interactions. The application allows users to build their custom dashboards for the UK real estate market by generating charts based on user-provided data. The use of fine-grained permissions and quotas in ClickHouse provides control over what the LLM can do on behalf of the user. The article concludes that leveraging the capabilities of LLMs, such as ClickHouse MCP Server and CopilotKit, opens up new possibilities for creating AI-powered tools that provide deeper insights and better user experiences.

The success rate for product changes is around 30%, indicating that many new features don't deliver the impact teams are hoping for. This lack of clarity and inability to isolate cause and effect can lead teams to double down on unsuccessful features. A better approach involves using A/B testing, which provides clarity on whether a feature is working or not. However, scaling A/B testing without starting from scratch requires a fast, flexible, and self-serve system that can handle large volumes of data and experiments. GrowthBook and ClickHouse offer a solution by providing an open-source platform that integrates with warehouse-native ClickHouse, enabling teams to run sophisticated A/B tests with flexibility and control, while also delivering speed and scalability.

ClickStack is a newly announced open-source observability solution built on ClickHouse, designed to provide a comprehensive and accessible experience for managing logs, metrics, traces, and session replays. Leveraging ClickHouse's performance and efficiency, ClickStack offers an open observability stack with features like OpenTelemetry collectors, a UI optimized for wide event data, natural language queries, and alerting, all running on the trusted ClickHouse engine. Historically, only teams with resources to build custom observability experiences on ClickHouse could fully benefit from it, while others relied on general visualization tools or commercial platforms, facing limitations with long SQL queries and underutilized ClickHouse capabilities. With ClickStack's release, supported by HyperDX, users now have a fair option for utilizing high-speed search and aggregation for high cardinality wide event data, scalable in an open-source format. The integration of OpenTelemetry and the tailored UI by HyperDX simplifies the observability process, ensuring efficient query building and pattern recognition without concealing ClickHouse's power, making modern observability approaches more accessible.

ClickHouse has announced the general availability of its Postgres CDC connector for ClickPipes, enabling seamless replication of Postgres databases to ClickHouse Cloud with just a few clicks. This integration, achieved in collaboration with PeerDB, aims to streamline enterprise-level Postgres use cases by offering robust real-time analytics capabilities. The connector allows companies like Ashby, Seemplicity, and AutoNation to efficiently manage large-scale data by reducing report generation time significantly and supporting mission-critical workloads. It features enhanced functionalities such as parallel snapshot loading, minimal replication delay, automatic schema replication, and secure connections, making it a comprehensive solution for modern data architectures. As part of its competitive pricing strategy, ClickHouse will introduce a cost-effective fee structure starting September 2025, emphasizing its goal of making seamless data integration accessible and affordable.

ClickHouse Government is a solution designed for the public sector, focusing on performance, cost savings, ease of deployment, and security. The company behind ClickHouse, Inc., founded in 2021, has developed a proprietary cloud-native architecture that supports ClickHouse Cloud on AWS, GCP, and Azure, with support for Bring Your Own Cloud (BYOC) deployments. This architecture is used to build ClickHouse Government, which provides performance, cost efficiency, and security features, including separation of compute and storage, object storage, and ZSTD compression algorithm. The solution is initially available in AWS and will be deployed to other clouds or hardware-based environments upon request. It offers proprietary features such as SharedMergeTree, improved insert throughput, and stronger consistency for select queries, while also providing compression benefits compared to other cloud providers. ClickHouse Government aims to help public sector customers achieve near-unbelievable levels of speed and compression, resulting in 2-10x improved performance and 3-5x cost savings over other providers. The solution is pre-configured with FIPS 140-3 level encryption, baseline security hardening, and deployment support via Helm charts, making it easier to implement and comply with security standards such as FedRAMP Moderate and IL 2-6.

The rise of rapid delivery has transformed the way people shop for groceries and convenience goods, with apps like Uber Eats, Flink, and DoorDash delivering products in under two hours. However, consumer packaged goods brands have struggled to keep up, lacking visibility into stock levels, search rankings, pricing, promotions, and other key metrics. Rapid Delivery Analytics (RDA) is a Paris-based startup that offers a digital shelf analytics platform built specifically for rapid delivery, providing real-time visibility into these critical areas across over 40 delivery apps in more than 100 countries. RDA's platform uses ClickHouse Cloud on AWS to power its analytics engine, which ingests and processes billions of rows of data daily, enabling fast query performance and scalability. With the help of ClickHouse Cloud, RDA has achieved cost savings, improved efficiency, and the ability to scale without sacrificing performance, allowing it to support large CPG brands like PepsiCo and Unilever in a rapidly evolving market.

MCP is a protocol for connecting third-party services to Large Language Models (LLMs). Creating an MCP server defines how a client can interact with the service. An MCP client connects to the server, allowing the LLM to interact with the service. MCP is becoming the de-facto protocol and has been implemented in ClickHouse, a database designed for interactive analytics at scale. Natural language interfaces are becoming popular across various domains, including data analysis, making it easier than ever to work with data. The expectation of speed and interactivity in user experience is universally expected, and LLMs are helping to round out people's skills. ClickHouse is the ideal database for agentic AI data workflows due to its fast analytical capabilities and support for interactive analytics at scale. Various frameworks such as Agno, DSPy, LangChain, LlamaIndex, and PydanticAI have been implemented to integrate with MCP, allowing developers to build applications that expose data to end-users and generate insights without SQL. The examples provided demonstrate how to initialize the MCP server and create agents using these frameworks, showcasing the potential of MCP in building production-grade applications with Generative AI.

The eBird dataset, which contains over 1.5 billion bird observations worldwide, was analyzed using ClickHouse, a column-store database management system. The data was loaded into ClickHouse using the `clickhouse-local` tool and then transformed to optimize its performance in a web-based application. The dataset was preprocessed by analyzing its structure, creating a table schema, and applying dictionary encoding to reduce storage requirements. The optimized table structure included materialized columns for efficient mapping applications, and indices were created for faster search capabilities. The data was loaded into ClickHouse using the `clickhouse-client` tool and resulted in a significant reduction in storage size compared to the original zip file. The database management system's ability to handle large-scale geographical datasets made it an ideal choice for this project.

ClickHouse, an open-source relational database management system, has made significant advancements announced at its Open House user conference. The company's Postgres CDC connector for ClickPipes is now Generally Available in ClickHouse Cloud, providing seamless integration and 10x faster syncs. Lightweight Updates have been introduced to improve performance on OLAP workloads by representing updates as "patches" that contain only the changed rows and columns. ClickStack, a new open-source observability stack built on ClickHouse, delivers a complete experience for logs, metrics, and traces. Additionally, ClickHouse has improved its performance with features like distributed caching, stateless workers, SQL join performance enhancements, and agent-facing analytics capabilities. These advancements aim to make ClickHouse the default data stack for modern businesses, providing a flexible and efficient environment for data exploration, AI/ML workloads, and observability tooling.

The latest version of ClickHouse, which is a high-performance data warehouse system, has been released with several new features and improvements. The release includes 15 new features, 23 performance optimizations, and 64 bug fixes. Some notable new features include the vector similarity index in beta, support for the Hive metastore catalog for Iceberg tables, and the ability to track when new functions were added to ClickHouse. Additionally, implicit table creation has been improved, making it easier to work with clickhouse-local. The release also includes a list of new contributors who have made significant contributions to the project. Furthermore, support for Geo types in Parquet files has been added, enabling users to query and analyze geospatial data more efficiently. Another notable feature is the tracking of function origin with the `introduced_in` field, which provides information on when functions were added to ClickHouse. The release also includes a presentation slides link, where users can learn more about the new features and improvements.