September 2025 Summaries

The September edition of "What's New in ClickStack" introduces several advancements for the open-source observability stack centered on ClickHouse, enhancing both functionality and user experience. Key updates include the ability to import and export dashboards via HyperDX, facilitating easier sharing and customization while maintaining compatibility through versioned JSON files. The release also enables custom configurations for the OpenTelemetry Collector, allowing users to modify its settings to better suit local deployments and specific monitoring needs. Additionally, the update expands analytical capabilities by allowing users to bypass standard aggregation types and directly utilize ClickHouse's extensive array of functions, thus supporting more in-depth data analysis. Performance improvements are highlighted by the introduction of time-window chunking, which optimizes query execution to prioritize recent data and reduce system load. The addition of a delta function for gauge metrics aligns with Prometheus’s behavior, enabling detailed tracking of changes over time, such as memory usage fluctuations, which can signal potential issues like memory leaks. These enhancements collectively aim to deliver a more powerful and seamless experience for ClickStack users, supported by contributions from a vibrant open-source community.

ProcessOut, a payment orchestration platform founded in 2015, aims to transform payments into a growth driver for businesses by offering smart routing, analytics, payment vaulting solutions, and more, enabling merchants to connect with multiple payment service providers and expand into new markets without accruing technical debt. The company recently transitioned from an Elasticsearch-based data architecture to ClickHouse Cloud to modernize its stack, improve performance, and reduce costs, resulting in a two-thirds decrease in analytics expenses. ClickHouse Cloud's simplicity, scalability, and SQL-based system have facilitated faster data ingestion and easier experimentation, allowing ProcessOut's team to iterate quickly and enhance their payment optimization tools. The platform now supports richer analytics, enabling the integration of multiple data sources to provide deeper insights into payment processes, aligning with ProcessOut's mission to simplify payments and empower brands to leverage them as a superpower for growth.

The recent public beta release of ClickHouse's "Ask AI" agent and remote MCP server marks a significant step towards enhancing agent-facing analytics, enabling AI agents to efficiently generate insights by querying data at machine speed. The "Ask AI" agent, integrated within ClickHouse Cloud, provides a seamless, natural language interface for users to conduct complex data analyses without writing SQL, while the "Docs AI" sub-agent offers direct, contextual answers to documentation queries. Meanwhile, the remote MCP server facilitates secure and straightforward access to ClickHouse Cloud for external agents, utilizing the Model Context Protocol for structured data access. These innovations aim to drastically reduce the time to insight from hours or days to mere seconds or minutes, as demonstrated by ClickHouse's internal use of the Dwaine agent, which has transformed team access to business data. However, the company acknowledges the challenge of AI hallucinations and emphasizes the importance of providing comprehensive context to mitigate inaccuracies. Overall, these features aim to bridge the gap between data questions and answers, inviting user feedback to refine these beta offerings.

The Azure Blob Storage connector for ClickPipes is now generally available, enhancing the capabilities of ClickHouse Cloud by allowing seamless file loading from Azure Blob Storage with improved performance and programmatic access via OpenAPI and Terraform. The connector underwent an extensive beta phase, proving its reliability and scalability for diverse data analytics workloads, and now features a custom HTTP client for better performance, aligning it with other object storage ClickPipes. It supports mature deployments with a new azureblobstorage source type in the ClickPipes OpenAPI specification and the ClickHouse Terraform Provider. The connector is designed for reliability, performance, and security, handling automatic retries, supporting secure access, and optimizing data ingestion. Fully integrated into the ClickHouse Cloud experience, it offers built-in metrics, detailed logs, and cost-effective pricing, making it a compelling alternative to third-party ETL tools. It is accessible to all ClickHouse Cloud customers, supporting a range of service tiers, and aims to simplify ETL processes while reducing data movement costs.

The text outlines a sophisticated system designed to improve the efficiency of bug tracking within a CI environment by grouping crash reports through stack trace similarity. This method addresses the challenge of distinguishing new bugs from known ones by leveraging a process of parsing and cleaning crash reports, calculating stack trace similarity, and grouping similar traces. Developed initially by Michael Stetsyuk for ClickHouse Cloud, the system uses ClickHouse to collect and analyze data while maintaining performance by separating collection and analysis on different clusters. The process involves cleaning stack traces to remove noise, grouping identical traces, and calculating similarity to identify significant new traces, which are then used to create or update GitHub issues. The system benefits from automatic updates and integration with GitHub, providing a streamlined approach to bug management and inviting collaboration through open issues labeled with "crash-ci."

Superwall, a small company with only 14 team members, plays a significant role in paywall monetization for mobile subscription apps reaching billions of users. By integrating Superwall's SDK, clients can perform real-time A/B testing, adjust global pricing, and offer discounts without app store updates. The company relies on a robust tech stack, initially built with Apache Kafka for data streaming and ClickHouse for analytics, to manage and analyze large-scale data in near real-time. Over time, Superwall transitioned to ClickHouse Cloud and WarpStream, which offered scalability, cost-effectiveness, and ease of operation, ultimately enhancing their platform’s durability and performance. These technologies allow Superwall to maintain focus on innovation, including AI-driven features like Demand Score, which predicts user subscription likelihood, and to continue supporting mobile app developers in optimizing monetization strategies.

SewerAI, a company founded in 2019, aims to modernize the sewer management industry by automating the tedious task of defect detection in inspection footage through artificial intelligence. They have developed the first web-based platform for underground infrastructure management, utilizing computer vision to streamline processes and enable real-time analytics powered by ClickHouse. The company's transition from Postgres to ClickHouse involved overcoming challenges in data handling and compliance, ultimately adopting PeerDB for seamless data replication. Their innovative "Megatable," a denormalized data structure, enhances analytical performance but initially strained computational resources, prompting optimization strategies like filtering before joining and focusing on recent data. Despite scaling challenges, SewerAI successfully maintained performance and customer satisfaction, proving that even niche industries can benefit from advanced data solutions.

Changan Ford, a joint venture between Changan Automobile Group and Ford Motor Company founded in 2001, underwent a significant digital transformation by establishing a Customer Data Platform (CDP) to enhance its digital marketing efforts. Initially running on ClickHouse's Community Edition, the CDP faced challenges as data demands grew, leading to high storage costs, system instability, and operational complexity. In 2024, Changan Ford migrated to ClickHouse Enterprise Edition on Alibaba Cloud, which offered compute-storage separation and serverless elasticity, reducing costs by 40% while improving system stability and operational efficiency. This transition allowed for real-time scaling and a more flexible, scalable, and cost-effective data infrastructure, enabling the company to enhance marketing strategies and explore new data-driven opportunities in after-sales analytics and supply chain optimization.

The evolution of observability tools has led to significant challenges for teams managing large-scale systems, primarily due to escalating costs and data silos. Initially conceived to provide confidence during system issues, observability often becomes financially burdensome as systems grow. Historical tools like Splunk and open-source alternatives such as Elasticsearch offered early solutions, but each had limitations in scaling and usability. Cloud-based platforms like Datadog further complicated matters with costly pricing models, leading teams to make difficult compromises between comprehensive coverage and affordability. OpenTelemetry presents a potential solution by standardizing data collection, thus reducing vendor lock-in, while columnar databases like ClickHouse offer efficient data storage and retrieval, enabling high-performance observability without prohibitive costs. However, effective observability requires more than just a robust database; it demands user-friendly interfaces to democratize access. ClickStack aims to address these needs by leveraging ClickHouse's capabilities, promoting an open-source model that supports diverse data types and integration without the constraints of previous systems.

The September 2025 edition of the ClickHouse newsletter highlights various updates and events, including a detailed exploration by Tom Schreiber on scaling GROUP BY operations across thousands of cores for ultra-fast aggregations. The release of a new Parquet reader promises significant performance improvements, while Airbyte has introduced a first-class ClickHouse connector that enhances data loading efficiency. Featured community member Gaurav Sen is recognized for his educational contributions on ClickHouse's applications in AI and system design. The newsletter also outlines numerous global events, meetups, and training sessions, emphasizing ClickHouse's expanding community and ecosystem. Additionally, articles discuss common user mistakes, Laminar's innovative use of ClickHouse for AI observability, and various technical insights into leveraging ClickHouse for advanced data engineering tasks.

The text is a comprehensive guide on building a ClickHouse-powered API integrated into a web application, specifically for those familiar with Postgres. It details the process of setting up a ClickHouse Cloud service to sync real-time data from Postgres, using Moose OLAP for TypeScript native interfaces, and creating type-safe APIs for analytical queries. The guide addresses the challenge of scaling analytics in user-facing apps and offers solutions like syncing with ClickHouse via Change Data Capture (CDC) and developing locally with MooseStack. It emphasizes the advantages of using ClickHouse as a fast analytical database, complemented by Moose OLAP's ORM-inspired schemas for seamless developer experience. The integration process includes generating TypeScript types from schema, running queries securely, and testing APIs locally with real-time feedback. The guide also covers the deployment of these setups using Boreal, which facilitates a streamlined transition from development to production with GitHub integration and CI/CD support, ensuring robust and efficient analytical capabilities alongside existing transactional databases.

In the fast-paced world of cybersecurity, Exabeam is leveraging ClickHouse Cloud to enhance its security operations platform, focusing on reducing the mean time to detection (MTTD) of threats. By utilizing AI and machine learning, Exabeam's platform is built on four key pillars—ingestion, threat detection, investigation, and response—aiming to streamline the process from data collection to actionable insights. At ClickHouse's Open House user conference in May 2025, Exabeam's engineering team detailed how they optimized their system to handle 1.2 million events per second across 10 global regions, managing over 1 trillion events and using strategies like compute-compute separation and schema consolidation to improve efficiency. This architecture supports real-time analytics, enabling rapid detection and response to threats while keeping compute costs under control. By optimizing their ingestion pipeline and adjusting their indexing approach, Exabeam has successfully decreased latency and memory usage, making their analytics engine both fast and scalable, ultimately aiding security teams in quickly addressing potential threats.

Lyft, a major rideshare company, has transitioned its analytics system from Apache Druid to ClickHouse Cloud to better manage the vast amounts of data generated from providing over 800 million rides annually across the U.S. and Canada. This shift was driven by the need for a more efficient, scalable, and cost-effective solution that could support both batch and real-time data processing. ClickHouse Cloud offers high performance, simplified infrastructure, and reduced operational costs, which have significantly improved Lyft's data operations. The migration involved overcoming challenges such as adapting to the new cloud features and reworking internal systems. The real-time analytics are crucial for Lyft's day-to-day operations, enabling quick decision-making in areas like supply and demand tracking and campaign targeting. Meanwhile, batch data supports long-term strategic insights and efficient resource use. Lyft continues to enhance its data platform by integrating ClickHouse Cloud more deeply into its broader data infrastructure, focusing on data completeness, discovery, and accessibility to foster a more data-driven culture.

The Postgres CDC connector in ClickPipes facilitates seamless data replication from Postgres to ClickHouse Cloud, enhancing real-time analytics capabilities. Supported by PeerDB, the connector enables both one-time migrations and continuous replication, addressing customer demands for integration with TimescaleDB, a time-series Postgres extension by TigerData. It caters to various use cases, including online data migrations, iterative migrations, and co-existence of TimescaleDB and ClickHouse for real-time applications. Key features include parallel snapshotting for fast initial loads, support for uncompressed and compressed hypertables, and automatic schema change replication. Additionally, ClickPipes offers comprehensive metrics and alerts for monitoring replication processes. The solution's design handles the complexities of TimescaleDB's hypertable architecture, ensuring reliable replication even when dealing with compressed data.

Intel Shanghai's performance optimization engineers, Jiebin Sun, Zhiguo Zhou, Wangyang Guo, and Tianyou Li, have been focusing on enhancing the scalability of ClickHouse on Intel's ultra-high core count processors. As modern processors, like the Granite Rapids and Sierra Forest, push core counts to unprecedented levels, databases like ClickHouse face challenges in fully utilizing these cores due to bottlenecks such as lock contention, cache coherence, and memory management. The engineers employed various profiling tools to analyze ClickHouse's performance, leading to significant optimizations. These include reducing lock contention by rethinking thread coordination, optimizing memory management with improved allocator configurations, increasing parallelism in hash table merging, leveraging SIMD instructions for more efficient string searches, and addressing false sharing by aligning data structures with cache lines. These optimizations have resulted in substantial speedups for specific ClickBench queries, demonstrating that ClickHouse can effectively scale on high core count systems, a crucial capability as processors continue to evolve towards even higher core counts.

Borderless360, an end-to-end logistics platform for ecommerce brands, emphasizes the importance of visibility in logistics operations to ensure seamless package delivery and tracking for customers. Founded in 2017, the company operates a global network, managing everything from inventory orchestration to last-mile delivery and cross-border services. As Borderless360's customer base and order volumes expanded, particularly post-pandemic, they encountered performance issues with their Postgres and Metabase analytics setup, leading them to adopt ClickHouse as a more efficient OLAP database. This switch significantly improved the speed of their analytics, reducing query response times from seconds to milliseconds, thereby enhancing customer experience and operational efficiency. The transition also lowered costs and enabled the development of an AI-powered logistics agent that provides real-time, natural language insights, an innovation that would not have been possible with the previous setup. With ClickHouse, Borderless360 has transformed its analytics capabilities, paving the way for advanced data science initiatives and a superior customer experience, reinforcing the company's commitment to making logistics effortless for end-users.

ClickHouse has emerged as a preferred database for real-time analytics due to its ability to handle high concurrency and low-latency queries, making it ideal for both analytics and observability. The unification of these domains is facilitated by tools like ClickStack, which integrates ClickHouse, HyperDX, and OpenTelemetry to provide a comprehensive observability stack. ClickStack simplifies the process of correlating analytics data with observability metrics, as demonstrated by the ClickPy project, which analyzes Python package downloads at scale. By instrumenting applications with ClickStack SDKs, developers can gain deep visibility into application performance and user experience through traces and session replays. This approach allows for the correlation of application metrics, such as package downloads, with performance metrics, like query latency, treating observability as a data problem to be solved within the same engine. This setup not only aids in identifying and optimizing slow queries and performance anomalies but also offers a unified view of data that enhances debugging and insight generation.

The latest ClickHouse release, version 25.8, introduces a host of improvements, including 45 new features, 47 performance optimizations, and 119 bug fixes, significantly enhancing its functionality and efficiency. A key highlight is the introduction of a faster native Parquet reader, which improves parallel processing and I/O efficiency, nearly doubling query speeds for Parquet files. Additionally, the update supports Hive-style partitioning, improves Data Lake functionalities with enhanced Iceberg and Delta Lake support, and offers initial PromQL integration for time-series queries. The Arrow Flight integration further broadens ClickHouse's compatibility with columnar data ecosystems, allowing efficient data exchange. Security features for S3 storage have been enhanced, and latency issues with Azure Blob Storage have been mitigated through the implementation of an aggressive retry logic, similar to the solutions applied to AWS and GCP. The release also welcomes a new cohort of contributors, reflecting the growing and supportive ClickHouse community.

ClickHouse, an open-source columnar database management system, is designed to execute GROUP BY queries with exceptional speed, transforming the capabilities of analytical workloads. Initially developed to handle data filtering and aggregation rapidly on a single node, ClickHouse has evolved to support complex, large-scale queries across distributed computing environments. Its architecture leverages parallel processing streams across CPU cores, utilizing columnar storage and vectorized execution for efficient data handling. The system's scalability is further enhanced through parallel replicas, a feature currently in beta, allowing queries to be distributed over multiple nodes in the cloud with shared storage, thereby achieving interactive speeds on massive datasets. This innovative approach facilitates dynamic load balancing and efficient data processing without the need for data reshuffling, making ClickHouse a powerful tool for modern data analytics in both single-node and cloud environments.

ClickHouse Cloud introduces a new feature called parallel replicas, enabling unprecedented horizontal query scaling by distributing a single query across thousands of cores. This feature allows complex queries, such as GROUP BY operations, to be executed on datasets as large as 100 billion rows in under half a second without pre-aggregation or data reshuffling. The technology achieves this by treating multiple nodes as virtual replicas, allowing for elastic scaling: as more nodes are added, query speed increases without data movement. The parallel replicas, currently in beta, enhance ClickHouse's capability to handle massive data throughput efficiently and are poised to become a benchmark for analytics performance, offering interactive speed across extensive datasets. This advancement allows ClickHouse to maintain its foundational goal of rapid data aggregation while extending its reach from single-node efficiency to cloud-scale operations, providing users with the flexibility to scale workloads seamlessly from individual nodes to large, distributed clusters with minimal configuration changes.