January 2025 Summaries

Electrum, Indonesia's leading electric motorcycle company, is shifting towards low-carbon, sustainable transport. The company's rapid expansion has triggered a massive data surge from IoT sensors and transactional data. To address this challenge, Electrum adopted ClickHouse, a columnar storage database that enables lightning-fast queries, processing only relevant data. This solution provides flexibility, efficiency, and scalability, allowing business users to query raw and aggregated data independently. With materialized views, Electrum can quickly summarize and analyze IoT data points in real-time, reducing processing time by up to 90%. The company has seen significant improvements in cost efficiency, with costs under $500 per month, and plans to scale its implementation to a multi-server setup. Electrum's journey showcases the power of the right database in solving complex challenges at scale, supporting their growing fleet, optimizing performance, and delivering real-time business insights.

We've created a tool called kubenetmon to monitor data transfer in ClickHouse Cloud, which provides visibility into network behavior and helps organizations forecast cloud costs. The solution collects connection data from conntrack, attributes connections to endpoints using Kubernetes informers, and stores the records in a SummingMergeTree table in ClickHouse, enabling efficient querying and analysis of network traffic patterns. By open sourcing kubenetmon, we aim to provide a valuable resource for teams managing complex cloud infrastructure.

ClickHouse consistently outperformed all other tested data stores with JSON support, both in storage efficiency and query performance. For analytical queries, it's not just faster — it's thousands of times faster than leading JSON data stores like MongoDB, thousands of times faster than DuckDB and PostgreSQL, and orders of magnitude faster than Elasticsearch. ClickHouse achieves this level of performance while maintaining storage efficiency — JSON documents in ClickHouse are even more compact than compressed files on disk, resulting in lower cost of ownership for large-scale analytical use cases. Using the ClickHouse native JSON data type gives you the best of both worlds – fast analytical queries and optimal compression on disk, without requiring upfront schema design and refinement. This makes ClickHouse an unparalleled general-purpose JSON data store—especially for use cases, where events are often in JSON format and cost efficiency and analytical query performance are critical, such as SQL-based observability.

I can provide a neutral, objective, and interesting summary of the text in one paragraph. The One Billion Documents JSON Challenge compares the performance of five data storage systems - ClickHouse, MongoDB, Elasticsearch, DuckDB, and PostgreSQL - when handling massive amounts of JSON data. The challenge uses a dataset with 10 billion rows, which is divided into eight subsets to test each system's efficiency. The results show that ClickHouse outperforms the other systems in both storage size and query performance. Specifically, it reduces storage size by up to 99% compared to Elasticsearch, which requires over 360 GB of disk space. Additionally, ClickHouse executes queries faster than the other systems, with a cold run time of just under 1 second for each query. The study highlights the benefits of using a native JSON data type in ClickHouse, which allows for efficient storage and processing of large amounts of JSON data.

ClickHouse Cloud has introduced compute-compute separation, a powerful approach to optimizing database performance and resource management by isolating compute resources for specific workloads, users, or business functions. This method allocates dedicated compute instances to different types of database operations, reducing the risk of interference between them. The introduction of warehouses in ClickHouse Cloud provides multiple benefits, including isolating workload performance, providing different compute sizes and idling settings, separate billing for each team, and improved high availability levels for different workloads. Compute-compute separation is particularly useful in high-demand environments where fluctuating workloads might otherwise affect query speed and reliability. The feature is designed to provide better tenant isolation and optimize resource consumption and costs.

ClickHouse Cloud was launched in December 2022, with over a thousand companies signing up for its managed service. Since then, it has been continuously improved with new features and capabilities added to meet evolving user needs. The platform now supports compute-compute separation, allowing multiple replicas to access the same data storage concurrently, and introduces a new mechanism for vertical scaling of compute replicas called "make before break" (or MBB). This results in more seamless scaling operations that are less disruptive to running workloads. Additionally, ClickHouse Cloud has introduced improved managed upgrades, single-replica services, performance and efficiency improvements, and an Enterprise Tier with advanced security and compliance features. The platform also offers enhanced support services, including a dedicated Technical Account Manager role, and will be restructuring its current Development and Production tiers to better match customer usage patterns.

ClickHouse has migrated its infrastructure from x86-based instances to AWS Graviton processors, leveraging the advantages of ARM architecture for better price-performance and energy efficiency. The migration aimed to improve performance without increasing costs. ClickHouse achieved significant improvements in query performance, with up to 76% better performance on certain queries, and reduced operating costs. The company has comprehensive AArch64 support, including automated builds, full functional test runs, JIT compilation, ARM Neon optimizations, query profiling, and introspection tools. The migration involved careful planning and execution, including compatibility assessments, benchmarking strategies, risk analysis, and mitigation measures. ClickHouse's journey with ARM architecture demonstrates the potential of ARM-based solutions for data-intensive workloads.

The key points of the text are that modern cloud database services like ClickHouse Cloud can utilize computer-computer separation, which separates compute resources for specific workloads, users, and business functions to optimize database performance and resource management. This approach reduces the risk of interference between different types of database operations. The introduction of a new concept called "warehouses" in ClickHouse Cloud allows multiple services to be grouped together and provides flexibility in managing resources. Weights are applied to each warehouse to enable the user to choose how much weight they want to assign to each warehouse. The warehouses can also have different settings, such as the number of nodes and node size, which can be adjusted independently for each service. Additionally, users can create separate read-only services that do not perform background maintenance tasks, reducing the impact on performance. However, there are some limitations to consider when using this feature, such as the fact that the first service in a warehouse cannot be idled or stopped. Overall, computer-computer separation and warehouses provide a powerful way to manage resources and improve database performance in ClickHouse Cloud.

ClickHouse Cloudは、2022年12月にリリースされた、驚異的な速さで構築され、すでに1,000社以上がマネージドサービスにワークロードを移行している、クラウドベースのデータベース管理サービスである。新しい機能として、ClickPipes、PeerDB、Arctype、マネージドアップグレード、BYOC (Bring-Your-Own-Cloud) デプロイオプションが追加され、コンピュート同士の分離、自動スケーリング、パフォーマンスと効率の向上、先進的なセキュリティとコンプライアンス、高度なディザスタリカバリ、ハードウェアとリージョンの高度なプロファイルが導入された。エンタープライズ向けの新プラン「Enterprise Offering」も導入され、厳しい要件を持つユーザー向けにセキュリティやコンプライアンス機能、ハードウェアやアップグレードに対する制御を提供する。

The ClickHouse Cloud Audit add-on for Splunk enables users to store and analyze ClickHouse Cloud audit logs directly into Splunk, facilitating easier integration into any environment. The add-on uses the ClickHouse Cloud API to securely pull audit logs from a user's organization, and its installation is straightforward requiring only a few steps. Once installed, users can gather required information such as Organization ID and Admin API Key, configure data input in Splunk, and start exploring audit logs through Splunk search and analytics tools.

Fuzzing has become a crucial research topic in software development, including databases, to identify issues such as crashes, bad output, and security vulnerabilities. ClickHouse is actively tested with various fuzzers, but there was a notable gap in their capabilities, particularly in generating complex queries while maintaining query correctness. To address this, BuzzHouse, a new fuzzer, has been developed over the past 5 months, which has already found about 100 new issues in ClickHouse. Fuzzing databases is challenging due to the complexity of database systems, including query processing and optimization, data storage, buffer management, and distributed processing. The key challenge lies in generating queries that cover a wide range of scenarios while ensuring query correctness. To overcome this, BuzzHouse employs various strategies, such as using syntax assumptions, simulating user behavior, and adapting to changing database features. By focusing on generating complex yet correct queries and identifying issues beyond simple crashes, BuzzHouse complements the existing suite of tools used to test databases, including AFL and libFuzzer for code coverage-guided fuzzing, SQLsmith for complex query generation, and SQLancer for query correctness. The development of BuzzHouse underscores its value in improving the robustness and reliability of databases like ClickHouse, with over 100 new issues already uncovered.

AMP is an ecommerce growth platform that offers analytics tools to Shopify merchants, helping them analyze and optimize their store's performance. However, as the company grew, its data infrastructure became complex and volume-intensive, leading to issues with real-time insights and customer expectations. To address these challenges, AMP migrated to ClickHouse Cloud, a scalable, long-term solution that enables real-time streaming, event replay, and automated infrastructure management. This move has simplified operations, created a strong foundation for growth, and provided merchants with timely analytics to make data-driven decisions. With the new setup, AMP can focus on innovation rather than upkeep, staying true to its commitment to customer obsession in today's ecommerce landscape.

The text discusses IBM Instana's challenge in monitoring the performance of its ClickHouse database without adding complexity or slowing it down. To address this issue, the team built a custom span exporter using OpenTelemetry, which captures and analyzes telemetry data from ClickHouse operations. The exporter is designed to be efficient, non-invasive, and easy to configure, with features such as automatic reconfiguration, preprocessing of spans, and support for metrics and profiling. By integrating the custom span exporter into its architecture, Instana can now monitor ClickHouse's performance in real-time without disrupting its role as a foundation for observability, providing better insights for DevOps teams and customers alike.

The ClickHouse newsletter for January 2025 covers various topics, including the release of version 24.12, which introduced support for the Apache Iceberg REST catalog and schema evolution. The newsletter also features articles on building a product analytics solution with ClickHouse, optimizing bulk inserts for partitioned tables, and exploring the Medallion architecture with ClickHouse. Additionally, there are videos from the All Things Open Conference, including talks by experts in data engineering and analytics. The newsletter highlights the work of community members, such as Jason Anderson, who shares his experience building a product analytics platform using ClickHouse. Various training events and meetups are also listed, including free training sessions and events in different regions. The newsletter concludes with a "Post of the month" section featuring an article by Dmytro Shevchenko.

The text discusses the use of Evidence.dev, a lightweight framework for building data applications and business intelligence products using SQL. It showcases how to create a dashboard to explore the BlueSky dataset by running queries at build time rather than run time, which improves page loading time and interactivity. The process involves creating a project, installing Evidence, configuring a connection to ClickHouse, writing queries, running them, and building an Evidence dashboard with multiple pages. Finally, it highlights deploying the project to Evidence Cloud or serving it locally using an HTTP server. The framework's approach of static-site generation and its ease of use make it a practical choice for creating fast-loading, interactive dashboards.

Increff, a retail SaaS company, was facing challenges in managing its complex supply chains and optimizing inventory with its previous business intelligence (BI) platform. The platform struggled to handle massive datasets of over 200 million records, causing query performance issues and bottlenecks. To address this, Increff reimagined its BI platform using ClickHouse Cloud, a scalable and high-performance data architecture that integrates with over 90 data sources. By adopting a new star schema design, partitioning, and optimized data types, Increff achieved sub-second query speeds and reduced data preparation times by 70-80%. The company is now exploring advanced features like projections, materialized views, and AI-driven tools to further improve performance and user experience. With the new architecture, Increff can support the world's leading retail brands in making smarter, faster decisions with confidence.

ClickHouse Cloud has emerged as a strong contender for m3ter's usage data workload, offering improved performance and cost savings compared to other database solutions. With its column-store architecture, ClickHouse Cloud is well-suited for handling large volumes of data and providing fast query performance. The service offers various features such as real-time monitoring, automated scaling, and high availability, making it an attractive option for companies like m3ter. By using ClickHouse Cloud, m3ter has achieved significant cost savings, with a reduction of around 85% compared to their previous database solution. Additionally, the service provides easy-to-use UI and user management controls, giving users sensible access levels and reducing the risk of data breaches. Overall, ClickHouse Cloud's performance, scalability, and cost-effectiveness make it an ideal choice for m3ter's usage data workload.

ClickHouse version 24.12 introduces several new features, including Enum usability improvements, Iceberg REST catalog and schema evolution support, reverse table ordering, and the ability to use JSON subcolumns as a primary key. The new release also improves JOIN performance with parallel hash join by default, automatic join reordering, optimization of JOIN expressions, and non-equi JOINs supported by default. Additionally, ClickHouse's new powerful JSON implementation stores the values of each unique JSON path in a true columnar fashion, allowing high data compression and maintaining query performance seen on classic types. The release also welcomes several new contributors to the ClickHouse community.