June 2024 Summaries

The ClickHouse Java client is being refactored to improve developer experience and prepare for future extensions. Issues with excessive complexity for simple operations, complex data insertion and retrieval, and unsafe low-level optimizations have been addressed in the new API. Key goals of the refactoring include an intuitive API, improved documentation, and cleaning up the code base. The new client is available in its early alpha release, and feedback from users is encouraged to further improve the product.

Tekion, an automotive software company founded by ex-Tesla CIO Jay Vijayan, has successfully implemented ClickHouse Cloud for its internal observability stack. The implementation resulted in significant improvements in storage optimization, ingestion performance, and query performance. Tekion's adoption of ClickHouse Cloud allowed the company to reduce its storage requirements by 10x, process up to 1.2 million records per minute without lag, and achieve faster query execution times with lower latency. The successful integration of ClickHouse Cloud into Tekion's platforms has enabled real-time data analysis and insights, helping the company adapt to evolving customer needs and market trends.

ClickHouse has announced the General Availability of ClickHouse Cloud on Azure. The architecture of ClickHouse Cloud is designed to be as cloud service provider (CSP) neutral as possible, allowing it to be reused across different CSPs such as AWS, GCP, and Azure. The Control Plane is built in AWS and interacts with the Data Plane in AWS, GCP, and Azure. ClickHouse Cloud's components are best described as two different and independent logical units: the Control Plane (the user-facing layer) and the Data Plane (the infrastructure-facing part). The architecture is designed to be CSP agnostic, but each individual CSP has its own quirks and differences. One challenge faced in adapting the architecture to Azure was the way cloud resources are organized. To address this issue, a solution was designed where a pool of Azure subscriptions for customer ClickHouse clusters is created, and whenever a new instance is created by a customer, it is randomly assigned to a subscription picked from the pool. This design works around the resource limit issue mentioned earlier. The next section will explain how they addressed these problems using cellular architecture.

ClickHouse Inc has expanded its real-time analytics service, ClickHouse Cloud, to Microsoft Azure, making it available on three major cloud platforms. This expansion offers users greater flexibility and choice in deploying and managing their analytics workloads. With availability on Azure Marketplace, customers can leverage existing Azure investments and benefit from integrations with Azure-specific services such as Microsoft Power BI and Azure Event Hubs. The ClickHouse Cloud is designed for handling real-time analytics on top of petabyte-scale datasets and offers features like simplicity, speed and scale, security and reliability, efficiency, and ecosystem integration. A 30-day trial with $300 free usage credits is available to new users.

The June ClickHouse newsletter highlights recent developments in real-time data warehouses. Key points include the introduction of the Dynamic type in the 24.5 release, performance improvements for CROSS JOINs, and functionality to read into archive files on S3. Michael Shi from HyperDX explains why they chose ClickHouse over Elasticsearch for observability data, citing its columnar data layout and sparse indexes as key differentiators. Tom Weisner provides a tutorial on using Python User-Defined functions in ClickHouse, while Atheer Alabdullatif from Tweeq discusses their choice of ClickHouse and other tools for their data platform. The newsletter also covers the use of ClickHouse to count unique users at scale and its role in ETL/ELT processes.

The company has launched a ClickHouse Open Source Software (OSS) Trust Center, which provides links to their Bug Bounty program, policies, and configurations. Subscribers will receive notifications of vulnerability fixes and can participate in the Embargo Notification Program for high-risk vulnerabilities. The company is also part of the CVE Numbering Authority (CNA) program to ensure vulnerability information is available through the CVE database with clear descriptions and remediation instructions.

Common Room, a customer intelligence platform founded in 2020, has integrated ClickHouse into its data management system to handle the growing complexity and volume of customer data. The company uses an AI-powered platform that aggregates data from various sources to provide GTM teams with a unified view of customers across digital channels. Initially relying on Postgres for both transactional and analytical datasets, Common Room switched to ClickHouse due to its ability to handle frequent updates without sacrificing performance. The adoption of ClickHouse has significantly improved the platform's efficiency in processing complex analytical queries on recent and historical data.

ClickHouse Cloud has introduced horizontal scaling in private preview, allowing users to control the number of replicas in their service according to workload demands. This feature is available for production instances and can be accessed via ClickHouse Cloud APIs or soon through the cloud console. Horizontal scaling differs from vertical scaling by adding more machines instead of increasing resources on a single machine, providing better fault tolerance and faster scaling. The horizontal scaling capability will improve in the future with self-service options and horizontal autoscaling based on workload demands.

Southeast Asia's leading ecommerce platform, Shopee, has successfully implemented distributed tracing using ClickHouse as an OLAP database management system to enhance its data analytics capabilities. The adoption of ClickHouse has supported the growing demands of internal teams efficiently and cost-effectively. Distributed tracing provides real-time insights into how data flows across the ecommerce platform, allowing hundreds of internal users to pinpoint exactly where delays or failures occur, make informed decisions, and implement fixes that enhance the platform's overall performance and reliability.

Peter Marton, CEO and Co-Founder of OpenMeter, discussed his company's data ingestion approach at a recent ClickHouse San Francisco meetup. OpenMeter helps AI and cloud companies adopt usage-based pricing models by providing a scalable metering system capable of handling millions of billable events per second. The new architecture is based on Kafka for event buffering, modified ClickHouse Kafka Connect Sink to handle deduplication, and ClickHouse Materialized Views with AggregatingMergeTree table engine for aggregation and data storage. This approach allows OpenMeter to meet the evolving demands of their customers in real-time analytics use cases.

Trip.com built a centralized logging platform using Elasticsearch in 2012 to address challenges posed by decentralized logging systems. However, as data volume increased, they faced issues with cluster stability and high costs. In search of an alternative solution, they migrated their storage and query layer to ClickHouse due to its columnar storage, vectorized query execution, high compression rates, and insertion throughput. The migration involved creating a new table design, setting up a similar architecture as Elasticsearch with master nodes (ClickHouse-Keeper instances), multiple query nodes, and data nodes, customizing Kibana for ClickHouse integration, and developing Triplog for managing tables, users, and roles. After the migration, they achieved significant storage space savings, faster query performance, and reduced operational costs. As their platform grew to 20PB, they faced new challenges such as performance issues due to poorly written queries and poor index choices by users. To address these issues, they moved away from a single cluster approach, implemented virtual table partitioning for efficient routing of queries, and used Antlr4 SQL parsing for query optimization. They also started exploring ClickHouse Enterprise Service (via Alibaba Cloud) to handle their high peak usage more gracefully.

ClickHouseとElasticsearchは、各自で強固なデジタルサービスの提供を目指している。 ClickHouseは、SQLに依存し、列指向ストレージ、ベクトル化クエリ実行、高圧縮率、高挿入スループットなど、多くの機能を備えています。これらは、私たちのログソリューションが求めるパフォーマンス、安定性、費用対効果を満たすものでした。 ClickHouseへの移行により、ストレージスペースの節約が50%以上に達し、既存のElasticsearchサーバーでClickHouseを用いてデータ量を4倍に増やせるようになりました。クエリパフォーマンスではElasticSearchよりも4倍から30倍速く、P90は300ms未満、P99は1.5秒未満です。 ClickHouse Enterprise Serviceのテストにより、エンタープライズエディションがコミュニティエディションより高並行ワークロードを処理できることが明らかになりました。これは、私たちにとって大きな運用負荷を取り除く可能性があると考えています。

The text discusses the author's experience with ClickHouse, a columnar database designed for real-time data and analytics. It emphasizes the importance of learning ClickHouse due to its growing popularity and unique capabilities compared to other databases. The author provides tips on how not to learn ClickHouse, such as assuming it is similar to relational databases like Postgres. Instead, they recommend using resources provided by ClickHouse, including documentation and training programs. They also suggest taking breaks during learning and becoming comfortable with the documentation for better understanding of functions and syntax. Finally, the author shares tips on preparing for the ClickHouse Developer certification exam.

ClickHouse version 24.5 introduces 19 new features, 20 performance optimizations, and 68 bug fixes. The release includes a dynamic data type for semi-structured data, the ability to read from archive files on S3, improvements in CROSS JOIN memory usage, and experimental support for non-equal conditions in JOIN ON clauses. Additionally, ClickHouse extends its functionality by allowing reading from ZIP files containing CSV files stored on S3. The new version also introduces several performance optimizations and bug fixes to improve the overall user experience.