November 2024 Summaries

ClickHouse has announced the private preview of its Postgres Change Data Capture (CDC) connector in ClickPipes, allowing customers to replicate their Postgres databases to ClickHouse Cloud with just a few clicks. The integration eliminates the need for external ETL tools and is natively integrated into ClickHouse Cloud through ClickPipes. The Postgres CDC connector was built in collaboration with several customers and design partners who are already running production-grade workloads. Key benefits include blazing fast performance, super simple setup, support for native features of both Postgres and ClickHouse, enterprise-grade security, and no vendor lock-in. Customers can sign up for the private preview by following a link provided in the announcement.

Attentive, an AI-powered marketing platform, has transitioned from a fragmented, multi-database setup to a streamlined solution powered by ClickHouse Cloud. The company's rapid growth and use of machine learning models necessitated simplifying their data architecture and migrating to a single database for improved scalability and handling larger data loads. After understanding the problem and choosing the right technology, Attentive tested ClickHouse with real-world demands before implementing it in their infrastructure. The migration has resulted in faster query completion times, reduced costs, and better scalability, allowing them to focus on feature development and expansion of their platform.

The November ClickHouse newsletter highlights significant developments in real-time data warehouses over the last month. Key points include the production readiness of Refreshable Materialized Views, an official Docker image, and a deep dive into the new JSON data type. Other topics covered are interviews with thought leaders, simplifying queries using dictionaries, building financial data pipelines, live updates on ClickHouse Cloud, and quick reads about various applications and uses of ClickHouse. The newsletter also invites readers to visit ClickHouse at AWS re:Invent and mentions the upcoming events in AMER and EMEA regions.

Refreshable Materialized Viewsが本番運用可能になったことと、公式のDockerイメージが公開されたことが、この月のClickHouseニュースに注目される主要な出来事です。Alexey MilovidovはData Talks on the Rocksにゲスト出演し、新しいJSONデータ型を実装した方法や、Alpha VantageとClickHouseで構築する金融データパイプラインについても紹介しています。また、バージョン24.10のリリースの新しい機能や、ClickHouse Cloudライブアップデートの内容なども含まれています。

Skool, an all-in-one community platform founded in 2019, has adopted ClickHouse as its database solution to support the company's growing user base and provide real-time observability, experimentation, and behavioral analytics. Initially using Postgres for both transactional and analytics workloads, Skool experienced bottlenecks due to running both workloads on the same technology. After evaluating different options, they chose ClickHouse, a columnar OLAP database management system, which brought immediate improvements in query performance. Over the past two years, Skool has expanded its use of ClickHouse for various purposes, including real-time observability through Grafana integration and behavioral analytics to monitor user behavior across key stages of the user journey. The adoption of ClickHouse has provided faster query times, enabling the team to focus on innovation and improving the user experience while ensuring scalability as data volumes increase.

ClickHouse is now officially supported as a data source for Microsoft Power BI, one of the leading business intelligence platforms globally. The ClickHouse team collaborated with Microsoft to develop the Power BI Connector, allowing users to easily query their ClickHouse instances regardless of whether they use ClickHouse Cloud or manage their own instances. This integration makes it simple to create interactive dashboards and charts based on data stored in ClickHouse using various Power BI flavors such as Desktop, Service, and Mobile applications. To get started with the ClickHouse Power BI connector, users need to install the ClickHouse ODBC driver from the official GitHub repository and follow the instructions provided in the documentation. The team encourages users to share their experiences and feedback on GitHub.

Huntress, a managed security platform founded by ex-NSA cyber operators in 2015, aims to make enterprise-level security accessible to small and mid-sized businesses. The company's ability to analyze and act on massive volumes of data is central to its success. In September 2024, co-founder and CTO Chris Bisnett explained how switching to ClickHouse Cloud has helped Huntress optimize performance, address scaling challenges, and simplify their big data workflows while saving tens of thousands of dollars each month. By using ClickHouse, Huntress was able to improve its SIEM solution in 6-8 months, cutting costs by more than 90% while improving performance. The new setup easily processes up to 200,000 records per second, allowing Huntress to scale as its customer base grows.

Tydo is a customer intelligence platform that helps ecommerce brands analyze their data to gain actionable insights into customers and purchasing patterns. The company initially used Postgres but switched to ClickHouse due to its ability to handle high-dimensional datasets while delivering sub-second query performance. Since adopting ClickHouse, Tydo has seen significant improvements in both performance and scalability, allowing clients to access data much faster and make swift, data-driven decisions. The new infrastructure's high-throughput performance also reduces latency and operating expenses, making it a cost-effective solution for large ecommerce brands. With ClickHouse as the backbone of its customer segmentation and AI applications, Tydo is well-positioned to continue scaling and delivering value to clients worldwide.

Temporal, a durable execution platform, has built an innovative observability system called Chronicle with the help of ClickHouse. The team chose ClickHouse due to its advantages over traditional time-series databases and reputation as a "pretty spectacular piece of technology." Since implementing Chronicle with ClickHouse, Temporal has seen significant improvements in ingestion speed, query performance, and data compression capabilities. As they continue to innovate, the team is confident that ClickHouse will remain a cornerstone of their observability strategy, supporting the platform's continued growth and enabling efficient handling of ever-increasing data volumes.

Weights & Biases (W&B) is an AI developer platform founded in 2017 that aims to help engineering teams overcome the challenges of taking AI projects from impressive prototypes to reliable production systems. The platform provides tools for tracking experiments, versioning datasets, and managing machine learning workflows, making AI development more efficient and reproducible. W&B relies on ClickHouse's high-performance, low-latency capabilities to handle the large data volumes that come with scaling AI projects for production. As more teams develop AI models, there is an expectation that they can seamlessly move these models into production to create real-world impact. However, this transition often proves difficult due to gaps in developer tools for GenAI operations. W&B addresses this issue by providing tools for tracking experiments, versioning datasets, and managing workflows, making AI development more accessible and efficient.

SQL-based observability with ClickHouse has matured significantly over the past year, making it a viable option for managing large volumes of logs, traces, and metrics data. The recent addition of JSON support in core ClickHouse, combined with improvements to the OpenTelemetry collector and Grafana plugin, have made SQL-based observability more accessible and efficient. Key takeaways from this blog post include: 1. SQL-based observability with ClickHouse is now a viable option for managing large volumes of logs, traces, and metrics data. 2. The recent addition of JSON support in core ClickHouse has made SQL-based observability more accessible and efficient. 3. Improvements to the OpenTelemetry collector and Grafana plugin have also contributed to the maturation of SQL-based observability with ClickHouse. 4. Users should identify their query access patterns early and choose a sensible primary/ordering key for their tables to ensure optimal performance. 5. Materialized views are essential in meeting performance expectations and transforming OTel data to fit users' access patterns. 6. Time series engines like ClickHouse can be used as the storage backend for Prometheus metrics, offering scalable, high-performance storage with native multi-node support in a memory-efficient architecture with support for high cardinality data. 7. Ongoing improvements are planned for the time series table engine to unlock more of ClickHouse's performance potential within Prometheus, providing a more efficient, scalable solution for robust, long-term metrics storage and analytics.

OpenTelemetry (OTel) is an open-source framework from the Cloud Native Computing Foundation (CNCF) that enables standardized collection, processing, and export of telemetry data. It supports multiple languages and provides instrumentation libraries for efficient observability in complex systems. The vendor-neutral approach allows teams to avoid lock-in with any single monitoring tool and switch between or combine different observability backends as needed. ClickHouse has decided to officially support and contribute to the OpenTelemetry exporter, which is now in beta for both logging and tracing. This integration helps teams collect telemetry data efficiently and gain insights into complex systems using SQL-based observability tools like ClickHouse. The OTel collector gathers telemetry data from SDKs or other sources and forwards it to supported backends, such as ClickHouse. The default schema provided by the OpenTelemetry exporter for logs, traces, and metrics is suitable for most common telemetry use cases. However, if users need a significantly different schema, they can create their own table schemas or use a ClickHouse materialized view to modify the existing ones. It's recommended to disable default table creation in production deployments to avoid conflicts between multiple exporter processes. The OpenTelemetry community encourages contributions from users and developers, including feedback, bug reports, feature suggestions, code improvements, and documentation updates. Contributions can directly impact and shape the future of observability tools like OTel+ClickHouse.

ClickHouseチームは、OpenTelemetryの公式サポートと貢献を開始しました。これにより、ClickHouse用のOpenTelemetryエクスポーターが提供されます。OpenTelemetryはCloud Native Computing Foundationから提供されるオープンソースフレームワークで、テレメトリーデータの標準化された収集、処理、エクスポートを可能にします。オブザーバビリティの分断されたベンダー主導の状況が支配していますが、OpenTelemetryは標準化された柔軟なアプローチをもたらします。これにより、開発者と運用チームがシステムの健康状態を把握し、分散システムの問題を診断できるようになります。 ClickHouseは、SQLベースのオブザーバビリティスタックの重要なコンポーネントです。 OpenTelemetry CollectorはSDKや他のソースからテレメトリーデータを収集し、サポートされているバックエンドに転送します。 ClickHouse用のOpenTelemetryエクスポーターはClickHouse管理者が提供したものであり、LogHouseとして使用されています。これにより、OpenTelemetryとClickHouseの統合が可能になります。 OpenTelemetryデータのスキーマに関する問題は、ClickHouseのテーブルスキーマをモデル化する際に受け入れるべき厳しい現実です。 ClickHouseエクスポーターには改善の余地がありますが、新しい最適化が見つかれば、ログ、トレース、メトリクスのデフォルトスキーマを改善できる可能性があります。 OpenTelemetryへの貢献は、コミュニティの協力的な力にあります。 コミュニティのメンバーになることを検討しても良いでしょう。 OpenTelemetryプロジェクトと頻繁にやり取りしている場合、その組織のメンバーになることを検討しましょう。

ClickHouse version 24.10 introduces 25 new features and 60 bug fixes, including improvements to clickhouse-local, table cloning, caching remote files, and production-ready Refreshable Materialized Views. The JSON data type has been rewritten for high performance handling of JSON data, making ClickHouse a powerful alternative to specialized document databases like MongoDB. Additionally, the latest version offers enhanced MongoDB integration support with improved querying capabilities.