Company
Date Published
Author
Dale McDiarmid
Word count
1457
Language
English
Hacker News points
None

Summary

We've simplified a real-time streaming use case using ClickHouse Cloud's new feature, ClickPipes, which reduces architectural complexity and minimizes maintenance overhead. ClickPipes is a fully managed ingestion service that connects to remote Kafka brokers, allowing users to leverage near-real-time data for insights and analytics. By utilizing ClickPipes, we can simplify the architecture used in a previous blog post, providing a more efficient and scalable solution for streaming data into ClickHouse Cloud. With ClickPipes, users can create pipelines with minimal code required, making it easier to integrate data from various sources into their ClickHouse services. The dataset used in this example is an Ethereum blockchain dataset made available by Google, which consists of four tables: Blocks, Transactions, Traces, and Contracts. The architecture uses Kafka Connect connectors to connect the public pub/sub topics to Confluent Cloud's infrastructure, reducing latency and providing a replay capability if required. ClickPipes simplifies this architecture further by allowing users to pull data from Kafka topics directly into their receiving tables, eliminating the need for deploying the Kafka Connector. By using ClickPipes, users can simplify their streaming architectures and take advantage of near-real-time data for insights and analytics without requiring extensive coding knowledge.