Company
Date Published
Author
Dale McDiarmid
Word count
3965
Language
English
Hacker News points
3

Summary

This summary provides an overview of how Confluent Cloud's Custom Connector can be used to build a zero-code Kafka pipeline for moving data from Google Pub/Sub to ClickHouse Cloud, specifically for the Ethereum blockchain dataset. The process involves creating a Pub/Sub subscription, deploying the Confluent Cloud Custom Connector, and configuring it to send messages to a Kafka topic. The Kafka Connect Google Cloud Pub/Sub Source Connector reads messages from the Pub/Sub topic and writes them to the Kafka topic. A ClickHouse connector is then deployed to receive and transform inserts from the Kafka topic into the desired format for insertion into ClickHouse Cloud. The workflow demonstrates how this architecture can be used to maintain a public dataset in near real-time, with various hints and tricks provided for optimizing performance and troubleshooting issues.