Home / Companies / ClickHouse / Blog / Post Details
Content Deep Dive

Unordered mode for GCS ClickPipes is now available

Blog post from ClickHouse

Post Details
Company
Date Published
Author
Get started today
Word Count
1,076
Language
English
Hacker News Points
-
Summary

ClickHouse Cloud now supports ingesting data from Google Cloud Storage (GCS) in any order using ClickPipes, enhancing flexibility and efficiency for event-driven analytics. Previously, files needed to be ingested in lexicographical order, but the new unordered mode, aided by Google Cloud Pub/Sub notifications, allows files to be processed as they arrive, regardless of order. This update addresses challenges like backfills and late-arriving data, ensuring that even files that don't follow a strict naming sequence are incorporated into analytics workflows. The unordered mode requires configuring Pub/Sub notifications for OBJECT_FINALIZE events and setting up a service account with the necessary permissions. With exactly-once processing guarantees, the new mode prevents duplicates, and failures are managed through automatic reprocessing. ClickHouse Cloud supports this feature through its Terraform provider, allowing for configuration as code, and plans to extend unordered ingestion capabilities to Azure Blob Storage, complementing existing support for Amazon S3.