Unordered mode for S3 ClickPipes is now available

ClickHouse Cloud introduces a new feature for ingesting data from Amazon S3, allowing files to be processed in any order using S3 Event Notifications via Amazon SQS, thus overcoming the previous requirement of lexicographical order. This advancement enhances the functionality of ClickPipes by providing a more robust and efficient method for handling unordered data, backfills, retries, and late-arriving files. The new mode employs a more dynamic approach by listening for ObjectCreated events in an SQS queue, which facilitates the immediate processing of new files without the need for constant polling. This method ensures exactly-once data ingestion semantics, preventing duplicates even in case of reprocessing. Setting up this system involves configuring an SQS queue, creating an IAM role, and establishing ClickPipe connections to S3, with the potential for further automation through ClickHouse's Terraform provider. This feature is currently exclusive to Amazon S3, with plans to extend support to Google Cloud Storage and Azure Blob Storage, promising enhanced flexibility and performance for data integration pipelines.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.