Reliable ingestion from AWS S3 using Apache Hudi™
Blog post from Onehouse
A new deltastreamer source has been developed to reliably and efficiently process new data files arriving in AWS S3 by using a log-based approach that leverages change notification and incremental processing, addressing the limitations of the previous method that relied on modification time precision. This new system, comprising the S3EventsSource and S3EventsHoodieIncrSource, utilizes AWS services such as SNS and SQS to capture and manage file events, thus ensuring all data events are eventually processed while providing fault tolerance and enabling asynchronous backfills. Designed for performance and scalability, the approach avoids bottlenecks like directory listings and allows for decoupled operation steps, which can be run independently or in tandem. Users need to configure specific AWS SQS settings and enable S3 event notifications to set up the pipeline, which is adaptable to other cloud storage solutions like Google Cloud Storage and Azure Blob Storage in the future. This system not only enhances reliability and efficiency but also offers flexibility for automatic backfills and resource management.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.