May 2023 Summaries

Apache Hudi, originally developed by Uber in 2016, is a framework that enhances data lakes by efficiently managing transactional data changes, making it a valuable tool for rapidly evolving data environments. Hudi's design includes five key record-level meta fields that provide unique identification, spatial and temporal distribution, and enable functionalities such as tracking changes, ensuring data integrity, and supporting operations like time-travel queries. These meta fields, while offering significant advantages like preventing duplicates, aiding in debugging, and optimizing storage operations, come with some storage overhead, which is efficiently managed through compression techniques. The benefits of using Hudi, such as improved data integrity and reduced compute costs, often outweigh these overheads, as demonstrated by Uber's use case, where they achieved significant cost savings in their data pipelines.

The new Onehouse PostgreSQL Change Data Capture (CDC) feature allows for the continuous streaming of data into a data lakehouse, utilizing open-source tools like Apache Kafka, Debezium, and Apache Hudi. This feature is designed to help organizations replicate data from OLTP databases to data lakehouses in near real-time, which is crucial for data-driven decision-making and unlocking critical business use cases. Onehouse offers a fully managed, cloud-native platform that automates data management tasks, reducing the time and expertise needed to build and maintain data infrastructure. This secure architecture ensures data remains within a customer's cloud account, providing peace of mind regarding data privacy and compliance. The platform also supports advanced table optimization services to enhance performance and cost-effectiveness, allowing users the flexibility to select their preferred query engines. Onehouse's CDC feature simplifies the setup of a data pipeline, enabling organizations to adapt quickly to changing business demands while maintaining data integrity and operational efficiency.

Apache Hudi is gaining traction among developers for its robust capabilities in managing large-scale datasets, particularly through the efficient execution of upserts. The choice between its two storage table types—Copy-On-Write (COW) and Merge-On-Read (MOR)—plays a crucial role in optimizing upsert performance. COW tables are simpler operationally and offer zero read amplification but come with higher write amplification due to synchronous merging. Conversely, MOR tables are more complex but are write-optimized, as they handle updates through log files that are later compacted, reducing write amplification but increasing read complexity. The selection of an indexing strategy is also vital, as Hudi's indexing layer minimizes full-table scans during upserts, with options ranging from global to non-global indexes like Bloom, Simple, and Bucket indexes, each catering to different workload requirements. Partitioning strategy impacts both query and ingestion performance, where overly granular partitions can lead to inefficiencies. The blog emphasizes tailoring configurations and tuning Hudi to specific workload types to maximize upsert efficiency, advising a balance between partition granularity and indexing choices.

Data streaming has become essential for modern businesses requiring real-time data processing, with Confluent's managed Apache Kafka platform and Onehouse's managed data lakehouse on Apache Hudi offering combined solutions that enhance data ingestion, management, and analytics. The process involves using Confluent Kafka to track real-time ecommerce order events and integrating them into Onehouse for analytics, using two main approaches: directly ingesting data into a mutable table for real-time updates and employing a multi-stage pipeline for historical data retention. Confluent's capabilities, such as the Datagen Source connector and Schema Registry, facilitate seamless data streaming while Onehouse's integration allows for efficient schema validation and data quality management. The final setup enables the data to be queried via engines like Amazon Athena, with possibilities for further data transformation and visualization using tools like Hex. This integration significantly reduces the time and resources required for managing data streaming infrastructure, allowing businesses to focus on leveraging their data for insights and growth.

In September 2020, the data lakehouse concept began gaining traction, offering a blend of raw data storage and structured querying that departed from traditional data handling methods, but many engineers faced challenges in adapting due to limited documentation and integration questions. While exploring data lakehouse technologies, the Flink community identified a need for a system supporting changelog ingestion and efficient row-level updates, leading to attempts at integrating Flink with Apache Iceberg, which proved minimally beneficial due to Iceberg's batch processing roots. Consequently, an integration with Apache Hudi was pursued, offering ACID semantics and supporting efficient upsert and delete capabilities, resonating well with the Flink community for use cases like incremental ingestion of database changelogs and near-real-time materialization views. As of May 2023, the Hudi-Flink integration is utilized by over 100 companies and more than 3600 enterprises, with ongoing contributions from global communities to enhance low-latency and near-real-time data applications. The integration redesigned the index layer to overcome challenges associated with Bloom filter indexes in streaming workloads, leveraging Flink's stateful operators for efficient data processing. The integration allows for streaming ingestion with changelogs, incremental ETL pipelines, and the creation of incremental materialized views, showcasing the complementary strengths of both technologies.