May 2022 Summaries

Apache Hudi 0.11.0 introduces a pioneering multi-modal indexing subsystem for Data Lakehouse architecture, designed to enhance query and write performance, particularly for large datasets. This new indexing system reimagines traditional indexing to accommodate the unique needs of Hudi's mutable change streams and diverse write patterns, aiming to bring significant performance improvements at the scale of data lakes. The implementation includes a scalable metadata table using the HFile format from HBase, which offers faster point lookups compared to formats like Parquet and Avro. The system supports a variety of indexes, such as files, column_stats, and bloom_filter indexes, and is designed to be extensible, allowing for future enhancements like bitmap and R-tree-based indexes. Asynchronous indexing and ACID transactional updates ensure that data integrity is maintained without impacting write latency. The multi-modal index improves file listing and data skipping, drastically reducing latency and I/O operations, while future developments aim to introduce more sophisticated indexing capabilities for large-scale deployments.