June 2022 Summaries

Recent discussions have centered on evaluating the performance of Apache Hudi, Delta Lake, and Apache Iceberg for data lakehouses, with concerns raised about the transparency and reproducibility of existing benchmarks, such as those presented by Databeans using TPC-DS. The critique highlights issues with undisclosed configurations, inaccessible benchmarking code, and inappropriate methods for Hudi, which skewed the results. The authors argue for benchmarks that are reproducible, open, and fair, providing their own rigorous approach that includes disabling Spark's dynamic allocation for consistency and sharing their modifications to Delta's framework. They emphasize the necessity for documented and accessible benchmarks to ensure reliable comparisons and express plans to continue publishing internal benchmarks that showcase Hudi's performance capabilities.

Hudi 0.11 introduces advanced data layout optimization techniques, such as Z-order and Hilbert Space Filling curves, to enhance data skipping in large, frequently queried tables. Data skipping, a method of using indexes to bypass irrelevant data during queries, reduces data scanning and improves execution time. The new version incorporates Multi-modal Indexes like bloom-filter and column stats index within the Metadata Table, providing robust support and efficient implementation. The column stats index stores min/max values and other metrics in a compact format using HFile, enabling efficient data retrieval by avoiding unnecessary full-index scans. This approach significantly boosts performance for large datasets, as demonstrated with the Amazon Reviews dataset, and offers substantial cost savings for cloud-based data lakes. Future improvements are planned to extend data skipping support to Merge-On-Read tables, add caching, and further optimize performance.

The blog post explores the development of an asynchronous indexing mechanism in Apache Hudi, designed to enhance the lakehouse architecture by allowing for high-performance, non-blocking index creation and management. This new system permits the dynamic addition and removal of indexes without interrupting ongoing write operations, thereby improving both write latency and operational resilience. Key to this capability is the use of Hudi's transactional database kernel, which coordinates read and write operations while ensuring consistency through a hybrid concurrency control model. The asynchronous indexing process is divided into scheduling and execution phases, leveraging Hudi's timeline as an event log to coordinate actions and maintain data integrity. The design is scalable and adaptable to various index types, supporting both batch and streaming workloads, and represents a significant step towards more sophisticated data management in the lakehouse architecture. Future enhancements may focus on easing index management through increased asynchrony and further integration with SQL and additional index types, inviting community collaboration to refine and expand the system's capabilities.