December 2022 Summaries

Apache Hudi experienced significant growth and development in 2022, marked by a 38% increase in pull requests and a doubling of its Slack community to over 2,600 users, with a median response time to questions of just three hours. The project saw key partnerships and integrations with major platforms like AWS, Presto, Trino, DBT, and Google, enhancing its capabilities in data indexing and analytics. Notable advancements include the introduction of a multi-modal index for improved query performance, asynchronous indexing, and support for schema evolution in Spark SQL. Community engagement was robust, with virtual meetups and participation in global events such as Re:Invent and Data+AI Summit. Looking ahead to 2023, Apache Hudi plans to release version 0.13.0, featuring enhancements like record-key-based indexing and consistent hashing to optimize performance, alongside a vision of transforming streaming data lakes into mainstream solutions with sub-minute commit SLAs and improved query efficiencies.

Apache Hudi, developed by Uber in 2015, introduced the Merge-On-Read (MOR) table format as a solution to improve the inefficiencies of Copy-On-Write (COW) tables in handling big data analytics, particularly in scenarios involving frequent updates. While COW tables rewrite the entire file for any modification, resulting in inefficiencies for streaming data, MOR separates updates into changelog files, which are later merged, reducing write amplification. Uber utilizes COW tables for append-only data, such as user interactions in their app, ensuring fast query performance. On the other hand, Shopee, an eCommerce platform, employs MOR tables to handle update-heavy workloads like real-time user interactions, benefiting from efficient writes and reduced costs. These two table types, each serving distinct needs, offer companies flexibility in optimizing their data storage strategies depending on their specific requirements, with COW excelling in scenarios necessitating rapid reads and MOR providing advantages in environments with frequent data updates.