February 2023 Summaries

Apache Hudi, an open-source data management framework, has enhanced its command-line interface tool, Hudi-CLI, to facilitate the management of data lakehouse tables, especially when dealing with thousands of tables. The latest update, version 0.13.0, introduces the hudi-cli-bundle, simplifying the process of launching the CLI tool and expanding its capabilities. Users can now efficiently inspect and monitor Hudi tables, perform table services such as clustering and compaction, and assist in disaster recovery operations. The tool offers various commands for monitoring commits, showing partition and file-level details, and managing file sizes, among other functionalities. The setup process for Hudi-CLI differs depending on whether Spark 2 or Spark 3 is used, with detailed instructions provided for both. The tool's enhanced features aim to improve the ease and efficiency of managing complex data environments, making it a valuable asset for organizations utilizing Apache Hudi.

Onehouse introduces Onetable, a feature that allows customers to expose their Apache Hudi tables as Apache Iceberg or Delta Lake tables, enabling native performance optimizations across various data lakehouse projects without the need to duplicate data. This approach addresses the growing need for decoupled data infrastructure, allowing different teams to leverage specialized compute frameworks like Apache Flink, Ray, and Dask. Onetable extracts table metadata into a common format, facilitating interoperability across different query engines and metadata formats, thus avoiding vendor lock-in and enhancing data portability. The initiative builds on Hudi's strengths in cost-efficient incremental data ingestion and transformation, offering a flexible data architecture that supports diverse query engines and optimizes performance. By syncing Hudi tables with other metadata formats, Onehouse aims to provide consistent snapshots and efficient data management, promoting a seamless and adaptable data lake experience. Initially an internal feature, Onetable is part of Onehouse’s commitment to open-source collaboration, and the company plans to open-source the project to benefit the wider data ecosystem.

Onehouse has emerged as a significant player in the data lakehouse landscape, offering an open, fully-managed, cloud-native service aimed at simplifying and enhancing the construction and operation of data lakes, while providing cost-effective and automated solutions akin to cloud data warehouses. Since its inception a year ago, Onehouse has engaged with over 100 organizations to address data lake and warehouse challenges, culminating in a $25M Series A funding round led by Addition and Greylock Partners. The company promotes a horizontal integration approach, enabling different engines to access a common, well-managed cloud data storage, thus addressing cost concerns and alleviating dependency on proprietary services. Apache Hudi, a key technology in their offering, facilitates real-time data processing and has gained traction for its capabilities in handling mutable transactional data. As Onehouse continues to innovate, the company envisions a future where data storage and management are decoupled from compute engines, reducing data lock-ins and driving the evolution of data lakehouse technology toward a more user-centric and efficient ecosystem.