May 2024 Summaries

Apache Hudi, initially launched at Uber as an incremental data lake, has evolved into a key open-source data lakehouse project alongside Apache Iceberg and Delta Lake. It is utilized by Onehouse to implement the Universal Data Lakehouse architectural pattern, offering features like near real-time ingestion, incremental processing, ACID transactions, and time travel capabilities. The blog outlines how to integrate Hudi with Databricks, a cloud-based data engineering platform, to leverage Hudi's efficient data management within Databricks' robust environment. This integration allows users to benefit from Hudi's capabilities for real-time processing, simplified data ingestion, and enhanced data operations, all while utilizing Databricks' Photon engine for fast query performance. The setup process involves configuring Databricks to support Hudi tables, enabling users to conduct data management tasks effectively. The blog provides a step-by-step guide to configure Hudi within the Databricks environment, emphasizing its potential to streamline data operations and promote efficient data processing for complex data structures.

Onehouse has announced that its multi-catalog synchronization feature now integrates with Snowflake, Databricks Unity Catalog, Google Data Catalog, AWS Glue Data Catalog, DataHub, Hive Metastore, and Apache XTable, enhancing its Universal Data Lakehouse platform. This integration enables users to query a single copy of data from various cloud query engines, facilitating seamless data access without the constraints of vendor lock-in. The initiative underscores Onehouse's commitment to open data architectures by promoting format and catalog interoperability, allowing organizations to leverage diverse data tools and workflows while avoiding data duplication and maintenance complexities. By supporting open table formats like Apache Hudi, Iceberg, and Delta Lake, along with open file formats like Apache Parquet, Onehouse ensures compatibility with multiple query engines and formats. The introduction of the Onetable project, now Apache XTable, exemplifies efforts towards enabling cross-table interoperability, thereby freeing organizations from proprietary catalog constraints and enhancing their ability to integrate and utilize data across various platforms.

JobTarget has significantly enhanced its data management capabilities by migrating to an Apache Hudi-based data lake architecture, coupled with an AWS Glue-based framework called LakeBoost, amid rapid data growth. By adopting this architecture, they have achieved automated data ingestion, efficient deduplication, and substantial improvements in storage and compute resource utilization, leading to faster querying and reduced costs. Apache Hudi provides ACID transactions and supports both streaming and batch processing, making it ideal for managing the challenges posed by JobTarget's expanding data needs. Soumil Shah, Data Engineering Lead at JobTarget, highlighted these benefits in his presentation at the Open Source Data Summit 2023, explaining how the AWS Glue framework allows for programmatic data ingestion without the need for infrastructure code. The system efficiently processes transactional data through a series of templated AWS Glue jobs, which automate data cleaning, transformation, and storage in a scalable manner. This approach has streamlined data operations, reduced manual effort, and provided a unified interface for data consumers, enabling them to query large datasets quickly using their preferred tools.

The article provides an overview of data warehouses, data lakes, and data lakehouses, highlighting their evolution and significance in modern data management. Data warehouses, the oldest technology, are optimized for structured data analytics but can be costly and lack flexibility for handling semi-structured or unstructured data. In contrast, data lakes offer a cost-effective solution for storing large volumes of varied data types, though they often lack full SQL support and efficient update capabilities. The data lakehouse emerges as a hybrid, leveraging the strengths of both predecessors by supporting all data types with improved query and update efficiency, facilitating a unified approach to data analytics and management. The piece also introduces the Universal Data Lakehouse architecture, which aims to eliminate vendor lock-in by allowing interoperability across different lakehouse formats, thus providing a more flexible, scalable, and cost-effective data infrastructure solution.

Onehouse offers a flexible and efficient solution for data transformations within ETL and ELT processes by providing both no-code and custom transformer options for its users. These transformations are crucial for optimizing data as it moves through the data lakehouse stack to generate insights and value. Onehouse's no-code transformations handle common needs like struct flattening and array explosion, while custom transformers allow users to implement specific business logic using Java within the Spark framework. This enables seamless integration into Onehouse stream captures, ensuring that data pipelines are both cost-effective and high-performing. Users can build, test, and upload their custom transformation code using Gradle or Maven, making them accessible for execution within the Onehouse platform. By sharing compute resources between ingestion, transformation, and optimization tasks, Onehouse ensures that operations are efficient and no resources are wasted. This approach not only facilitates the creation of tailored data pipelines but also offers significant cost savings.