November 2023 Summaries

Data engineers face challenges in efficiently implementing Change Data Capture (CDC) for real-time updates from transactional databases to analytics tables due to limitations of traditional data warehouses and data lakes. Data warehouses, while capable of fast updates, are often proprietary and costly, whereas data lakes, though cost-effective and open, struggle with real-time updates due to their batch-processing nature and immutable storage. A new solution, the universal data lakehouse, combines the advantages of both systems by using Hudi technology, as managed by Onehouse, offering a serverless approach that is both flexible and cost-effective. This solution allows for the integration of data lake affordability and openness with the real-time update capabilities of data warehouses, enabling efficient CDC processes. The Hudi project supports mutable data operations like inserts, updates, and deletions by cleverly using metadata to manage immutable storage, thereby providing high performance at reduced costs compared to traditional databases. Onehouse simplifies the creation and management of this data lakehouse through an easy-to-use interface, making it accessible without the need for extensive DevOps resources. This approach is gaining popularity, especially among enterprise-scale organizations, for its efficient handling of end-to-end CDC, offering significant savings in time and resources.

The Open Source Data Summit, a virtual event organized by Onehouse, brought together thousands of participants globally to explore the evolving role of open-source software in managing and innovating with data. The summit featured over 30 speakers who discussed the integration of open-source technologies like Apache Hudi, Apache Iceberg, and Delta Lake in creating robust data infrastructures. Highlights included the introduction of OneTable, a project enhancing interoperability between different data lakehouse formats, and discussions involving leaders from major tech companies such as Google, Microsoft, and Uber. The event underscored the shift towards open-source solutions to prevent vendor lock-in and foster a collaborative ecosystem. Attendees could access all sessions online, featuring insights into the latest advancements in data management, including Robinhood's and Walmart's implementations of data lakehouses and Uber's innovative data infrastructure. The summit emphasized community engagement, encouraging contributions to open-source projects like Hudi, and showcased Onehouse's commitment to advancing open-source data technologies.

OneTable is an open-source project created by Onehouse to provide seamless interoperability between the data lakehouse table formats Apache Hudi, Apache Iceberg, and Delta Lake. Unlike introducing a new table format, OneTable offers tools and abstractions that allow for omni-directional conversion of metadata between these formats without copying or rewriting data. This functionality enables users to query tables as native Delta, Hudi, or Iceberg tables, thanks to common in-memory models and interfaces for translating schema and metadata. Initially supported by major tech companies like Microsoft and Google, OneTable aims to facilitate diverse community collaboration and plans to submit for incubation into the Apache Software Foundation. The project's open-source nature encourages contributions and engagement, driving its future development and potential integration with other data technologies.

The response addresses criticism of Apache Hudi by refuting claims made in a blog post, emphasizing Hudi's commitment to maintaining robust ACID guarantees and highlighting the misinformation presented. The author clarifies Hudi's design decisions, such as instant time handling, concurrency control, and incremental read patterns, arguing that these choices are intentional to balance performance and safety. The text contrasts Hudi's approach with Iceberg, pointing out perceived shortcomings in Iceberg’s specifications and implementations, and underscores the importance of valuing diverse community contributions over strict adherence to specifications. The author advocates for open-source collaboration and innovation, underscoring Hudi's ongoing efforts to enhance interoperability through projects like OneTable and emphasizes the need for constructive, informed discussions within the open-source community.

Change Data Capture (CDC) is an essential process for moving data efficiently from transactional databases to analytical systems, ensuring that the freshest data is available for analytics, machine learning, and AI. As data volumes grow and the need for real-time analytics increases, CDC has become crucial for organizations to maintain up-to-date information without burdening production systems. The blog post discusses the evolution of CDC from its initial use for live backups to its current role in enabling seamless data integration across complex architectures. It introduces the concept of the data lakehouse, which, alongside CDC, helps streamline data infrastructure by supporting mutable data through technologies like Apache Hudi. This combination allows for quick data ingestion and processing, offering significant advantages for businesses that require real-time insights. CDC leverages various methods, such as log-based techniques, which are typically the most efficient, but implementing and maintaining CDC requires expertise in various tools and technologies. The blog highlights the importance of CDC in modern data operations and how it can drive competitive advantages by providing timely data for decision-making processes.