February 2024 Summaries

In the Hudi Live Event, Notion's software engineers Thomas Chow and Nathan Louie detailed their evolution of data infrastructure in response to a 10x growth in data scale over three years, shifting from a single PostgreSQL database to a sharded setup, and ultimately adopting Apache Hudi for a universal data lakehouse architecture. This transformation was driven by the need to manage the rapid doubling of data every six months to a year, which challenged their previous systems and increased demands on data processing and analytics, especially with the introduction of generative AI features. The new architecture, integrating Postgres, Debezium CDC, Kafka, and Apache Spark, achieved significant cost savings of over a million dollars a year and improved performance, with historical Fivetran syncing times reduced from a week to two hours. This infrastructure supports the Q&A AI feature, allowing efficient processing of large data volumes and enabling real-time updates through a vector database, crucial for AI capabilities within Notion's platform.

At the Open Source Data Summit, Walmart data engineers Ankur Ranjan and Ayush Bijawat presented on their strategic transition from a data lake to a data lakehouse architecture, emphasizing the pivotal role of Apache Hudi in this transformation. The shift was driven by the need to overcome data lake challenges, such as maintaining data integrity, and to leverage the combined benefits of data lake and warehouse architectures, including faster row-level operations and better transaction support. Apache Hudi was chosen for its superior capabilities in enabling both streaming and batch processing, alongside its strong support for open source software formats. This tool improves data management through features like record keys, precombined keys for upsert sorting, and efficient indexing, enabling better organization and reducing potential error vectors in data operations. By transitioning to a data lakehouse, Walmart experienced enhanced upsert and merge operations, improved schema enforcement, and the ability to efficiently handle duplicates, ultimately leading to reduced developer overhead and data bifurcation. The presentation effectively conveyed the advantages of adopting a data lakehouse model, illustrated with relatable examples, highlighting how Apache Hudi optimizes data workflows at Walmart.