March 2024 Summaries

At the Open Source Data Summit 2023, Girish Baliga, Director of Engineering at Uber, discussed the company's data infrastructure evolution, which enables Uber to process and analyze vast amounts of data efficiently across its global operations. Uber's data platform employs the Lambda architecture to manage both real-time and batch workloads, and it supports various analytics use cases including streaming, real-time, interactive, and batch analytics. Apache Hudi plays a crucial role in reducing data latency and improving data processing efficiency within this architecture. The platform uses Presto SQL as its primary query language, supported by Flink SQL and Spark SQL for more specialized needs, alongside programmatic APIs for complex scenarios. To enhance reliability and performance, Uber has implemented several technical customizations, such as smart query routing and multi-region deployments, and is transitioning towards a hybrid data environment with on-premises and cloud-based components. These innovations underscore Uber's commitment to leveraging open-source technologies to maintain a scalable and robust data infrastructure.

The evolution of data architecture has led to the development of the data lakehouse, a hybrid model that combines the strengths of data warehouses and data lakes. Historically, data warehouses were designed for structured data, offering robust query capabilities but at high costs and complexity, while data lakes provided inexpensive, flexible storage for a wide range of data types but lacked the structure and accessibility of data warehouses. The data lakehouse emerged as an innovative solution to unify these systems, enabling organizations to manage large volumes of diverse data with improved performance and cost-efficiency. Key projects in this space include Apache Hudi, Apache Iceberg, and Delta Lake, each offering varying levels of openness and performance. The concept of the universal data lakehouse (UDL) aims to create an ideal architecture that leverages the best aspects of both previous systems, providing a single source of truth for data that can be processed by various compute engines. Companies like Onehouse are promoting interoperability with tools like OneTable, which supports cross-platform data management, encouraging a flexible and open approach to data architecture.