May 2025 Summaries

The blog discusses the challenges and solutions related to managing Apache Iceberg snapshots for data pipelines that require frequent writes for near-real-time ingest latencies. Apache Iceberg, a popular open table format, allows for schema evolution and time travel but faces difficulties in metadata management and scaling with frequent writes. Onehouse, described as the "Open Lakehouse Company," utilizes Apache Hudi and Apache XTable to manage metadata for large production tables, addressing issues such as slow snapshot expiration and orphan file deletion. The solutions include implementing a custom FileCleanupStrategy to skip scanning data files, using Hudi's cleaning table service for efficient file management, and leveraging Hudi's write markers to track and clean partial write operations. These strategies aim to maintain data freshness SLAs without resorting to costly daily or weekly maintenance operations. By asynchronously running table optimizations and maintenance services, Onehouse ensures continuous ingestion without impacting data freshness, supporting their vision of a Universal Data Lakehouse.

Onehouse has introduced Quanton, a new query execution engine designed to improve the price-performance of ETL workloads by 2-3 times compared to existing solutions. Integrated into the Onehouse Compute Runtime, Quanton allows users to run Apache Spark and SQL jobs more efficiently, offering significant cost reductions for Apache Hudi and SQL ETL workloads without requiring code changes. The platform promotes a Universal Data Lakehouse approach, supporting open table formats like Apache Hudi, Iceberg, and Delta Lake, and enabling interoperability with various data engines such as Databricks and Trino. Quanton is engineered to optimize compute and storage resources by minimizing the quantum of work in ETL processes, making it a cost-effective alternative to traditional data platforms. It also facilitates seamless integration and management of Spark and SQL pipelines, ensuring improved performance, compatibility, and cost transparency while avoiding vendor lock-in.

Cloud data platforms are essential for businesses, but evaluating their price-performance for ETL (Extract, Transform, Load) workloads remains a challenge due to standard benchmarking tools failing to capture the complexities of modern cloud data ecosystems. As companies increasingly rely on data lakehouses and open table formats, understanding real-world ETL costs becomes critical, especially given the rise of streaming data and mutable data patterns. Traditional benchmarks like TPC-DS and TPC-DI fall short in accurately modeling ETL workloads due to their outdated assumptions about data immutability and simplistic update/delete patterns. To address these gaps, Onehouse introduces Lake Loader, an open-source benchmarking tool designed to simulate realistic ETL workloads across DIM, FACT, and EVENT tables, offering a more accurate measurement of ETL performance and cost. This tool allows for consistent comparisons across platforms like AWS EMR, Databricks, and Snowflake, and aims to advance the standardization of ETL benchmarking by providing insights into incremental data changes, concurrency, and the impact of mutable operations.