September 2021 Summaries

Ziyue Guan from Bytedance discusses the construction of an Exabyte-level data lake using Apache Hudi for Bytedance’s recommendation system, highlighting the system's requirements, design decisions, functionality support, performance tuning, and future work. The system leverages BigTable for near real-time processing and uses data lakes in scenarios like feature engineering and model training, facing challenges such as irregular data, large throughput, and complex schema. Hudi was chosen over other data lake engines due to its ecosystem openness and support for global indexes, with Spark being used for its mature support despite Hudi's limited Flink compatibility at the time. Functionalities like MVCC and schema registration systems were developed, alongside performance enhancements like optimized serialization and compaction processes. Future plans include improving productization, supporting ecosystem diversity, enhancing cost and performance, and expanding storage semantics, with ongoing recruitment efforts for their recommendation architecture team in various global locations.