Introducing Loon: A New Storage Engine for Vector Data That Never Stops Changing

Loon is a new storage engine designed for Milvus and Zilliz Vector Lakebase to address the challenges of managing constantly evolving vector datasets, which traditional storage systems struggle with due to frequent updates, schema changes, and diverse access patterns. Loon's architecture incorporates hybrid file formats, row ID alignment, and a versioned Manifest to efficiently support both analytical scanning and point reads, enabling seamless coordination across multiple systems involved in AI data workflows. This design allows for the physical separation of different column types while maintaining logical coherence, reducing the need for frequent data rewriting. The Manifest serves as a central source of truth, ensuring that multiple systems can read and update the dataset consistently without duplicating data, thereby facilitating efficient online search, offline analysis, and external computation. Loon is integrated into Milvus 3.0 beta and Zilliz Vector Lakebase, providing a versioned, lake-native foundation that enhances the performance and scalability of vector databases.