January 2026 Summaries

The blog post discusses the challenges and methodologies involved in optimizing the Lance file format for achieving high IOPS (Input/Output Operations Per Second), particularly aiming for 1,000,000 IOPS using NVMe drives. The author explores the intricacies of random access, commonly used in search workflows, and explains the importance of benchmark tests that consider both latency and throughput. The post highlights the complexities of achieving high IOPS due to synchronization overhead and system calls and details the process of reworking the scheduler and implementing io_uring to improve performance. Initial benchmarks on various hardware setups revealed bottlenecks, leading to iterative improvements in task management and I/O operations. Through a combination of a new scheduler and io_uring, the system achieved 1,500,000 IOPS, though further work is required to maintain high recall and address compute-storage separation in future architectures. The author plans to explore additional workflows, such as key-value stores, to optimize query performance further.

Table formats like Iceberg, Delta Lake, and Lance provide a structured way to manage data, primarily through their path management strategies, which significantly impact the portability and operational complexity of data at scale. While Iceberg originally used absolute paths for file references, it is transitioning to relative paths to enhance portability without the need for path rewrites during relocation. Conversely, Delta Lake began with relative paths to ensure zero-rewrite portability but later incorporated absolute paths to accommodate features like shallow cloning. Lance, however, prioritizes predictability and strict portability by using a fixed directory structure and exclusively relative paths, allowing datasets to be copied without metadata modifications. Lance introduces a multi-base path model, enabling a single dataset to span multiple storage locations while maintaining maximum portability, as demonstrated by Uber's AI infrastructure team, which required distributing datasets across multiple S3 buckets. This model supports various use cases, including multi-region data distribution, efficient disaster recovery, and AI experimentation workflows, by explicitly defining base paths in the manifest, thereby simplifying operational tasks like garbage collection and credential management.

The text explores the integration of LanceDB and CocoIndex to maintain data freshness and support multimodal AI applications. LanceDB serves as a scalable storage solution for managing large datasets, while CocoIndex facilitates incremental data processing, ensuring continuous updates from source to target data. The document further delves into the use of DSPy for declaratively specifying tasks for language model interactions, aiding in feature extraction. A case study involving a recipe application demonstrates how these tools work together to handle real-time data processing and feature enrichment using a combination of text and image embeddings. The approach prioritizes data freshness, crucial for reliable downstream tasks like search and analytics, by processing only new or altered data. Additionally, the text discusses the potential enhancements for search apps, including hybrid retrieval and agentic search, emphasizing the evolving nature of multimodal AI and the importance of robust data infrastructure.

Lance is an open-source lakehouse format designed to manage multimodal AI data by addressing the fragmented nature of current retrieval systems, where embeddings, text, metadata, and analysis layers are often separately stored and processed. It spans the file, table, and catalog layers of the lakehouse stack to ensure data stability and scalability. The introduction of the Lance × DuckDB extension allows users to seamlessly perform retrieval and analytics tasks using SQL within DuckDB, a portable SQL query engine. This extension makes it possible to run vector, full-text, and hybrid searches directly on Lance datasets and supports operations such as joining, aggregating, and materializing results without leaving the DuckDB environment. By integrating with cloud services and object storage, the extension provides an efficient solution for retrieval-augmented generation (RAG) systems and multimodal data management, allowing teams to iterate quickly and manage data artifacts effectively without the need for repeated data copying.

Lance SDK version 1.0.0 has been released, marking a significant milestone with the adoption of semantic versioning and a community-driven release process, ensuring clearer versioning and compatibility guarantees. The new release includes features such as GeoArrow-based GEO types, spatial SQL, and improved indexing techniques, alongside enhanced search capabilities demonstrated in Wikisearch, a search engine that efficiently navigates 41 million Wikipedia articles using LanceDB’s semantic and keyword search. The Lance community is engaging through events like the upcoming Community Sync and Data Day Texas talk, while contributors from organizations such as ByteDance, OpenAI, and Uber have played a pivotal role in advancing the project's capabilities. The release also involves organizational changes, such as the migration of catalog integrations to a new repository, and highlights contributions to the development of lance-graph and Lance REST namespace support within Apache Gravatino.

Modern AI-native data infrastructure has evolved beyond simple storage solutions to address the diverse structures and use cases of complex data, making multimodality a crucial aspect of current systems. This evolution requires systems that not only store various data types like images, videos, text, and audio but also support different access patterns, such as scans and random access, to handle analytics, search, and training tasks efficiently. Traditional architectures often lead to fragmented systems where structured data, vectors, and large binary objects are managed separately, resulting in governance issues and increased complexity. LanceDB, built on the open-source Lance format, addresses these challenges by providing a unified system that can store and manage all data types together, supporting both online and batch workloads with a single source of truth. This approach reduces the need for multiple performance copies across systems and simplifies data governance, offering versioned datasets that can evolve without excessive rewriting. The infrastructure allows for fast data evolution and efficient retrieval, which is essential as AI agents increasingly require the ability to act within environments and access underlying data in real time. This shift emphasizes the importance of storing, querying, and persisting data effectively to support a wide range of use cases and workloads, without the need for extensive expertise in distributed systems.