Branching and Shallow Cloning in Lance: Towards a "Git for AI Data"

Lance's multi-base layout provides a unified approach to branching, tagging, and shallow cloning, crucial for modern ML/AI workflows, enabling data scientists to experiment on production datasets safely and ML engineers to create reproducible snapshots for model training. This innovation builds on the limitations of previous systems like Apache Iceberg and Delta Lake, addressing issues of performance bottlenecks, governance isolation, and observability. By combining the benefits of both systems, Lance's design allows for efficient cross-location data referencing, strong governance, and clear cost attribution, while maintaining the intuitive Git-like experience developers are familiar with. The approach supports ML/AI teams in using tags for data snapshots, branches for isolated experimentation, and shallow clones for independent management, all within a portable format that facilitates cross-cloud compatibility. This lays the groundwork for a potential Git-like version control experience for datasets, enhancing the management and collaboration on data projects.