A new series of posts details the development of a file reader for the Lance v2 file format, focusing on improving parallel file reading by eliminating traditional row groups. The author explains how decoupling CPU batch size from I/O read size allows for more efficient data processing by creating mini-batches from data pages, which reduces RAM usage and maintains performance. The discussion touches on the limitations of infinite parallelism, emphasizing the importance of balancing I/O and CPU parallelism to avoid latency issues, especially when dealing with modern disks and cloud storage. The ideal read order prioritizes pages with lower row numbers to optimize performance, and internal benchmarks show that Lance v2 is significantly faster than its predecessor, Lance v1, when processing large datasets. This approach is not exclusive to Lance v2 and could be applied to other formats like Parquet, potentially enhancing their performance as well.