Optimizing ColPali for Retrieval at Scale, 13x Faster Results

ColPali, a tool for document retrieval from visually rich PDFs, faced challenges in scaling for large datasets due to the computational demands of generating and comparing numerous vectors per page. To address this, a hybrid optimization strategy was implemented, combining pooling to reduce the computational load and reranking to maintain accuracy. Specifically, the strategy involved compressing data using mean and max pooling to reduce vectors per page from 1,030 to 38, followed by a two-stage retrieval process where pooled embeddings quickly identified candidates, which were then refined using high-resolution embeddings. Experiments using a custom dataset demonstrated a 13x improvement in retrieval speed with minimal loss of precision, particularly with mean pooling, which maintained nearly identical quality to the original method. Future explorations may include column-wise pooling and other optimizations to enhance both speed and memory efficiency.