ColPali: Efficient Document Retrieval with Vision Language Models 👀

ColPali is an innovative document retrieval method that leverages Vision Language Models (VLMs) to efficiently retrieve documents by focusing on the image representation of document pages rather than traditional text-based methods. This approach, supported by the PaliGemma model and inspired by the ColBERT late interaction mechanism, enables fast indexing and querying by embedding page images through a series of patches processed by a vision transformer and language model. The method is evaluated using the Visual Document Retrieval Benchmark (ViDoRe), which assesses the ability to retrieve visually rich information from documents, showing superior performance compared to other systems, especially with complex visual tasks such as infographics and tables. ColPali's interpretability allows users to visualize which document patches correspond to specific query terms, enhancing the understanding of document content. The model's training involved using a diverse dataset of query-document image pairs, utilizing both Visual Question Answering datasets and a broad collection of PDF documents.