Multimodal Embedding & Reranker Models with Sentence Transformers

The blog post discusses the enhancements in the Sentence Transformers Python library with its v5.4 update, which introduces multimodal embedding and reranker models capable of processing and comparing texts, images, audio, and videos within a unified API. These multimodal models enable diverse applications such as visual document retrieval, cross-modal search, and retrieval-augmented generation (RAG) pipelines by mapping inputs from various modalities into a shared embedding space. The update provides expanded capabilities for encoding and ranking mixed-modality inputs, allowing users to compare texts against images or other media types. While multimodal reranker models offer superior quality by scoring mixed-modality pairs, they operate slower than embedding models, which are more suitable for initial retrieval tasks. The post also covers installation instructions, supported input types, and configurations for using these models, along with examples of embeddings and reranking processes, illustrating how these models can be applied in practice.