Learn to Binarize CLIP for Multimodal Retrieval and Ranking

Binary embedding is a technique that transforms high-dimensional data into binary vectors, allowing for efficient storage and computation, particularly useful in large-scale multimedia retrieval. This text discusses the integration of binary embedding within the CLIP framework to enhance multimodal retrieval and ranking performance. It highlights the challenges and limitations of binary quantization when applied at test-time, noting a significant performance degradation due to reduced information granularity. To address this, the blog explores pseudo-quantization during training, utilizing continuous functions like tanh and sigmoid to approximate the binary quantization process and improve retrieval outcomes. The sigmoid activation consistently outperforms tanh across all metrics, maintaining a substantial percentage of the original float embeddings' performance. Moreover, the text notes that incorporating pseudo-quantization during training preserves float embeddings' performance well, averaging 99.7% of their original effectiveness. The use of Hamming distance slightly surpasses cosine similarity for binary embeddings, although it is limited to binary vectors. Sensitivity to the quantization scale is observed, necessitating careful experimentation to optimize performance across different data splits. Despite the improvements, a performance gap remains due to the inherent precision loss, underscoring the need for further model adjustments.