Embedding Tradeoffs, Quantified

Vespa, a platform frequently used for hybrid search combining lexical features like BM25 with semantic vectors, faces challenges in selecting the optimal embedding model that balances cost, quality, and latency. The MTEB leaderboard is often used for model selection but lacks practical deployment metrics such as inference speed on specific hardware and the impact of quantization. The blog details experiments conducted to address these gaps, focusing on models with fewer than 500 million parameters and widely used in production, evaluated on various hardware setups like Graviton3, Graviton4, and T4 GPU. Notably, the experiments revealed significant trade-offs, such as a 32x memory reduction and 4x faster inference with minimal quality loss using techniques like model quantization and vector precision adjustments. The results emphasized the benefits of using hybrid retrieval methods, which consistently outperform pure semantic searches, and underscored the importance of testing models on domain-specific data due to variations in performance across different contexts. The article concludes by encouraging users to leverage Vespa's interactive leaderboard to find the most suitable embedding model for their specific needs, considering factors like multilingual support and document length, and suggests potential improvements through fine-tuning and Vespa's flexible ranking system.