Fine-Tuning Sparse Embeddings for E-Commerce Search | Part 3: Evaluation and Hard Negatives

In Part 3 of a series on fine-tuning sparse embeddings for e-commerce search, a SPLADE model, trained previously, is evaluated using Qdrant for indexing products, running retrieval benchmarks, and implementing hard negative mining, showing notable improvements over BM25 and off-the-shelf SPLADE models. The evaluation employs standard information retrieval metrics such as nDCG@10, with the fine-tuned model achieving a 28% improvement over BM25 and a 19% improvement over the off-the-shelf SPLADE, highlighting the importance of domain-specific training. Hybrid search with sparse and dense vectors is explored, revealing that while it provides moderate improvement with the off-the-shelf model, it can degrade performance when the sparse model is finely tuned. Hard negative mining through ANCE shows potential for additional performance gains by challenging the model with difficult negative examples, although it adds complexity. The study emphasizes the benefits of fine-tuning for domain adaptation, enhancing query expansion, term weighting, and domain-specific vocabulary, while managing latency effectively in a production setting.