The Case Against LLMs as Rerankers

The blog post evaluates the efficacy and practicality of using large language models (LLMs) versus specialized rerankers for reranking tasks in AI applications, highlighting the limitations of LLMs in terms of cost, latency, and performance when paired with strong first-stage retrieval methods. The study showcases rerank-2.5, a purpose-built reranker, as significantly more cost-effective, faster, and more accurate in improving retrieval quality compared to state-of-the-art LLMs like GPT-5 and others, especially when employed alongside robust initial retrieval techniques. While LLMs offer convenience and strong performance across diverse domains, their general-purpose nature and higher computational demands make them less suitable for reranking compared to specialized models. The study emphasizes that combining specialized rerankers with strong first-stage retrieval methods yields the best results, with rerank-2.5 outperforming LLMs by notable margins in both cost and accuracy metrics. Furthermore, the findings suggest that increasing the number of documents for reranking beyond a certain point offers diminishing returns, reinforcing the importance of an optimal initial retrieval setup.