Improving Search Ranking with Few-Shot Prompting of LLMs

Large Language Models (LLMs) are being utilized to generate synthetic labeled data for training ranking models, offering a cost-effective solution to the challenge of acquiring high-quality annotated data. By employing few-shot prompting with a handful of human-annotated examples, LLMs can create extensive amounts of synthetic queries, which are then used to train smaller, more efficient ranking models. This approach mitigates the biases inherent in click-model-derived labels and addresses the cold-start problem in new domains lacking interaction data. The process involves generating synthetic queries offline, using LLMs to avoid the computational expense of real-time inference, and employing a robust ranking model to ensure the quality of these queries. Experiments using the open-source flan-t5 model on the trec-covid dataset demonstrated significant improvements in retrieval effectiveness compared to existing zero-shot and unsupervised models. This method, which leverages prompt engineering to enhance query specificity, has been open-sourced and deployed in a Vespa application, highlighting its potential to revolutionize information retrieval by improving the quality of training data and retrieval outcomes.