How Search Quality Shapes RL Outcomes

The study explores the impact of different search backends on reinforcement learning (RL) outcomes by comparing an agent trained with Exa against one trained with a SERP-based backend. The research finds that agents trained with Exa outperform those trained with SERP in terms of pass@k performance across various benchmarks, achieving higher accuracy with less computational cost in both training and inference phases. The Exa-trained agents demonstrated better sample efficiency, retrieving more relevant information with fewer actions, which enhanced learning and reduced the sparsity of rewards. Furthermore, the Exa-trained agents maintained superior performance even when the search backend was switched at inference, suggesting that the skills learned with Exa are transferable. This indicates that the choice of search engine significantly affects the efficiency and effectiveness of RL training for language models, emphasizing the importance of using a robust search backend like Exa for optimal results.