December 2024 Summaries

Modern coding assistants and agents utilize code retrieval systems, which often rely on embedding models to perform vector-based searches for relevant code snippets, docstrings, and documentation within extensive repositories. Despite their widespread use, challenges remain in evaluating the quality of these systems due to a lack of diverse benchmarking datasets and methodologies. Voyage AI addresses these issues by exploring typical subtasks of code retrieval, such as text-to-code, code-to-code, and docstring-to-code, and examining existing datasets like CodeSearchNet and CoSQA, which suffer from limitations such as noisy labels and lack of deep algorithmic reasoning. To improve code retrieval evaluation, Voyage AI proposes creating new datasets by repurposing question-answer datasets and leveraging code repositories like GitHub. They have developed a combination of public and proprietary datasets, avoiding contamination and providing robust evaluation metrics across various code embedding models. Future efforts include sharing in-house datasets for collaborative research and exploring the use of large language models as judges for retrieval performance.

Voyage-code-3 is a next-generation embedding model designed for code retrieval, outperforming its predecessors OpenAI-v3-large and CodeSage-large by significant margins across 32 code retrieval datasets. It offers reduced storage and search costs through support for smaller dimensions and quantized formats like int8 and binary, enabled by Matryoshka learning and quantization-aware training. The model maintains high retrieval quality despite lower precision, with flexible embeddings ranging from 256 to 2048 dimensions and a context length of 32K tokens. Addressing the unique challenges of code retrieval, voyage-code-3 is trained on a diverse, high-quality code corpus and evaluated on datasets tailored to real-world applications, demonstrating superior performance in various retrieval tasks such as text-to-code and code-to-code. Users can further enhance retrieval quality with binary rescoring, and the model is accessible with an initial free allocation of tokens for exploration and experimentation.