June 2024 Summaries

RAG fine-tuning has shown significant improvements in code generation accuracy, offering 3.7x faster speed and a cost reduction of up to 150x compared to existing models like Claude 3 Opus and GPT-4o. By leveraging the Together API and Morph Labs' advanced technologies in codebase search and synthetic data generation, this approach enables personalized code assistants with repository-level context and fine-tuning an open-source LLM, making these models more practical and valuable tools for developers. The technique addresses the limitations of outdated knowledge and hallucinations in LLMs, achieving up to 19% quality improvement, 1.1x faster speed at 37.5x cost reduction compared to GPT-4o, while offering 16% better accuracy than Claude 3 Opus.

We introduce SpecExec, a new speculative decoding method for interactive LLM inference on consumer devices, which achieves speeds of 4-6 tokens per second with 4-bit quantization or 2-3 tokens per second with 16-bit weights. This approach is based on the classical concept of "speculative execution" applied to LLM inference and leverages the spikiness in token probability distributions in modern large language models. By using a powerful draft model to deterministically construct a large draft tree containing the most likely continuations of the input text, SpecExec directly applies speculative execution to LLM inference, achieving significant speedups over autoregressive decoding with offloading, with relative speedups ranging from 4.6x to 18.7x on various consumer GPUs. The method is particularly suited for the offloading regime and targets large language models that cannot fit on consumer GPUs due to limited memory availability. SpecExec outperforms other speculative decoding methods like SpecInfer, achieving faster speeds with larger budgets, and shows promise in making LLMs more accessible and usable by a broader audience.

Together MoA introduces a novel approach to harness the collective strengths of multiple LLMs, leveraging their diverse capabilities and insights to improve state-of-the-art quality. By adopting a layered architecture where each layer comprises several LLM agents, MoA effectively integrates diverse models, resulting in a more robust and versatile combined model. The reference implementation, Together MoA, significantly surpasses prior leader GPT-4o on AlpacaEval 2.0, achieving a score of 65.1% with only open-source models. This approach is based on the collaborativeness of LLMs, where an LLM tends to generate better responses when presented with outputs from other models. MoA categorizes its roles into proposers and aggregators, proposing initial reference responses and synthesizing them into high-quality responses through multiple layers. The study demonstrates that integrating a wider variety of inputs from different models significantly enhances the output, highlighting the value of leveraging diverse perspectives and capabilities that different models offer. Together MoA method significantly outperforms strong closed-source models in terms of accuracy and quality, making it an exciting approach for enhancing AI systems.

Dragonfly is an instruction-tuning Vision-language architecture that enhances fine-grained visual understanding and reasoning about image regions by employing multi-resolution zoom-and-select strategies. This approach allows for a detailed and efficient visual understanding of complex image data in specific domains, such as biomedical imaging. The model achieves competitive performance on vision-language benchmarks like commonsense visual QA and image captioning, outperforming prior models including Med-Gemini on multiple medical imaging tasks. Dragonfly's effectiveness is attributed to its ability to focus on fine-grained details of image regions, enabling better commonsense reasoning and fine-grained understanding of high-resolution image data.

Axiomic is a new Python library for building AI agents, aiming to make agent building easy yet production-ready. It integrates with Together API, leveraging industry-leading performance for multi-agent workflows. Axiomic's demo, GEAR Chat, showcases four agents working together in a chat application, improving steerability, portability, and reliability by specializing each agent on a task. The Elect agent uses multishot prompting to implement a multiclass text classification task, making evaluation across models easy with Axiomic. The demo demonstrates the potential of leveraging sophisticated agents for multi-step AI applications, with the growing ecosystem of tools making building these applications easier. Axiomic's focus is on providing a flexible and production-ready solution for building AI agents.