July 2023 Summaries

Together AI has released LLaMA-2-7B-32K, a 32K context model built using Position Interpolation and Together AI's data recipe and system optimizations. This model extends the original LLaMA-2 to 32K long context, achieving comparable perplexity and quality to state-of-the-art closed-source models. The power of this base model lies in its ability to be fine-tuned for targeted applications, such as multi-document question answering and summarization. To support this, Together AI has updated their inference and training stack with FlashAttention-2 and other optimizations, allowing for efficient inference and fine-tuning with 32K context. The community is encouraged to build on this work by exploring ways to extend the context length of open-source models, preparing better data for long-context tasks, and improving system support for long-context training and inference.

The researchers at Together AI have developed a new model architecture called Monarch Mixer, which aims to increase efficiency while maintaining quality in Transformers. The Monarch Mixer (M2) is a sub-quadratic approach that replaces the traditional Transformer architecture with a more efficient one, enabling it to scale more efficiently and train faster. The first target for M2 is BERT, the most popular model used for language tasks, and M2-BERT has been shown to be 25% more parameter-efficient than BERT while matching its quality. The researchers have also explored the potential of long-sequence models with Monarch Mixer, which could enable scaling to longer sequences without significant loss in performance. The code and checkpoints for M2-BERT are now available on GitHub, and further releases and updates will be made in the coming weeks.

Tri Dao, a Chief Scientist at Together AI, has released FlashAttention-2, an algorithm designed to speed up training and inference of large language models by up to 4x and achieving 72% model FLOPs utilization on NVIDIA A100 GPUs. The new version is built from scratch using primitives from NVIDIA's CUTLASS 3.x and its core library CuTe, providing clean abstractions and powerful building blocks for maximum speed. FlashAttention-2 achieves a 2x speedup over the previous implementation, reaching up to 230 TFLOPs/s on A100 GPUs, and is available in open source on Github. The algorithm is designed to work with existing models and can be used for training, fine-tuning, and inference of large language models. With its improvements, FlashAttention-2 enables models with twice as long context lengths while maintaining an interactive experience, making it a significant breakthrough in the field of natural language processing.

Together AI has announced a strategic partnership with Snorkel AI to enable enterprises to build custom large language models (LLMs) on their data in secure environments. This collaboration aims to provide businesses with a full-stack solution for AI development, allowing them to overcome the limitations of publicly available LLMs and create proprietary models that are specialized to their domain and use case. By partnering with Snorkel AI's data development platform and Together AI's training, fine-tuning, and inference cloud, organizations can build high-quality LLMs that are tailored to their specific needs, providing a significant advantage over existing solutions.

Together AI has launched two products, Together API and Together Compute, to help developers build with open-source AI models in a cost-effective and efficient manner. These cloud services offer a full stack solution for training, fine-tuning, and running leading open-source AI models, including over 50 models currently hosted by Together AI. The goal is to make AI accessible by reducing costs, which can be significant due to the large number of parameters in generative AI models. Together API provides an easy-to-use fine-tuning API with optimized private endpoints for low-latency inference, while Together Compute offers clusters of high-end GPUs paired with a distributed training stack at a cost-effective price point. This move is seen as part of the "Linux moment" for AI, where open-source solutions are becoming increasingly popular, and Together AI aims to be part of this movement by providing a platform for developers to build and release models, datasets, and research in the open.