July 2024 Summaries

Llama 3.1, an open model rivaling top models, has sparked discussion on Twitter about differences in implementation decisions, optimizations, and quality testing processes among providers. A quick evaluation of Llama-3.1-405B showed significant variations in inference services, with some providers ranking high in GSM8K while others struggled with benchmark tests like AlpacaEval 2.0. The impact of these differences can be substantial, with a percentage point difference affecting the success or failure of an application task. To address this, Together AI has developed a five-step quality testing approach: reference matching, perplexity, analytic capability testing, generative capability testing, and qualitative testing. Their flagship implementation, Together Turbo, offers near-negligible differences in quality from the reference implementation with faster performance and lower cost, currently using FP8 quantization.

Together AI has partnered with Meta to release Llama 3.1 models for inference and fine-tuning, offering accelerated performance at full accuracy. The largest openly available foundation model, Llama 3.1 405B, rivals the best closed source models in AI and is now available on the Together Inference Platform. This platform delivers horizontal scalability with industry-leading performance, empowering developers to build Generative AI applications at production scale. With innovations like FlashAttention-3 kernels and custom-built speculators, the Together Inference Engine enables unmatched performance, accuracy, and cost-efficiency for Llama 3.1 models. Over 100,000 developers and companies are already building and running their Generative AI applications on the Together Platform, while new serverless endpoints and dedicated instances are available for deployment. The launch of Llama 3.1 models also includes significant advancements like expanded context length to 128K, support across 8 languages, and new security and safety tools for responsible development.

Together AI and NVIDIA have collaborated to power Llama 3.1 models for enterprises on NVIDIA DGX Cloud, bringing industry-leading Together Inference Engine to NVIDIA AI Foundry customers. This collaboration empowers enterprises to leverage openly available models like Llama 3.1 running on the Together Inference Engine on NVIDIA DGX Cloud, enabling highly optimized inference capabilities with unmatched performance, accuracy, and cost-efficiency. The partnership introduces the highly optimized Together Inference Engine to DGX Cloud, offering companies efficient and scalable AI inference capabilities, while allowing them to fine-tune models with their proprietary data for higher accuracy and performance. The collaboration marks an inflection point for open source AI with the launch of Llama 3.1 405B, the largest openly available foundation model, which offers unmatched flexibility, control, and state-of-the-art capabilities in general knowledge, steerability, math, tool use, and multilingual translation. This partnership enables enterprises to deploy endpoints with the highest performance, scalability, and security on NVIDIA DGX Cloud, accelerating the adoption of open-source AI among developers and enterprises.

The Together Inference Engine 2.0 introduces new Turbo and Lite endpoints, providing faster decoding throughput and higher quality models than commercial solutions. The new endpoints offer performance, quality, and price flexibility, allowing enterprises to scale their applications without compromising on any aspect. With the release of Together Turbo and Together Lite, developers can now build Generative AI applications at production scale with the fastest engine for Nvidia GPUs and the most accurate and cost-efficient solution. The engine achieves over 400 tokens per second on Meta Llama 3 8B by leveraging advanced techniques such as FlashAttention-3, faster GEMM & MHA kernels, innovations in quality-preserving quantization, and speculative decoding. The new endpoints are available starting today for Llama 3 models, with plans to roll out across other models soon.

The success of Llama-3 has been incredible, proving that open source models are rapidly catching up to closed models without compromising private ownership. Customers have been using their own proprietary data to fine-tune small OSS models like Llama-3 to run tasks with greater accuracy than even top closed-source models can achieve. A custom model created through this process outperforms leading OSS alternatives, surpasses a top closed-source alternative, and achieves over 90% of GPT-4o's accuracy while being much faster and cheaper, giving users full ownership of the model and weights. This approach results in smaller, faster, and more accurate models for specific tasks without compromising flexibility or ownership, all at a lower cost than using proprietary models like GPT-4.

FlashAttention-3 is a new version of the FlashAttention algorithm designed to speed up attention mechanisms in large language models by leveraging the capabilities of modern Hopper GPUs. It achieves 1.5-2.0x faster performance than its predecessor, FlashAttention-2, with FP16, reaching up to 740 TFLOPS and utilizing 75% of an H100 GPU's maximum capabilities. With FP8, it reaches close to 1.2 PFLOPS, while maintaining accuracy. The new algorithm uses powerful abstractions from NVIDIA's CUTLASS library and incorporates three main techniques: exploiting asynchrony with warp-specialization and interleave block-wise matmul and softmax operations, incoherent processing that leverages hardware support for FP8 low-precision, and overlapping GEMM and softmax to take advantage of the asynchronous nature of new instructions on Hopper GPUs. These optimizations enable more efficient GPU utilization, better performance with lower precision, and the ability to use longer context in large language models, ultimately unlocking new capabilities such as long context.