September 2024 Summaries

Today, Together AI launched Llama 3.2 vision and lightweight models, as well as the release of Llama Stack. The company is partnering with Meta to integrate these models, and will be one of the first API providers for Llama Stack. Key features include a free Llama 3.2 11B Vision Model, optimized vision models (11B, 90B), lightweight model (3B), and new Llama Stack APIs on Together AI. The launch aims to power rapid development of multimodal agentic apps.

The Together Enterprise Platform is a comprehensive solution for managing the entire Generative AI lifecycle, enabling businesses to train, fine-tune, and run inference on any model, in any environment. It optimizes model performance and GPU utilization, offering up to 50% lower operational costs and faster inference speeds. The platform supports various deployment options, including private deployments within a user's VPC or on-premise infrastructure. Key features include continuous model optimization, access to over 200 models, enhanced GPU orchestration, and new Enterprise plans for scalability. Companies like Salesforce, The Washington Post, Zoom, and Zomato have already deployed their GenAI apps using the Together Enterprise Platform.

Together AI has appointed Kai Mak as its Chief Revenue Officer (CRO) to accelerate the adoption of Generative AI among businesses globally, marking a significant milestone in the company's growth trajectory. The appointment comes at a pivotal moment as the demand for robust and flexible AI infrastructure increases, with over 150,000 developers and companies using the Together Platform to power their Generative AI applications. As CRO, Kai brings extensive experience in building and scaling go-to-market teams, having led the revenue growth of Webflow by 10x and serving as interim CMO for a year. He is well-positioned to drive the company's mission of serving businesses across the spectrum, from startups to established enterprises, with a unique blend of expertise in product-led growth and enterprise sales. With Kai on board, Together AI aims to establish itself as the AI Acceleration Cloud for businesses globally, delivering unparalleled performance, accuracy, and cost while keeping models and data secure under user control. The company's comprehensive platform empowers developers and businesses to seamlessly design, develop, and manage their entire Generative AI lifecycle, from training and fine-tuning to inference, with a focus on open research, open science, and trust among researchers, developers, and enterprises.

The authors propose distilling large-scale Transformer models into hybrid linear RNNs like Mamba, preserving impressive generative capabilities while significantly enhancing efficiency. This approach combines the strengths of both Transformers and linear RNNs to create models that are powerful yet highly efficient. The authors demonstrate the effectiveness of this method through experiments on various benchmarks, including the OpenLLM Leaderboard, showing that the distilled hybrid models outperform open-source models in terms of performance and efficiency. Speculative decoding is also proposed as a means to accelerate inference speed for these models.

The NVIDIA H200 Tensor Core GPU is a high-performance computing (HPC) and artificial intelligence (AI) workhorse, designed to excel in both AI and HPC workloads. With its advanced Hopper architecture, the H200 provides 40% faster inference performance on Llama 2 13B and 90% faster performance on Llama 2 70B, demonstrating significant improvement in handling large-scale language models. The GPU's substantial memory and bandwidth allow it to handle even the most data-intensive applications with ease, minimizing bottlenecks and enabling real-time processing of vast datasets. Together AI's custom-built Together Kernel Collection (TKC) offers up to 24% speedup for operators used frequently in training and up to 75% speedup for fundamental operations used in FP8 inference, significantly accelerating common AI operations.

Speculative decoding for high-throughput long-context inference has been reevaluated, revealing that it can significantly improve throughput and latency. The analysis shows that as sequence lengths increase, bottlenecks shift from being compute-bound to memory-bound, making speculative decoding more effective. Two algorithmic innovations, MagicDec and adaptive Sequoia trees, have been proposed to take advantage of this shift. MagicDec uses a fixed context window in the draft model to speed up drafting, while adaptive Sequoia trees adaptively choose the tree size that maximizes speedup. These innovations can achieve significant speedups, up to 2x for LLaMA-2-7B-32K and 1.84x for LLaMA-3.1-8B on 8 A100 GPUs, making them an essential part of throughput optimization systems for long-context workloads.