March 2024 Summaries

Together AI has raised $106M in a new round of financing led by Salesforce Ventures, with participation from several prominent investors, including Coatue, Lux Capital, and Kleiner Perkins. The company aims to expand its serverless APIs for inference and fine-tuning, which have over 45,000 registered developers, and is widely integrated into various AI application development frameworks. The new funding will support the expansion of its roadmap, including features to support large enterprises, and plans to expand compute capacity internationally. Salesforce Ventures' Managing Partner Paul Drews believes that Together AI has emerged as a leading solution for open-source demand in AI adoption, and the company is confident in its ability to bring its ambitious vision to fruition. The funding also highlights the growing importance of open-source models in generative AI, with models like Llama-2, Mixtral, Qwen, Gemma, and StripedHyena rapidly narrowing the benchmark gap between best closed and open LLMs. Together AI offers a platform for developers to easily develop with and deploy open-source models, and its capabilities grow with the best of what open-source AI has to offer. The company is also expanding its cloud network and working with over 10 GPU cloud platforms today, offering a seamless AI-centric cloud experience built over a multi-cloud substrate.

We introduce Sequoia, a scalable, robust, and hardware-aware speculative decoding framework that improves large language model inference speed on consumer GPUs with offloading, as well as high-end GPUs without any approximations. By creating large trees of speculated tokens, Sequoia achieves significant speedups, reaching an average time between tokens (TBT) of 0.57s on a single RTX-4090, outperforming highly optimized offloading systems and DeepSpeed-Zero-Inference. On high-end GPUs, Sequoia improves decoding speeds by up to 4.04x for larger models, making it suitable for various model sizes and hardware configurations. The framework leverages dynamic programming algorithms, sampling without replacement, and a hardware-aware optimizer to select optimal tree sizes and depths for each hardware configuration, providing robustness and scalability. With Sequoia, users can host large language models like 70B on consumer GPUs without approximations, boosting AI-generated content applications, and achieving significant speedups on high-end GPUs in the small-batch setting.

BASED: Simple linear attention language models balance the recall-throughput tradeoff` Based, a simple efficient architecture combines sliding window attention and linear attention to offer high-quality language modeling with strong associative recall capabilities. At inference time, Based decodes without a KV-cache, enabling a 24x throughput improvement over Transformers with Flash-Attention 2! The Based architecture outperforms prior sub-quadratic models on real-world recall-intensive tasks and in-context learning, while offering fast generation speeds. By using just two well-known, familiar attention-like building blocks, sliding window attention (with tiny window sizes) and linear attention (with Taylor series approximation of exp(QK^T)), we can outperform the strongest sub-quadratic architectures on language modeling and achieve massive speedups over optimized Transformers! The choice of featurization matters as well, with our Taylor map being surprisingly simple yet effective. IO-aware algorithms for the Taylor linear attention forward pass and inference reduce data movement between slow HBM and SRAM, unlocking efficiency. Based achieves up to 24x higher throughput than FlashAttention-2 in next token prediction, making it a promising architecture for language modeling and other applications.