December 2023 Summaries

Mamba-3B-SlimPJ has emerged as a strong contender to Transformers, with linear scaling in sequence length and fast inference, rivaling some of the best 3B Transformer architectures. The Mamba model was trained on 600B tokens on the SlimPajama dataset, under the Apache 2 license, and matches the quality of some of the best 3B Transformers such as BTLM-3B-8K with 17% fewer FLOPs. Mamba is a promising architecture for building foundation models, particularly in diverse applications like language, genomics, audio, and video. The model's training details include using the same hyperparameters as Mamba-3B on the Pile dataset but with a longer learning rate decay to accommodate more tokens. Evaluations show that Mamba-3B-SlimPJ matches the quality of very strong Transformers with 17% fewer training FLOPs and can be used for experimentation, understanding, chat, and instruction-tuned models.

Mixtral 8x7B, a high-quality sparse mixture of experts model (SMoE) with open weights, has been released by Mistral. This model is now live on the Together Platform, offering up to 100 tokens per second and a competitive pricing at $0.0006/1K tokens. Mixtral outperforms Llama 2 70B on most benchmarks, making it the strongest open-weight model with a permissive license, providing the best cost/performance trade-offs. The model can handle a context of 32k tokens and supports multiple languages including English, French, Italian, German, and Spanish. It also shows strong performance in code generation and can be finetuned into an instruction-following model that achieves a score of 8.3 on MT-Bench. Users can easily switch from OpenAI to Mixtral by adding their API key, changing the base URL, and using one of the open-source models. Additionally, the RedPajama-V2 Dataset is conceptualized as a foundation for creating high-quality datasets and should be filtered out depending on the application's intended use.

We're excited to announce our partnership with Meta to make Llama Guard available through the Together Platform, allowing users to leverage an LLM-based input-output safeguard model to moderate any open model hosted on the platform. Llama Guard is a competitively performing model that provides developers with a pretrained solution to defend against generating potentially risky outputs. It can be used as a standalone classifier or as a filter to safeguard responses from 100+ models, and is also available in our playground for testing and experimentation.

The StripedHyena models, including StripedHyena-Hessian-7B (SH 7B) and StripedHyena-Nous-7B (SH-N 7B), are an alternative to the popular Transformer architecture, offering improved efficiency in training, inference, and memory usage. The models build on research on designing efficient sequence modeling architectures, including H3, Hyena, HyenaDNA, and Monarch Mixer. StripedHyena achieves comparable performance with state-of-the-art Transformers on short and long-context evaluations, is faster and more memory-efficient for long-sequence training, fine-tuning, and generation, and has a reduced memory footprint during autoregressive generation. The models are designed using a hybrid of attention and gated convolutions arranged in Hyena operators and are optimized using model grafting techniques. StripedHyena is the first alternative architecture competitive with strong Transformer base models of the same size or larger, at scale, and can be used as a generalist baseline for long-context tasks. The models are designed to push the boundaries of model architectures beyond Transformers and inspire the open-source community to explore new exciting builds with diverse architectures.