January 2024 Summaries

Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model introduced in a paper that outperforms existing models such as Llama 2 70B and GPT-3.5 across various benchmarks, including mathematics, code generation, and multilingual tasks. Utilizing a routing network that selects two experts per token, the model accesses 47B parameters but actively uses only 13B during inference, leading to improved efficiency and faster processing speeds. The fine-tuned version, Mixtral 8x7B – Instruct, excels in instruction-following tasks with reduced biases and surpasses the performance of other leading models. Both versions of Mixtral are released under the Apache 2.0 license, facilitating open-source integration and broad accessibility, and contributions have been made to the vLLM project to support this integration.

OpenAI is set to retire 33 models, including the well-known GPT-3 (text-davinci-003), on January 4th, marking its largest model deprecation to date. Users are advised to transition manually to the newer gpt-3.5-turbo-instruct model. The /fine-tunes endpoint models like ada, babbage, curie, and davinci will also be retired, necessitating a shift to the /fine-tuning endpoint and models. While the majority of older embedding models will be shut down, the text-embedding-ada-002 model will remain active. Chat models, including GPT-4 and gpt-3.5-turbo, will not face immediate deprecation, but users are encouraged to transition to more cost-effective -1106 models. Azure will follow a similar deprecation schedule, with some model shutdowns announced on July 5th. The Assistants API beta v1 will be retired by December 18, 2024, and users should move to Beta v2. Other changes include the discontinuation of GPT-4.5-preview by July 14, 2025, and various text moderation models being phased out by October 27, 2025, with recommended shifts to newer models. For detailed guidance, OpenAI provides a comprehensive Deprecations guide and Model Map.