March 2023 Summaries

OpenChatKit now offers a 7B parameter model quantized to 8-bit, allowing it to run on many consumer-grade GPUs, providing faster performance and enabling a wider audience to use the model. The new model is fine-tuned on user feedback and exhibits similar quality to the existing 20B parameter model. User feedback plays a crucial role in improving the quality of the models, with the OIG training dataset being fine-tuned using feedback from the LAION community chat contributions repository. The updates also include improved performance for the 20B parameter model, multi-GPU support, and training scripts for all models. The value of user feedback is highlighted through examples of how it has improved the quality of writing emails.

FlexGen is a high-throughput generative inference engine for large language models (LLMs) designed to run on limited resources, such as a single commodity GPU. It can be configured under various hardware constraints by aggregating memory and computation from the GPU, CPU, and disk, and solves a linear programming problem to find efficient patterns for tensor storage and access. FlexGen compresses weights and attention caches to 4 bits with negligible accuracy loss, enabling larger batch sizes and higher throughput. By achieving this, FlexGen outperforms state-of-the-art offloading systems when running on a single 16GB GPU, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. The code is available online, allowing users to explore its potential for various applications.

OpenChatKit provides a powerful, open-source base for creating both specialized and general-purpose chatbots for various applications. It's built on top of EleutherAI’s GPT-NeoX model, fine-tuned with 43 million high-quality instructions, and includes tools to fine-tune the model for specific tasks. The kit has four key components: an instruction-tuned large language model, customization recipes, an extensible retrieval system, and a moderation model. OpenChatKit is released under an Apache-2.0 license, allowing users to deeply tune, modify or inspect the weights for their own applications or research. The project includes tools for ongoing improvement with community contributions, such as process for dataset contributions, incorporating feedback, and Hugging Face app for user reports. It's built on top of the Together Decentralized Cloud, which enables fast and efficient training, and is 100% carbon negative.