January 2024 Summaries

S-LoRA is an optimization technique for running thousands of separate large language models (LLMs) simultaneously on a single GPU, addressing the cold-start problem that occurs when infrequently-used models are loaded. This approach leverages fine-tuning methods like LoRA to reduce the required memory and computational resources, enabling efficient serving of multiple task-specific fine-tuned models. By utilizing a tiered caching architecture and custom CUDA kernels, S-LoRA can load many adapters from the same base model onto one GPU, improving throughput and reducing response times, making it possible to deploy many small specialist models efficiently.

Axis, a platform providing multinational enterprises with market insights and regulatory data, developed a new product offering to build an index of global regulations. They faced challenges in scaling their service due to the high cost of GPT-4 and GPT-3.5 models. Fine-tuning their models using OpenPipe's SDK proved to be beneficial, allowing them to lower costs by over 95% per token compared to GPT-4-Turbo. Axis also utilized OpenPipe's automatic evaluation functionality and pruning rules dataset optimization feature, which helped improve model quality and reduce input token usage by over 50%. By deploying multiple fine-tuned models in production, Axis has built a defensible product that powers all customer traffic, outperforming GPT-4 in some cases.

The company has released a new fine-tune model, Mistral 7B Fine-Tune Optimized, which outperforms the base 7B model and even surpasses GPT-4 in some cases. Automated evaluations have also been announced, allowing users to quickly assess their fine-tuned models' performance relative to GPT-4. The company has made several other product improvements, including openAPI API support for uploading request logs, increased dataset size limits, pruning rule enhancements, and improved access controls. These updates aim to increase ecosystem compatibility, improve model performance, and enhance user experience.