June 2023 Summaries

LangChain has added a new integration with Baseten, allowing developers to power their apps with models hosted on Baseten. This integration enables the use of models such as Falcon and WizardLM in LangChain projects. The Baseten platform also features improved model logs, showcasing deployment progress and potential issues. Additionally, HuggingFace's Falcon-40B has taken the top spot on the open LLM leaderboard, performing well on benchmark tasks and being licensed for commercial use. Moreover, Baseten offers a model library that enables developers to deploy their own instance of open-source foundation models on autoscaling production infrastructure, allowing them to adapt these models to fit specific use cases through techniques such as prompt engineering and fine-tuning.

Large language models (LLMs) have powerful general capabilities but may not precisely fit specific use cases. To adapt LLMs for any use case, three techniques can be employed: prompt engineering, retrieval-augmented generation, and fine-tuning. Prompt engineering involves adding additional detail to prompts to improve model results, often using few-shot prompting strategies such as providing example inputs and outputs. This technique is useful for data formatting and is considered an inexpensive and effective way to get a wide range of behaviors from an off-the-shelf model. Retrieval-augmented generation adds vector databases to provide extra information in every call to the LLM, allowing for more efficient passing of relevant context. This approach is particularly useful for document search, help center chatbots, and domain-specific writing. Fine-tuning modifies the underlying model directly to add new corpus or modify behavior, often used in combination with embeddings and prompt engineering. The cost of each technique varies, with prompt engineering being low-cost and retrieval-augmented generation being medium-cost. Fine-tuning has a high up-front cost but returns customized results at a lower ongoing cost. By starting with the most basic approach and trying more customization if needed, users can effectively adapt LLMs to meet their specific use cases.

At a startup called Baseten, an internal hackathon was held where employees were given 8 hours to build their dream projects related to the company, resulting in several innovative features that accelerated the work of Baseten and widened its lead as the best-in-class infra provider for ML teams. The projects included infrastructure updates such as a byte range downloader to speed up model weights downloads, a redesigned infrastructure roadmap called "Beefeater" for multi-cluster deployment and data security, and faster inference services through WebSockets and a theoretical Federated Inference Engine. Additionally, the hackathon featured improvements to developer experience, including integrations with LangChain and GitHub, as well as Looker dashboards and updated documentation that made the platform more accessible to everyone. The winner of the hackathon was a feature called "I Stream 4 Ice Cream" which allowed builders to see all models stream their progress as they run inference, and the company plans to hold many more hackathons in the future.

Falcon-40B is an impressive large language model (LLM) released by Technology Innovation Institute (TII) in Dubai, which has made it to the top of the OpenLLM leaderboard. It can be used commercially but requires serious hardware to run, such as two A100 GPUs, making it challenging for developers to deploy and use it. However, a Truss model package is available that makes it easier to deploy Falcon-40B on Baseten, a model serving infrastructure, allowing users to quickly deploy the model and access its power, including logging, monitoring, and autoscaling. The model's performance is comparable to GPT-3.5, and it has been successfully deployed and tested with example prompts and responses that showcase its capabilities.

Baseten's model library is a platform that allows users to deploy popular open-source models in a few clicks, providing easy access to high-quality models for various applications. The library offers a range of features, including autoscaling, model logs, and model metrics, as well as predictable per-minute pricing and flexible scaling options. Users can also adapt the models by downloading the Truss package, which allows for interactive configuration and version control. With Baseten's model library, users can quickly deploy open-source models, such as Whisper and Stable Diffusion, and start building products on top of these models without requiring extensive expertise in deploying and serving machine learning models.

This post aims to provide a high-level understanding of foundation models, which are trained on broad and massive datasets, can be adapted for various downstream applications, and utilize unsupervised and semi-supervised learning methods. Foundation models have been used to develop popular apps such as Lensa and ChatGPT, and several open-source alternatives like ChatLLA MA. Training a foundation model requires significant amounts of data, which is often provided by organizations like OpenAI and Google. The process of adapting these models for downstream tasks involves fine-tuning or in-context learning methods, which can be computationally expensive but allow the models to perform tasks beyond their original training scope. Foundation models are available for various applications such as text generation, speech recognition, image generation, and more, and can be downloaded and customized using Truss, an open-source model serving framework.

The latest updates from Baseten include refreshed foundation models, including Stable Diffusion 2.1, Whisper, and Alpaca, which can be deployed directly to users' accounts for control over resource configuration and autoscaling. Exciting new open-source ML models such as WizardLM, a ChatGPT-like LLM, and Bark, a generative AI for audio, have been released, offering similar behavior and results at no cost. Additionally, information about NVIDIA's A10 GPU is provided, which is a workhorse for model serving, and the Baseten team recently held a hackathon, with several projects making their way into the platform as new features. The meetup schedule continues with an event in NYC on June 15, and the Baseten team looks forward to sharing updates from the hackathon soon.