July 2023 Summaries

Baseten offers a scalable, cost-efficient, and secure AI/ML infrastructure solution that can help startups get started quickly without requiring extensive expertise. Building one's own infrastructure can provide full control but also requires significant time and resources. Purchasing AI/ML infrastructure services can offer lower barriers to entry and competitive pricing, but may limit configuration control and require reliance on the provider for issues. Baseten provides a user-friendly model deployment process with logs and health metrics, horizontal scalability, blazing fast inference times, autoscaling features, and customizable computing resource management, making it an attractive option for those looking to streamline their infrastructure needs.

Llama 2 is a state-of-the-art open-source large language model that can be used for chatbots, and it comes pre-tuned for chat applications in three different sizes: 7B, 13B, and 70B. LangChain is a toolkit that allows developers to build with LLMs like Llama 2, using closed-source models or open-source models like Llama 2. The tutorial adapts the Create a ChatGPT Clone notebook from the LangChain docs, but instead of asking the model to behave as a Linux terminal, it asks Llama 2 to behave as an NPC for a video game. Llama 2 has a context window of 4,096 tokens, which is twice as much as Falcon and equal to the base version of GPT-3.5. LangChain lets developers take advantage of Llama 2's large context window to build a chatbot with just a few lines of code. The tutorial goes through building a chatbot that takes advantage of Llama 2's large context window, using Baseten to host Llama 2 for inference and the latest versions of the necessary Python packages.

Stable Diffusion XL 1.0 is a highly capable text-to-image model that can generate stunningly detailed and accurate images from simple prompts, requiring an A100 for invocation and taking around 8-12 seconds to produce an image. It can be deployed in two clicks from Baseten's model library or packaged as a Truss on GitHub. The model has the option of using the Stable Diffusion Refiner model to add accuracy to difficult-to-generate details like facial features and hands, which can be chosen with the use_refiner parameter.

We're excited about the latest foundation models from Meta and Stability AI, including Llama 2, FreeWilly1 and FreeWilly2, and SDXL 1.0. These models offer significant improvements in understanding meaning, generating relevant responses, and answering complex questions related to specialized domains. We also highlight NSQL 350M, a specialist foundation model that can generate accurate and precise SQL queries without further adaptation. Additionally, we discuss the growing repository of available foundation models, including LayoutLM Document QA, which is ideal for question-answering on documents with no GPU needed. With these models, developers can build more efficient and effective AI applications.

The Baseten model autoscaling features are designed to automatically adjust the number of replicas in response to incoming traffic, ensuring that only paid-for computing resources are used. This is achieved through setting a minimum and maximum number of replicas, with scale-to-zero functionality allowing models to be put to sleep after a period of inactivity. The autoscaling window, scale down delay, and concurrency target controls further fine-tune the scaling behavior, while cold starts enable rapid response to new traffic with minimal delays. These features are designed to work together to ensure efficient use of computing resources, allowing developers to focus on building their models without worrying about the underlying infrastructure.

The latest open source foundation models are being released at an incredible pace, with each model deployable in a couple of clicks from the Baseten Model Library. The Falcon-7B and Falcon-40B models, developed by the Technology Innovation Institute (TII), have been gaining popularity due to their high-quality data and robust performance, making them ideal for text generation and problem-solving applications. MusicGen is another model that's generating a lot of buzz, taking text or melodic inputs and producing high-quality music samples. WizardLM introduces Evol-Instruct, a new training dataset created by combining human-generated instructions with LLaMA. MPT-7B Base, a seven billion parameter model trained in under 10 days, offers an affordable alternative to large language models while still matching the quality of LLaMA-7B. These models can be deployed on Baseten in just a few clicks, making it easy for developers to integrate them into their projects.