November 2023 Summaries

The Baseten team has introduced several new resources to help developers switch from closed-source inference endpoints to open-source ML models. A bridge tool allows users to experiment with open-source LLMs, such as Mistral 7B, by making just three code changes. The team also released a guide on model inference math and introduced Stable Video Diffusion, a research image-to-video model that creates short videos by adding movement to images. These new resources aim to make it easier for developers to customize their models, improve performance, and gain independence from closed-source endpoints.

The NVIDIA A10 and A10G GPUs are interchangeable for most model inference tasks due to their shared GPU memory and bandwidth, despite having different specs, particularly in tensor core compute. The A10 prioritizes tensor compute, while the A10G has a higher CUDA core performance. However, for most model inference tasks, including running seven billion parameter LLMs like Whisper and Stable Diffusion XL, the A10 and A10G have similar performance due to being memory bound rather than compute bound. This is confirmed through calculations of ops:byte ratio and arithmetic intensity, which show that the A10 and A10G have comparable performance for most popular models. The key factor in choosing a GPU for model inference is ensuring enough VRAM to run the model, with memory bandwidth having a higher impact on inference speed than tensor core compute.

Stable Video Diffusion is now available for researchers and developers to deploy in their own applications, marking a significant leap forward for open source novel video synthesis. The system builds on the image synthesis capabilities of Stable Diffusion to generate high-quality, coherent videos from single still images. It can produce 14 or 25 frame videos that continue the scene or action depicted in the seed image, and its performance surpasses previous video AI models in qualitative evaluations. The model also offers fine-tuning capabilities on downstream data, making it suitable for various applications such as stock footage animation, personal photo generation, presentations, film previsualization, and more.

If you’re using the ChatCompletions API and want to experiment with open source LLMs, a new endpoint has been created on Baseten that’s compatible with OpenAI's API, allowing for seamless migration to models like Mistral 7B. This bridge enables one-click model deployment, zero pip install commands, and only three tiny code changes are needed to update from GPT-3.5 to open source LLMs. The new endpoint supports streaming inference as well, and a curated library of open source models is available for exploration.

Open source ML models offer developers a level of control and customization over proprietary model APIs, allowing them to choose the best-suited architecture for their needs. By building on top of open source models, developers can access a wide range of capabilities that would otherwise be lacking from black box endpoint providers. Using open source models directly provides full control over inputs, outputs, and environment, protecting against "model shift" where endpoint providers change underlying models without notice. This independence also enables customization optimizations, such as reducing latency or optimizing GPU usage, to meet specific use case requirements. Additionally, using dedicated hardware for inference gives developers more control over their spend, predictable costs, and reduced attack surface, while also enabling better regulatory compliance. With open source models, developers can estimate expected request volume and provision hardware accordingly, automatically scaling up and down within configured limits in response to traffic.

Switching from a closed source ecosystem where you consume ML models from API endpoints to the world of open source ML models can seem intimidating, but it offers a vast number of models and flexibility in customization options. Choosing an appropriate GPU for model inference is crucial as some models require powerful GPUs while others can run on less expensive ones. Optimizing latency, throughput, quality, and cost is also possible with open source models, allowing users to align their use case requirements. Deploying and integrating the new model endpoint is a necessary step after selecting the right hardware configuration and optimizing the model for the desired outcome.

We want to use the full power of our GPU during LLM inference to maximize performance, but first we need to determine if our inference is compute bound or memory bound. Calculating the operations per byte possible on a given GPU and comparing it to the arithmetic intensity of our model's attention layers reveals where the bottleneck is: compute or memory. By understanding this, we can optimize our GPU usage and make the most of our compute capacity. Batching is a practical strategy for optimizing memory-bound inference, as it increases the model's arithmetic intensity by doing more computation for the same number of loads and stores from memory. Evaluating GPUs for LLM inference requires considering factors like latency sensitivity, batch size, and communication costs to choose the best GPU for our use case. Understanding the math behind profiling transformer inference is essential to controlling costs and improving performance during model serving, and real-world benchmarks can help account for factors that theoretical calculations may overlook.

Pinning ML model revisions for compatibility and security is a best practice when using open source models to prevent breaking changes or security vulnerabilities in your codebase. When working with open source packages, it's common to pin versions to prevent backwards-incompatible changes and new security vulnerabilities. This can be done by pinning package versions or model revisions, which gives you the chance to review any changes to your dependencies before updating. By doing so, you protect against unexpected breaking changes and malicious code execution. Maintaining a private copy of a model also provides similar protections and benefits, such as getting the same protections as pinning a model revision, ensuring your application isn't affected if the model is moved or deleted, and applying your own updates to the model.

Text embedding models transform text into vectors that represent its semantic meaning, enabling various use cases such as search, retrieval-augmented generation with LLMs, recommendations, classification, and clustering. These models encode chunks of text into vectors using tokenization, context windows, dimensionality, and similarity functions. Choosing the right model depends on the use case and compute resources, with popular open-source models like all-MiniLM-L6-v2, all-mpnet-base-v2, jina-embeddings-v2-base-en, LEALLA-base, and instructor-xl available for different applications. Packaging these models into Truss allows for easy deployment and inference, making it possible to create embeddings from a corpus of text and compare their similarity using various methods.