March 2024 Summaries

The text discusses the use of NVIDIA's Multi-Instance GPU (MIG) feature on H100 GPUs, which allows developers to split a single physical GPU into two or more virtual GPUs, each with its own memory and compute resources. This feature enables efficient model serving for machine learning models by providing equal or better performance compared to A100 GPUs at a 20% lower cost. The fractional H100 GPUs offer advantages such as support for FP8 precision, increased flexibility, and availability of GPUs across cloud providers and regions. The guide provides an overview of how MIG works, the specs of fractional H100 GPUs, and what performance to expect serving models on H100 MIG-based instances.

The Baseten platform has made significant improvements in model performance and developer experience, including optimized models for FP8 quantization, Multi-Instance GPUs, and TensorRT-LLM. The company has also introduced a new REST API endpoint for automating key model management tasks, allowing users to manage models and workspace properties with ease. Additionally, Baseten has released a benchmarking guide for Mistral 7B and is actively researching new techniques for faster inference, offering substantial cost savings on high-performance model deployments.

The text discusses the benefits of quantizing large language models (LLMs) like Mistral 7B using a data format called FP8, which offers faster inference performance while maintaining output quality comparable to the original FP16 model. The authors used a pre-release library compatible with the TensorRT-LLM ecosystem to quantize the model and found significant improvements in latency, throughput, and cost per million tokens. They validated the output quality using both quantitative (perplexity benchmark) and qualitative checks, ensuring that the quantized model's performance meets production requirements. Benchmarks showed that FP8 offers a 33% improvement in speed, an 8.5% decrease in latency, and a 24% reduction in cost per million tokens compared to the original FP16 model. The authors also explored how batch size and sequence length affect performance and provided guidance on selecting optimal configurations for specific use cases.

Baseten has achieved industry-leading performance for key latency and throughput metrics using Mistral 7B, with a time to first token of under 130 milliseconds, 170 tokens per second, and a total response time of 700 milliseconds. The company's dedicated model deployments offer substantial benefits in terms of privacy, security, and reliability, allowing developers to adjust various settings to optimize for latency, throughput, or cost. By experimenting with different batch sizes and sequence lengths, users can find the optimal configuration for their production workloads, taking into account factors such as infrastructure overhead, tokenization accuracy, and model output value. Baseten's optimized inference engines provide levers to make tradeoffs around these metrics, enabling users to achieve a lower cost at scale than shared endpoint providers.

TensorRT is a software development kit for high-performance deep learning inference, offering significant performance gains through optimization at the CUDA level on compiled models. To use TensorRT in production, one needs to know their compute needs and traffic patterns, as well as choose a supported model and GPU architecture. Optimizing model weights with TensorRT can result in 40% lower latency and 3x higher throughput for large language models like Mixtral 8x7B, and even more impressive gains on larger GPUs like the H100. By working closely with NVIDIA engineers and leveraging best practices, developers can achieve world-class performance on latency and throughput sensitive tasks.

FP8 is an 8-bit floating point data format that enables more efficient model inference with larger dynamic range compared to INT8, making it suitable for quantizing LLMs' activations and offering better performance improvements without significant degradation of output quality.

We founded Baseten in 2019 to accelerate the impact of machine learning, but it's still challenging for businesses to run large models in production due to difficulties with compute, scalability, cost optimization, and more. To address this, we've built Baseten to be a performant, scalable, and reliable way to run machine learning workloads, with features such as autoscaling, cold starts, secure workflows, and native support for large models. We've seen significant adoption from companies like Descript, Picnic Health, and Writer, and have recently raised an additional $40M to accelerate our growth and build new features, including multi-cloud support, integrations with best-in-class runtimes, and updates on fine-tuning, evals, and training. With this funding, we're poised to further revolutionize the machine learning landscape.

The benefits of globally distributed infrastructure for model serving include increased compute availability by providing access to a wider range of GPUs across multiple cloud providers, cost savings through competitive pricing, redundancy for better uptime by shifting workloads to regions with capacity, lower latency by locating servers near users, and data residency compliance by specifying constraints on region and cloud provider. A globally distributed infrastructure also abstracts away complexities, leaving customers with the benefits without the headache of operating workloads on multiple platforms.