December 2024 Summaries

We've added three new observability features to Baseten: activity logging, LLM metrics, and customizable metric views. These enhancements aim to simplify monitoring and debugging of production AI workloads by providing timestamped records, request-level metrics for LLMs, and a unified, customizable dashboard view. This allows developers to quickly identify performance issues, optimize resource allocation, and maintain the reliability required for mission-critical AI applications.

Speculative decoding is an optimization technique designed to improve the latency of large language models (LLMs) by leveraging two models: a larger target model and a smaller draft model, both running on the same GPU. This approach reduces latency by generating potential output tokens with the smaller draft model, which can be accepted or rejected by the larger target model, thereby speeding up inference. The technique offers significant improvements in terms of time to first token (TTFT) and time per output token (TPOT), but comes with limitations, such as reduced throughput and quality when used with high batch sizes. To maximize benefits, it's essential to select an appropriate draft model, fine-tune it for specific use cases, and reduce orchestration overhead. Speculative decoding is particularly useful in scenarios where latency is a critical factor, such as code generation or meeting low-latency service level agreements (SLAs) for large models.

With the introduction of our Speculative Decoding Engine Builder integration, developers can now add speculative decoding to their production LLM deployments as part of a streamlined TensorRT-LLM Engine Builder flow, allowing for ultra-low-latency inference. This integration is particularly useful for latency-sensitive LLM applications, such as live translation, chatbots, and coding assistants, where best-in-class performance is required without compromising output quality. By using our pre-optimized config files or further tuning settings according to their needs, developers can leverage state-of-the-art model performance optimizations for their mission-critical production AI workloads. The integration has been shown to halve latencies with no effect on output quality and provides a two-tiered approach that balances ease of use with control over parameters, making it an ideal solution for applications using large models in production.

Speculative decoding is an inference optimization technique designed to improve the latency of LLM inference by coordinating two models on a single model server: a larger target model (e.g., Llama 70B) and a smaller draft model (e.g., Llama 8B). To support speculative decoding in production, the authors had to tackle issues such as inefficient batching, high time-to-first-token (TTFT), crashes, and unreliability. They implemented a mechanism that synchronizes the execution of the draft and target models, ensuring only one can run on the GPU at a time. This setup improves batching, TTFT, and stability by unlocking batching with scheduled and queued worker execution. The authors also fixed issues related to chunked prefill and KV cache re-use in TensorRT-LLM's request scheduling mechanism. With these improvements, speculative decoding is now production-ready, supporting streaming output, structured output, request termination support, and OpenAI spec compatibility. Benchmark results show that speculative decoding can reduce p50 latency by up to 90% for code generation tasks, with some tests showing improved TTFT and faster overall speed. The authors plan to continue improving the performance and stability of speculative decoding on TensorRT-LLM while contributing bugfixes back to the maintainers.

At Baseten, they've developed the fastest, most accurate, and cost-efficient Whisper transcription pipeline for production AI workloads, achieving over 1000x real-time factor and a word error rate of just 10.0 on the Rev16 benchmark. Their optimized pipeline uses a two-stage approach, chunking audio using voice activity detection to process longer files and remove unnecessary GPU processing. They've also implemented a custom hardware and scaling framework, Chains, to build multi-step inference pipelines that can be customized for optimal performance while keeping costs low. By optimizing Whisper transcription accuracy and speed, Baseten's pipeline is the most accurate and cost-efficient on the market, enabling users to reliably transcribe hours of audio in seconds.

Baseten introduces Custom Servers, a feature that allows developers to deploy production-ready model servers directly from any Docker image using just a YAML file. This new capability complements Truss Server, which is ideal for Python-based serving without writing server code. Custom Servers are best suited for pre-configured images like vLLM or proprietary Docker images. With full support for Baseten's suite of infrastructure optimizations, developers can easily convert any existing Dockerized model server into an elastic autoscaling service.