May 2026 Summaries

Diffusion models, while revolutionary in image generation, face challenges with computationally expensive iterative sampling processes, making real-time generation difficult. Timestep distillation emerges as a solution by compressing the sampling process from typically 20 steps to 4-8 steps, offering 2-3x speedups without compromising image quality. This approach cannot be combined with DiT cache due to differing assumptions about timestep outputs in distilled models. The study explores applying timestep distillation techniques to the FLUX.2 model using Distribution Matching Distillation (DMD), which trains a student model to match the teacher model's sample distribution rather than individual denoising steps. Key to this process is the two-timescale update rule (TTUR) and a GAN discriminator, which provide stability and sharpened image quality through adversarial supervision. Engineering optimizations utilizing NVIDIA's FastGen framework and techniques like mixed precision training, FSDP, and activation checkpointing allow the distillation process to handle the large memory demands of models like FLUX.2. As a result, FLUX.2 achieves a 2.5x speedup with minimal quality degradation, illustrating the potential of these techniques for efficient image generation.

The introductory post in a series examines the rise and implications of open-source AI models, highlighting their growing prevalence and impact on the AI industry. A pivotal moment occurred when the open-source model DeepSeek R1 was released, narrowing the intelligence gap with closed-source models and signaling a shift in AI's global power dynamics. Open-source models, such as those available on Hugging Face, allow for public access to model weights, enabling customization and specialization for specific use cases. This contrasts with closed-source models, which restrict access to their weights and training data. The discussion also addresses the cost advantages of open-source models, which are generally cheaper due to competition among inference providers and optimization research. While open-source models still lag behind closed models in some capabilities, they have proven effective for specific tasks through fine-tuning. The broader debate involves questions about the accessibility and control of AI technologies, the computational resources needed for training, and the geopolitical implications of open-source AI development, especially as Chinese labs gain prominence. This series will further explore how these models function, their optimal use cases, and their significance for software engineers.

Continual learning represents a transformative shift in machine learning, where models improve incrementally with each user interaction, contrasting with the traditional static approach that relies on periodic updates. This paradigm is being advanced by Trajectory, which focuses on integrating production traces to identify model failures and retrain models continuously. Core challenges include developing an infrastructure that supports dynamic models, allowing them to evolve with user input, and creating a product layer that unifies model learning with product functionality. Baseten and Trajectory have collaborated to develop an inference layer that rapidly deploys updated models, enabling swift iteration and improvement, with the ultimate goal of serving numerous model variants across different users. The process involves merging LoRA adapters into base models, validating these before deployment, and using A/B testing to assess performance. Despite significant progress, the journey towards achieving real-time model improvement and serving diverse models to individual users is ongoing, with future enhancements focusing on further optimizing the deployment and training processes.

Baseten has significantly optimized image generation serving for Flux.2 [dev] and Qwen-Image models, achieving up to 2.3x and 1.6x speedups on NVIDIA Blackwell GPUs and 1.9x and 1.1x on NVIDIA Hopper GPUs compared to SGLang. These improvements reduce single-request latency, crucial for latency-sensitive applications like creative tools and marketing, enhancing user experience, throughput, and cost efficiency. The optimizations leverage hardware-aware quantization, memory improvements, and specialized kernels, with Baseten FP4 on B200 GPUs delivering notable latency reductions to under one second for Flux.2 [dev] and significant speedups for Qwen-Image. The Baseten Inference Stack supports various image generation parameters, ensuring reliability and efficiency in production settings, and is adaptable to other models, facilitating low-latency, high-reliability deployments.

Voice technology is increasingly prominent in interacting with large language model (LLM) systems, and the Qwen3-TTS model family, optimized by vLLM-Omni, provides cost-effective and high-performance text-to-speech (TTS) solutions for various applications such as voice agents, language learning, and enterprise call infrastructure. These models achieve significant cost efficiencies, approximately $3-$4 per million characters, which is notably less expensive than many closed-source alternatives. The model's architecture allows for separate processing stages for generating and decoding acoustic tokens, enhancing concurrency and reducing costs. The system is designed to handle multiple requests simultaneously, improving throughput while maintaining high-quality voice output. Additionally, enhancements such as dynamic frame accumulation, speaker embedding caching, and word timestamps further optimize performance, making Qwen3-TTS suitable for real-time applications and voice cloning. Baseten's implementation offers these capabilities at a fraction of the cost compared to other providers, with ongoing updates to support the open-source community and facilitate custom voice fine-tuning.

Baseten Loops is a Python SDK designed to streamline the process of transitioning from reinforcement learning (RL) training to production inference by abstracting hardware complexity and focusing on training scripts. It addresses challenges faced by machine learning teams, such as gaps in open-source RL libraries, inefficiencies in deploying models to inference, and unpredictable performance on shared-infrastructure platforms. Loops offers a solution by providing a dedicated infrastructure that simplifies operations like sharding, memory management, and parallelism strategy while enabling predictable throughput and runtime. It supports advanced features like asynchronous RL and long-context capabilities, and integrates easily with existing workflows by allowing seamless migration through simple code changes. Currently in beta, Baseten Loops supports SFT and RL for various model families and aims to develop tools like Rollout Manager to further enhance the RL inference process and reduce the friction between training and production deployment.

Speculative decoding (SpecDec) has become a prominent technique to enhance the latency of large language models (LLMs) by having a smaller draft model propose tokens that a target model verifies, effectively improving speed by over twofold. EAGLE, a popular SpecDec method, utilizes the hidden states of the target model to predict draft tokens but is limited by autoregressive processing, often capping speedups at roughly 2x. To overcome this, DFlash was introduced to surpass the limitations of autoregressive drafting by using a single forward pass with bidirectional attention to predict multiple tokens, resulting in a significant speed boost. In practice, Baseten's implementation of DFlash shows a 3x speed improvement over the baseline on various benchmarks, outperforming both EAGLE and other DFlash implementations like vLLM and SGLang in throughput and latency. DFlash leverages the target model's hidden states, allowing for deeper draft models without sacrificing speed and improving the quality of speculative drafts. Testing across benchmarks like GSM8k, MATH-500, and NVIDIA’s Nemotron post-training dataset, Baseten's DFlash consistently outperforms other techniques in terms of inference speed and accuracy, demonstrating its effectiveness in bridging the quality of autoregressive decoding with the speed of diffusion LLMs.

Baseten has introduced the Frontier Gateway, a managed API gateway designed to streamline the launch of production-grade, multi-tenant inference APIs for model labs without the need to build or buy separate infrastructure. Built on Baseten Dedicated Inference, this gateway minimizes latency and integration overhead by co-locating with inference infrastructure, offering features such as authentication, federated API key management, rate limiting, billing, and white-label branding. This innovation comes amidst a transformative period for AI development, where accessibility to tools and talent has democratized the field, leading to a surge in new model labs. By focusing on research rather than infrastructure, these labs can leverage the Baseten Frontier Gateway to efficiently manage complex workflows and performance-optimized workloads, thus maintaining their competitive edge. Poolside, an early adopter, has reported impressive results, highlighting the speed and quality of Baseten's platform, which is now available for onboarding new labs.