July 2026 Summaries

Building expressive speech experiences that emulate human nuance requires extensive datasets and advanced models like Qwen3-TTS, which support zero-shot voice cloning and can be fine-tuned for richer capabilities. This fine-tuning involves curating a larger dataset, often with professional voice actors, and applying a modified training recipe to create single-speaker checkpoints for deployment. Qwen3-TTS offers two primary instant voice cloning modes: in-context learning (ICL) and speaker-embedding-only, each with distinct trade-offs in terms of speaker consistency and latency. Fine-tuning seeks to enhance text-audio alignment without ICL's overhead by training on extensive text-audio pairings, capturing a voice identity that is embedded directly into the model. This process involves using a centroid embedding derived from multiple clips, which stabilizes the representation and eliminates the need for reference audio at inference. Fine-tuning shows operational and perceptual benefits, such as improved expressiveness, even in data-constrained environments, and can be further enhanced by incorporating reinforcement learning signals. The Qwen3-TTS fine-tuning recipe available in the ml-cookbook allows users to self-deploy on Baseten Training, facilitating a seamless transition from training to inference.

Baseten Switch is a local Mac application designed to facilitate the seamless integration and comparison of various open and closed machine learning models within popular coding harnesses like Claude Code and Codex. Written in Go, this app allows users to dynamically route traffic between native closed models and open models from Baseten, providing a flexible way to optimize performance, cost, and reliability. By enabling easy toggling between models via a menubar icon, Baseten Switch eliminates the need for reconfiguring endpoints or restarting sessions, thus simplifying the process of testing different models within the same harness environment. Users can track detailed performance and cost metrics, such as tokens per second and total tokens generated, to make informed decisions about their model usage. Additionally, Baseten Switch ensures data privacy by routing requests through local routers to Baseten's inference endpoints, without the company accessing user prompts or metrics directly.

Over seven years, the Kimi K3 model has evolved significantly from its predecessor, GPT-2, through substantial architectural advancements, increasing its scale by a factor of 22,580. While GPT-2 utilized a decoder-only architecture requiring repetitive computations for each token, Kimi K3 introduces innovations like the KV cache and linear attention to optimize memory usage and computational efficiency. Moreover, the Kimi K3 employs a blend of Kimi Delta Attention and Multi-head Latent Attention (MLA) to enhance memory retention, alongside a Mixture-of-Experts layer to manage capacity more effectively. The key improvements in Kimi K3 include the implementation of Gated DeltaNet, which combines adaptive memory management with precise key-value association learning, and the use of selective retrieval mechanisms such as Attention Residuals (AttnRes) to mitigate hidden-state growth. These enhancements allow the model to allocate capacity purposefully, reducing memory traffic and improving overall performance without relying solely on scaling up parameter count.

Baseten for Model Labs is a comprehensive suite of products and services aimed at enabling closed-weight model labs to distribute and monetize their models effectively. It introduces the Frontier Gateway, a managed inference gateway that allows labs to serve models in production using a white-labeled API, and has been successfully used by various labs such as Poolside and WRITER. Recognizing a demand from developers for easy access to specialized models without additional onboarding, Baseten launched a platform that facilitates this access via the Baseten Model Library, offering a new distribution channel for model labs. The platform provides labs with production-ready infrastructure, increased visibility among AI developers, strong intellectual property protection, and go-to-market support, thereby allowing labs to focus on creating models while Baseten manages operational complexities. With the involvement of partners like NVIDIA and PyannoteAI, Baseten aims to foster a diverse AI ecosystem where both open and closed models thrive, optimized for specific domains and use cases.

Mani Parkhe has joined the Baseten engineering team, bringing with him extensive technical and leadership experience from his prior roles at Databricks, Uber, LinkedIn, Cadence Design Systems, and Mentor Graphics. Having been a founding engineer on MLflow and the Tech Lead for the Model Registry and Feature Store at Databricks, Mani is well-equipped to tackle complex technical and organizational challenges. At Baseten, he will lead cross-functional projects focusing on Dedicated Inference, Training, and Model APIs, leveraging his expertise to deliver solutions informed by his diverse background. In addition to his technical prowess, Mani has been a mentor to many engineers, which adds to the excitement of his new role as the company expands its engineering team to enhance its AI production platform.

Baseten announced day-0 support for Kimi K3 on their Model APIs, highlighting the collaborative efforts with Moonshot AI, Inferact, and RadixArk in developing this new open frontier model. With 2.8 trillion parameters, Kimi K3 is significantly larger than previous models, presenting various challenges in building a performant inference API. Utilizing new architectural techniques like Kimi Delta Attention and Attention Residuals, the model scales beyond the trillion-parameter threshold, leveraging extremely sparse experts and a novel vision encoder. The development process involved hardware provisioning, loading substantial model weights, and establishing baseline performance with leading open-source inference engines such as vLLM and SGLang. Rigorous benchmarking, facilitated by tools like the Kimi Vendor Verifier, ensured high-fidelity model serving, while extensive configuration and performance optimization strategies, including Tensor and Expert Parallelism, were employed to enhance latency and throughput. The large-scale deployment across multiple regions and cloud providers incorporates KV-aware routing to optimize system-wide throughput, crucial for handling the anticipated high demand for Kimi K3's capabilities in various applications ranging from coding to video editing.

Inference engineers have traditionally overlooked tokenization time due to its minimal impact compared to other processes like prefill and decode, but this has changed with the introduction of models like Kimi K3 that support input sequences of up to one million tokens. These extended sequences are prevalent in agentic workloads where repeated loops demand efficient tokenization. In response, the Baseten Tokenizer (Basetenkenizer) was developed to optimize these processes, particularly within the Baseten Inference Stack, offering significantly faster tokenization speeds compared to the previous Python tiktoken implementation. It provides up to an 18x increase in speed for million-token sequences while maintaining exact token ID parity, utilizing a Rust-based system with specialized optimizations such as pre-tokenization scanners, stack-resident BPE merge tiers, multi-core semantics, and zero-copy NumPy ownership transfer. These enhancements improve efficiency and reduce time-to-first-token (TTFT) for long input sequences, especially when cache hits are frequent, thus making Basetenkenizer the fastest measured path for K3 online serving from typed, pre-rendered segments while gigatokens excels in offline dataset tokenization.

Baseten recently released a highly optimized API for the GLM-5.2 model, achieving impressive speeds of up to 280 tokens per second and an average of 100 tokens per second, with performance more than doubling since its launch. This includes a fast API version designed for reduced latency, using Tensor and Expert Parallelism, which trades throughput for speed and consequently has higher token prices. The APIs, running on NVIDIA B200 GPUs, have been enhanced through optimizations in scheduling, NVFP4 weights, and speculative decoding profiles, leading to significant reductions in batch size to minimize resource competition. These developments have garnered positive market feedback, confirming the API's superiority in both benchmark tests and real-world application. Further improvements are planned, and the GLM-5.2 Fast API is available for public use on Baseten, with ongoing learnings from the Kimi K3 project expected to inform future enhancements.

Optimizing Large Language Model (LLM) inference is crucial to reducing costs and improving response times, often hindered by idle GPUs, repeated work, and excessive data movement. The process comprises two main phases: prefill, where the model processes the prompt, and decode, where it generates responses token by token. Key optimization techniques include continuous batching, which allows new requests to be processed without waiting for existing ones to complete, and speculative decoding, which uses smaller models to predict multiple tokens in advance, verified by the main model. Quantization reduces model size and speeds up computations by using lower-precision formats. Other strategies involve KV cache optimizations for efficient data handling, kernel and intelligent request routing to improve parallel computation, and disaggregation of prefill and decode phases to better allocate GPU resources. The choice of techniques depends on specific workload constraints, whether they are compute-bound during prefill or memory bandwidth-bound during decode, and often involves a combination of methods like quantization, speculative decoding, and KV cache reuse to achieve the best results.

In the evolving landscape of AI model selection, companies are re-evaluating their strategies to optimize for cost, reliability, and task-specific performance rather than defaulting to the most powerful models. Harnesses, which are essential for making large language models (LLMs) functional, must be tailored to each model's unique training structure to avoid degrading performance. The flexibility to switch models is becoming increasingly valuable as it allows businesses to adapt quickly to new offerings, manage costs, and maintain reliability. Open-weight models are catching up with closed models, offering more customization options and enabling access to markets with lower price sensitivity. Fine-tuning and reinforcement learning (RL) provide avenues for growth beyond just improving profit margins, allowing for more accessible AI-enabled work by lowering costs. As companies like Gamma and Notion explore further advancements such as image generation and data governance systems, the emphasis remains on using the right model for the right task, evaluating both the model and the provider to maintain control over pricing and quality.

GLM-5.2 Fast is a new Model API tier designed for real-time applications that require high per-user throughput, serving the same weights as the standard GLM-5.2 but on infrastructure optimized for agentic workloads. These workloads involve systems where a primary agent coordinates tasks among subagents, necessitating rapid and efficient inference calls. GLM-5.2 Fast enhances the workflow by being intelligent enough to manage tasks, cost-effective enough to scale, and fast enough to maintain competitiveness, while offering OpenAI-compatible API endpoints without the need for infrastructure management. Switching from the standard GLM-5.2 to the Fast version requires only a simple one-line change in the model slug, allowing users to route different workloads to the appropriate tier based on latency requirements. The service is open for access at launch, offering ease of use and tighter performance guarantees suitable for variable, bursty workloads.

The H100 and H200 GPUs are designed to optimize AI inference workflows, with each offering distinct advantages for different use cases. The H100 is cost-effective for small-to-mid-sized models with lower or sporadic traffic, benefiting from Multi-Instance GPU (MIG) technology that allows partitioning for parallel processing of smaller models. Conversely, the H200, equipped with larger HBM3e memory and higher memory bandwidth, excels in handling very large models and workloads requiring extensive memory and context windows. It is particularly advantageous for memory-intensive applications due to its ability to fit large model weights and KV cache on a single node, enhancing performance for long-context inference. Both GPUs leverage NVLink for improved data transfer rates between GPUs and employ asynchronous programming to maximize throughput by overlapping data loading and computation. The choice between H100 and H200 depends on specific needs regarding model size, traffic volume, and budget, with H100 being suitable for embeddings and speech models, while H200 is preferable for large language models demanding maximum memory.

GLM 5.2, a leading open-source language model, has been successfully enhanced with vision capabilities without compromising its original text-only functions. This was achieved by training a small 2-layer MLP with 50 million parameters, which allowed GLM 5.2 to match Claude 4.5 Haiku's performance on the MMMU-Pro benchmark. The project involved adapting a vision tower from Kimi K2.6 and training only the vision projector to align visual inputs with GLM's language model. This alignment, facilitated by a process called grokking during supervised fine-tuning (SFT), allowed the model to generalize well, even recognizing famous individuals not present in the training data. The reinforcement learning phase restored GLM's ability to reason about images, despite initial struggles, and demonstrated the model's capability to generate reasoning traces after minimal updates. The successful integration of vision into GLM 5.2 opens up new possibilities for using the model in multimodal applications.

Baseten's model performance team has achieved significant improvements in the video generation speed of the Wan 2.2 model, a leading open-source text-to-video generation tool, achieving a 53.6x speed increase by reducing the video generation time from over two minutes to just 2.75 seconds per clip. This enhancement is attributed to several key optimizations, including timestep distillation that reduces the diffusion process to four steps, custom kernel engineering for faster execution, and NVFP4 quantization that improves the throughput of tensor operations. These advancements not only decrease the generation cost per video significantly but also require an efficient and scalable infrastructure to handle increased traffic demands. The team has implemented autoscaling and queuing mechanisms to ensure stable service delivery and has established content guardrails to mitigate misuse, ensuring that video generation is conducted responsibly. A public demo of this high-performance video inference setup is available through July 31, 2026, showcasing Baseten's prowess in runtime optimizations and infrastructure development.

NVIDIA Nemotron 3 Embed offers two models, 8B and 1B, designed to enhance AI retrieval systems by converting text and code into embeddings that facilitate finding relevant information. The 8B model excels in retrieval accuracy, making it suitable for complex, accuracy-critical tasks, while the 1B model achieves 95% of the 8B's accuracy but with faster indexing, ideal for scenarios requiring frequent updates. Both models are available on Baseten, supporting AI agents, enterprise search, and code retrieval, and are integrated with turbopuffer to provide efficient semantic search capabilities. NVIDIA emphasizes the importance of balancing retrieval quality with indexing speed, and offers in-domain fine-tuning to improve accuracy for specific use cases, allowing teams to optimize the models for production workloads without needing to manage the underlying infrastructure.

Inkling, developed by Thinking Machines Lab, is a multimodal, autoregressive transformer model with 975 billion parameters, designed to process text, images, and audio inputs and generate text outputs. It features a mixture-of-experts architecture, activating only a portion of its parameters per task, which balances performance with efficiency. Inkling is supported from day one on the Baseten Platform, where it can be accessed through Model APIs and Dedicated Inference deployments. This model, built for breadth and optimized for developers creating AI-powered applications, offers open weights for customization and deployment. Despite its substantial infrastructure requirements, Baseten's autoscaling and multi-cloud capacity management enable reliable and scalable deployment. Inkling aims to extend human will and judgment, making it ideal for tasks ranging from coding assistance to chatbots and retrieval-augmented generation systems.

StepFun's Step 3.7 Flash is a cutting-edge, 198-billion-parameter sparse MoE vision-language model now available in the Baseten Model Library, optimized for lower cost and smoother autoscaling. This flagship multimodal reasoning model features an efficient architecture with only 11 billion parameters activated per token, native image and video input, a 256k token context window, and flexible reasoning capabilities. These features enable creative applications such as transforming whiteboard photos into actionable plans and converting receipts into spreadsheets. The model's FP8 quantization reduces memory requirements, allowing deployment on 4×H100s, making it cheaper and easier to deploy compared to larger configurations. Its Mixture-of-Experts architecture, hybrid attention, and speculative decoding ensure efficient handling of production workloads, offering a robust tool for teams to process diverse inputs effectively.

In the evolving landscape of AI applications, the integration of model-specific harness profiles is becoming crucial for enhancing capabilities beyond a model's original design. LangChain Deep Agents harness profiles enable developers to tailor AI models by encoding modifications to prompts and tools, improving model performance without altering the underlying structure. Recently, the LangChain team released a tuned Deep Agents harness profile for NVIDIA Nemotron 3 Ultra, achieving top agent accuracy and significantly reduced costs compared to closed alternatives. This advancement allows developers to seamlessly utilize NVIDIA Nemotron 3 Ultra with Baseten's model API, demonstrating the potential of these harness profiles in creating efficient, cost-effective AI solutions. An example application involves setting up an AI agent to review workflows for compliance, showcasing how these harness profiles facilitate high-performance, low-cost AI deployments with minimal code adjustments.

H100, H200, and B200 GPUs each provide distinct advantages based on memory, compute, and cost, catering to varying AI inference needs. The choice of GPU affects model latency, throughput, and cost, with the H100 being ideal for smaller models and sporadic traffic through its cost-effective Multi-Instance GPU (MIG) capability, the H200 accommodating very large models like DeepSeek-R1 due to its extensive memory capacity, and the B200 excelling in high-throughput production inference with its FP4 support and superior memory bandwidth. These GPUs utilize SXM connections for faster GPU interactions and NVLink for efficient weight and activation transfers, crucial for running large models across multiple GPUs. Additionally, innovations like the Blackwell architecture's FP4 and Tensor Memory Accelerator enhance memory efficiency and throughput, while asynchronous programming optimizes data movement, reducing idle times during inference. The optimal GPU choice hinges on specific AI workload requirements, such as model size, traffic volume, and budget considerations.