September 2026 Summaries

Baseten has partnered with OpenAI to become an early open-model inference provider in the OpenAI B2B Marketplace, enabling enterprise customers to use existing OpenAI commitments for Baseten-served open models through Codex and the Responses API. The partnership supports multi-model agentic coding workflows, allowing organizations to route tasks between open and proprietary models based on quality, cost, latency, and capacity. Baseten highlights its infrastructure for these workloads, including performance optimization for long-context code generation, active-active deployments across more than 90 clusters and 20-plus clouds, immediate availability of major new open models, agent sandboxes through Blaxel, and enterprise controls such as zero data retention, regional deployment options, and granular access management. The company also offers engineering and applied research support to help customers tune deployments and customize models, positioning the collaboration as a way to increase the value and flexibility of AI coding systems.

Baseten has joined NVIDIA’s Agent Safety Platform initiative and the Open Secure AI Alliance, emphasizing open safety controls across application, runtime, and infrastructure layers for production AI agents. Following its acquisition of Blaxel, Baseten is introducing Carbon in private preview, a fourth-generation agent execution platform using microVM sandboxes, dedicated IPv6 addresses, persistent storage, and tightly controlled networking to support large-scale autonomous workloads. Carbon adds kernel-level capabilities, runtime policy enforcement, manual snapshots, sandbox forking, and rapid deployment from snapshots, enabling integrations such as NVIDIA OpenShell, which can quarantine potentially unsafe agents. The platform is intended to help organizations connect agent monitoring with practical controls including permissions, isolation, and recovery to known-safe states.

Baseten announced the appointment of Sheila Vashee as Chief Marketing Officer, bringing more than 20 years of marketing and growth leadership experience from Figma, Opendoor, and Dropbox. Vashee describes the expanding open AI ecosystem as an opportunity for companies to select, optimize, and own intelligence suited to their products, while Baseten provides infrastructure that helps deploy models reliably at scale. The company highlights customers including OpenEvidence, Harvey, and Cursor as examples of organizations using AI to support clinicians, legal professionals, and software developers. Vashee says enterprises need control over model selection, inference, post-training, and execution, and positions Baseten as a platform that reduces the complexity of managing this stack. Baseten aims to become an intelligence infrastructure provider for AI-focused companies and is hiring to expand its production AI inference platform.

LangChain has introduced LangSmith Fine-Tuning, a workflow that lets users convert successful LangSmith agent traces into supervised fine-tuning datasets for open models using the open-source smithtune CLI. Training is executed through Baseten Loops on dedicated GPUs within a user’s Baseten workspace, using the Loops Python SDK and the user’s own API key and billing account. The CLI checks which models are available for training before jobs are scheduled, saves checkpoints directly on Baseten, and can deploy evaluated checkpoints to Baseten Dedicated Inference without requiring model-weight transfers. The integration is intended to give teams with production agent traces a relatively direct path to custom models optimized for their own agents, including through coding-agent-assisted setup. Smithtune is available on GitHub, while Baseten Loops remains in early access and may require workspace enablement.

Large language model post-training adapts broadly pretrained base models to specific tasks through supervised fine-tuning, preference alignment, reinforcement learning, and distillation. Supervised fine-tuning teaches desired formats, instructions, tone, and task behavior from expert prompt-answer examples, while parameter-efficient approaches such as LoRA reduce cost by training small adapter matrices instead of all model weights. RLHF uses human-ranked outputs to train a reward model and then optimize the LLM, commonly with PPO, for subjective qualities such as helpfulness and safety, whereas DPO achieves preference alignment more simply by directly learning from chosen and rejected response pairs without a reward model or online reinforcement learning. RLVR uses automatically verifiable outcomes, such as correct answers or passing tests, and GRPO compares multiple generated responses without requiring a separate critic, making it well suited to coding, mathematics, tool use, and multi-step agents. Distillation trains a smaller student model to imitate outputs from a stronger teacher, with on-policy variants using privileged information or evaluation-selected self-generated answers when no superior teacher is available. In practice, these methods can form a pipeline, often beginning with SFT, followed by preference optimization and verifiable-reward reinforcement learning, while prompt engineering may be sufficient for simpler, narrowly defined failures.

NVIDIA Nemotron 3 Diarization is an open, transformer-based end-to-end speaker diarization model available through Baseten’s Model Library for prerecorded audio, live streaming, and real-time speaker-attributed transcription. Rather than using separate segmentation, speaker-embedding, and clustering stages, it processes 16 kHz audio into per-speaker activity probabilities for up to eight speakers, maintaining speaker identities across chunks through lightweight state management. A single checkpoint supports configurable algorithmic latency profiles from 320 milliseconds to 30.4 seconds, allowing developers to balance responsiveness and accuracy without redeploying models. Reported benchmarks show relatively small accuracy declines at ultra-low latency and improved diarization error rates over prior Streaming Sortformer versions on datasets including AISHELL-4, while remaining competitive across multilingual benchmarks. Per-speaker activity signals can also support voice activity detection, speaker-specific endpointing, interruption detection, and turn-taking in voice agents. Baseten offers dedicated batch, WebSocket streaming, and diarized transcription presets, with the transcription option integrating NVIDIA Parakeet V2 ASR to handle overlapping speakers in a unified pipeline. On an NVIDIA RTX PRO 6000, Baseten reports capacity exceeding 500 concurrent low-latency diarization streams, or roughly 190 streams with transcription enabled, alongside higher batch throughput than its tested NVIDIA reference configuration.

Baseten has introduced Hosted Tools, an ecosystem designed to add capabilities such as web search, code execution, and browser automation to models and agents running on its platform without requiring developers to build and orchestrate each integration themselves. Its first offering, Baseten Grounded Inference, enables models to decide when to use web search and iteratively retrieve current, specialized information through providers including Exa, Keenable, Parallel, and You.com, with configuration support for Messages, Chat Completions, and Responses APIs. The service runs tool calls server-side within Baseten’s inference stack, streams live tool activity to clients, supports configurable research depth, and is intended to reduce network round trips, context reprocessing, and agent latency; Baseten reports a 15% end-to-end latency decrease compared with client-side tools in its benchmarks. Hosted Tools also provides a common interface, unified billing and usage reporting, mixed client and hosted tool support, and provider choice for production-scale AI applications. Baseten plans to expand the platform with tools such as code execution and browser use, supported by its acquisition of Blaxel for isolated persistent sandboxes and storage, while Grounded Inference is currently available in a playground preview with limited requests and free testing credits.

AI gateways centralize decisions that accompany every model request, including authentication, routing, usage limits, reliability handling, metering, and auditing, preventing these responsibilities from being fragmented across application code and infrastructure. The text distinguishes between access gateways, which help an application use multiple external model providers through a unified interface, and serving gateways, which enable model owners to expose their own models as secure, multi-tenant commercial APIs for customers. Serving gateways validate customer identity, select healthy and appropriate model deployments, enforce tenant isolation and rate or token limits, attribute consumption for billing, and connect customer-facing requests to underlying inference events for support and auditing. Organizations evaluating gateways should consider alignment with their customer and entitlement model, isolation during traffic spikes or degradation, inference-specific awareness, deployment location, portability, and the remaining systems needed for subscriptions, billing, and customer support. For model providers, placing serving-gateway functions close to inference can apply policy before costly execution and improve operational visibility. Baseten presents its Frontier Gateway as a managed serving-side option for Baseten-hosted models, offering branded endpoints, API-key management, usage limits and attribution, and integration with its dedicated inference infrastructure, while emphasizing that it is not intended as a universal proxy for third-party model APIs.

AI inference is the stage in which a trained model applies learned patterns to new inputs, such as generating chatbot responses, translating text, filtering spam, or processing voice commands, whereas training involves the computationally intensive process of learning from large datasets over time. In production, an inference request moves from an API or interface through authentication, routing, queueing, and model servers, where specialized runtimes tokenize inputs, process them on CPU and GPU hardware, generate outputs, and return or stream results to users. Building these systems is difficult because developers must balance low latency, high reliability, and efficient costs, which can conflict as demand scales. Performance is commonly assessed through latency measures such as time to first token and completion time, throughput measures including tokens or requests per second, and cost per token, with techniques such as batching, optimized kernels, model parallelism, intelligent routing, autoscaling, and multi-cluster infrastructure used to improve results.

LangChain is collaborating with Baseten to train and deploy custom models that power LangSmith Engine, an in-platform agent designed to help users diagnose and improve AI agents autonomously. By fine-tuning open-weight models on agent traces, LangChain can specialize models for tasks such as examining GitHub repositories to identify issue causes and draft related code or prompt changes, while smaller models can be tailored for classification tasks like identifying failure modes or linking traces to open issues. Baseten Loops provides API-based managed infrastructure for supervised fine-tuning, reinforcement learning, long-context workloads, checkpoint evaluation, and direct deployment through Baseten’s inference platform. The partnership aims to shorten the path from model experimentation to production by combining LangChain’s agent evaluation expertise with Baseten’s training and inference infrastructure.

DeepSeek has released the open-weight DeepSeek-V4.1-Flash, a 552-billion-parameter multimodal mixture-of-experts model designed for coding agents that supports text and image inputs, text output, and a one-million-token context window, and it is available through Hugging Face and Baseten’s Model API. Although it has 552B total parameters, it activates 8B parameters during input prefill and 16B during output decoding through a Causal Encoder-Decoder architecture, which splits computation between encoder and decoder stages to reduce prefill costs and improve efficiency for agentic workflows with repeated context. DeepSeek reports that the model improves on earlier V4-Flash and V4-Pro models in coding, long-horizon agent tasks, and visual reasoning, while using fewer active parameters, though benchmark results indicate that human oversight remains advisable for complex automation and consequential image-based reasoning. Its KV cache is reported to require about one-quarter the memory of V4-Flash’s cache through architectural changes, compressed sparse attention, and lower-precision FP4 caching, potentially improving cache reuse, throughput, and time to first token. DeepSeek is replacing V4-Flash platform traffic with V4.1-Flash and plans to reroute V4-Pro traffic as well, while Baseten supports deployment using KV-cache-aware routing intended to send requests to replicas that already hold relevant context.

Blaxel is joining Baseten to combine Blaxel’s stateful agent-execution infrastructure with Baseten’s model training and inference platform, with the stated goal of supporting large-scale autonomous AI agents on a single integrated cloud. Blaxel contributes isolated microVM sandboxes, persistent storage through Agent Drive, and networking designed for agents to run code, use tools, retain context, and communicate securely, while Baseten provides scalable inference infrastructure across multiple regions. The companies argue that colocating inference, compute, storage, and networking can reduce latency, improve reliability, and lower costs for long-running agent workloads, including post-training workflows informed by agent activity. Blaxel says its existing product and customer support will continue unchanged, while Baseten plans to incorporate Blaxel’s primitives, beginning with sandboxes, into expanded agent-focused offerings.

Pyannote’s Community-1 and Precision-2 speaker diarization models, which identify who spoke when in audio, have been optimized to improve production performance while retaining strong accuracy for applications such as meeting transcription, call analytics, and media archives. The optimizations include improved GPU scheduling and batching, reduced CPU-GPU memory transfers, fused inference passes, GPU-specific tuning, index-based clustering that reduces the cost of processing long recordings, and mixed, quality-aware numerical precision across pipeline stages. Baseten reports up to a 9.6x reduction in long-audio latency for Community-1, enabling 20 hours of audio to be diarized in about two minutes on one GPU, and about 3.2x higher Precision-2 throughput with a 0.5 percentage-point absolute diarization error tradeoff. Reported VoxConverse diarization error rates are 11.1% for Community-1 and 8.8% for Precision-2, and planned additions include voiceprint enrollment and cross-session speaker identification using embeddings already generated during diarization.

Baseten reports that its deployment of Qwen3 ASR 1.7B Streaming leads Coval’s September 2026 early-access voice AI benchmark on the speech-to-text quality-latency Pareto frontier, achieving the lowest word error rate while operating about five times faster than OpenAI’s tested option. Coval provides reproducible, independent evaluations that account for datasets, model versions, normalization methods, and latency measurement, assessing STT through both transcription accuracy and speed and TTS through time to first audio. The report argues that voice-agent performance depends on the entire inference stack rather than model selection alone, as delays from STT, language models, TTS, networking, orchestration, voice activity detection, autoscaling, and provider-to-provider transfers compound during conversations. It emphasizes evaluating complete conversational turns and tail latency alongside median results, since inconsistent performance under load can degrade user experience. Baseten attributes reliable low-latency operation partly to optimized deployments, dedicated hardware, and infrastructure placement, while noting that benchmark results from a fixed region provide a baseline and real-world performance also depends on deployment architecture, location, workload type, quality requirements, and cost targets.

Baseten describes a managed rollout system for reinforcement learning that synchronizes compressed delta weight updates from customer-controlled trainers to independent global GPU clusters without requiring shared networks, schedulers, memory, or direct trainer-to-GPU connections. Trainers publish full checkpoints and XOR-based byte-level weight deltas to object storage, while the Baseten Delivery Network distributes the data so clusters download weights once and reconstruct policies locally; in a GLM-5.3 fixture, 716.56 GiB of logical XOR data reportedly compressed to 1.55 GiB with zstd. To avoid decompressing and scanning the entire logical model output, the company developed a sparse zstd decoder that interprets compression commands directly, tracks zero runs, and emits only changed bytes and necessary repeated nonzero values. These changes are converted into retry-safe absolute tensor values, translated into vLLM’s GPU memory layout, and batched to reduce many small GPU operations into fewer transfers. Requests are paused only during the final GPU update, which Baseten says reduced the pause on an 8×B300 replica from roughly 12 seconds to 6 seconds, while the full publication-to-new-policy process averages about 36 seconds and checkpoint promotion continues asynchronously after service resumes.

Baseten has launched a backend MCP server, documentation MCP server, and open-source skill package to help coding agents such as Claude Code, Codex, Cursor, Gemini CLI, VS Code, and Windsurf manage its inference platform more efficiently. The backend server exposes Baseten REST API capabilities for deployments, autoscaling, logs, secrets, and training jobs, while the documentation server enables unauthenticated searches of Baseten resources and the skill guides agents toward appropriate tools and workflows. These tools allow agents to deploy and debug models, monitor deployment health, tune autoscaling, answer operational questions, and clean up resources, with each action labeled as read-only or mutating to support approval safeguards. Baseten reports that its benchmarks across 320 runs found average reductions of 7.5% in time, token use, and cost, with operation-heavy tasks improving by as much as 57% and acceptance rates rising from 89% to 97%. Although custom scripts may remain more efficient for long, fixed workflows, Baseten positions the MCP bundle for exploratory and iterative tasks such as debugging and tuning.

Selecting an open-source model for post-training depends primarily on use case, tool compatibility, benchmark performance, and costs driven by active parameters, total parameters, and KV-cache size. Reinforcement learning is particularly sensitive to generation costs because it requires repeated model rollouts that are evaluated and used as reward signals to adjust behavior, while supervised fine-tuning and preference optimization offer other specialization approaches. DeepSeek-V4-Flash is positioned as an economical option for long-context text tasks due to its 13B active parameters and compressed KV cache, while GLM-5.2 targets more expensive workloads with asynchronous RL tooling. Kimi K2.6 offers multimodal and agentic capabilities, and K2.7 Code builds on it with stronger coding performance and lower reasoning-token usage for coding agents. NVIDIA’s Nemotron-3-Super-120B emphasizes efficient long-context reasoning, a hybrid architecture with limited KV-cache growth, native FP4 support, and compatibility with established NVIDIA tooling. The Qwen3 family is presented as a broadly supported default across sizes and price ranges, suitable for applications from lightweight classification and retrieval to larger training workloads.

Inference engineering applies the concept of an efficient frontier to balance model-serving outcomes such as latency, throughput, cost, and quality, distinguishing between techniques that shift deployments along existing tradeoffs and those that improve overall performance. Batch sizing and GPU parallelism strategies, including tensor, expert, and attention data parallelism, allow operators to prioritize either faster individual responses or greater total throughput, while quantization generally improves serving efficiency but may introduce quality tradeoffs. Frontier-expanding approaches include kernel and runtime optimization, which reduce resources required per token; speculative decoding, which accelerates generation by predicting and validating likely tokens; and prefill/decode disaggregation, which assigns inference phases to specialized workers and can improve throughput while maintaining or modestly improving latency. The effectiveness of these methods depends on workload characteristics, and their benefits can compound when hardware and software improvements are combined.