November 2026 Summaries

10 posts from Baseten

Filter
Month: Year:
Post Summaries Back to Blog
No summary generated yet.
Nov 16, 2026 1,870 words in the original blog post.
Baseten reports optimizing DeepSeek V4.1 Flash 1017 through more than 80 kernel and inference-stack improvements to create a faster variant aimed at low-cost, high-throughput agentic workloads. The company argues that mid-sized “flash” models offer a favorable balance of capability, speed, and price, citing benchmark and Artificial Analysis results that place the model on a cost-performance frontier despite its mixture-of-experts architecture. Its principal hardware-level techniques include kernel fusion, which retains intermediate data in on-chip memory and reduces launch overhead; Programmatic Dependent Launch, which overlaps independent portions of successive GPU kernels; and host loop minimization using CUDA graphs to reduce CPU-side scheduling delays, particularly in Python-based SGLang serving. Examples include improved fused routing and manifold-constrained hyper-connection kernels, with the latter reportedly achieving a 30% speed increase at batch size one. Baseten concludes that model-specific understanding, GPU-aware engineering, and automated kernel optimization can reduce inference latency and cost while preserving output quality.
Nov 10, 2026 1,920 words in the original blog post.
Genspark and Baseten Applied Research post-trained the open-weight Qwen3.5-397B model into a Deep Research agent that improved Genspark’s internal quality score from 0.612 to 0.759, exceeding several cited closed and open frontier models while crawling roughly five times more sources and generating longer, better-cited reports. The collaboration used Genspark’s production agent environment, evaluation harness, and reviewer feedback to identify that report synthesis was a larger initial weakness than web research, leading to a staged approach that applied reinforcement learning to search breadth and tool use, supervised fine-tuning to writing quality, and end-to-end reinforcement learning to combine both capabilities. Reward shaping combined an LLM quality judge with question-specific targets for distinct searches and websites, helping encourage broader but relevant research while limiting incentives to pad tool activity. The resulting model averaged 21.7 sites crawled and 28.5 tool calls per query, compared with 4.5 sites and 7.9 calls for the base model, and was quantized to NVFP4 for production deployment on two B200 GPUs per replica while maintaining target throughput.
Nov 10, 2026 2,787 words in the original blog post.
Baseten has announced the general availability of Serverless Loops, a training SDK for supervised fine-tuning and reinforcement learning on open-weight frontier models that uses a Tinker-compatible API and connects directly to Baseten Dedicated Inference. The service manages distributed-training infrastructure, including GPUs, kernels, model architectures, reliability, and numerical correctness, while offering post-training at up to 256,000-token context lengths, per-token pricing, and support for Qwen-3.6-35B-A3B, GLM-5.3-Flash, and GLM-5.3 models. Trained adapters can be deployed to Baseten’s autoscaling inference platform with one click or downloaded as Hugging Face Safetensors files. Baseten reports RL benchmark gains on held-out τ-bench tasks, including Qwen accuracy increasing from 51% to 72% after 50 steps, and cites early use by NVIDIA, Datadog, LangChain Labs, and Ambience Healthcare for vision, agentic, software-engineering, and clinical-language applications. Planned developments include additional models, quantization, score centering, and technical reports on training and inference optimizations.
Nov 06, 2026 1,095 words in the original blog post.
On-policy self-distillation (OPSD) trains a model on its own rollouts using a version of itself given privileged context, such as a reference answer, as a teacher, aiming to provide denser feedback than outcome-based reinforcement learning while avoiding mismatches between training and inference behavior. The authors argue that this token-level feedback is often biased toward a teacher’s preferred wording, formatting, and direct-answer behavior rather than toward actions that improve task correctness, causing it to suppress useful reasoning behaviors such as checking, hedging, and backtracking. They cite literature and experiments indicating that initial gains on smaller or non-reasoning models often weaken or reverse under stronger evaluation settings, on held-out tasks, and for larger models that already reason effectively. Their analysis of math rollouts finds that teacher disagreements frequently concern stylistic or semantically irrelevant tokens rather than mathematical errors, while experiments using unrelated worked solutions suggest that observed benefits may stem from induced answer structure rather than factual guidance. Attempts to remove teacher bias, including the authors’ contrastive OPSD method, still led to eventual training collapse in factual-learning experiments, with models producing hedging language instead of reliably answering correctly. The update concludes that OPSD is not a general replacement for reinforcement learning or a dependable reasoning-training method, though it may remain useful for distilling broadly desired contextual behavior such as system prompts or constitutions.
Nov 06, 2026 2,606 words in the original blog post.
Baseten has joined the NVIDIA Nemotron Coalition to collaborate on advancing open frontier models and helping enterprises customize, train, and deploy AI for specialized business needs. The partnership covers day-zero model launches, reinforcement-learning environments, post-training, dedicated inference infrastructure, and voice AI, with Baseten contributing production insights from enterprise workloads to NVIDIA’s Nemotron open-model ecosystem. Baseten has developed training environments for complex instruction following, long-horizon reasoning, planning, visual understanding, and tool use, reporting improvements in multimodal benchmark performance after post-training. Customer examples include CodeRabbit, which improved agreement with its code-review routing policy from 75.8% to 80.7% while reducing inference costs by 50.4%, and Harvey, which uses post-trained Nemotron models for legal workflows serving tens of thousands of lawyers. Baseten also supports real-time speech recognition with Nemotron 3.5 ASR Streaming and plans to expand its Loops platform, which combines managed reinforcement-learning workflows with dedicated production inference while allowing customers to retain ownership of their trained model weights.
Nov 05, 2026 834 words in the original blog post.
Baseten has introduced Sandboxes, persistent execution environments intended to support AI agents performing long-running tasks such as software development, research, and business workflows. Each sandbox uses a microVM for hardware-level isolation, can suspend while preserving its filesystem, memory, and running processes, and can resume or boot from standby in about 25 milliseconds, allowing compute costs to fall to zero during inactivity. The platform also provides built-in observability and is designed to scale to thousands of environments quickly. Sandboxes are described as the first component of Baseten’s planned agent-focused cloud, which will also include Agent Drive for shared distributed storage and Agent Proxy for controlled networking, credential injection, permissions, auditing, and firewalling. Baseten argues that agents require cloud infrastructure optimized for many stateful, secure, small-scale workspaces rather than conventional cloud systems built around a small number of large applications, and it offered Sandboxes free to run until November 16.
Nov 05, 2026 865 words in the original blog post.
Baseten has introduced Baseten Router, a model-routing system in public preview through Baseten Grid that selects between a frontier closed model, Opus 5.5, and a cheaper open-weight model, GLM 5.3 Flash, according to predicted task success and cost. On held-out coding benchmarks covering long-horizon software engineering, codebase Q&A, and terminal tasks, the company reports performance within 0.8 percentage points of always using Opus while reducing costs by 72%, with configurable thresholds for different cost-quality trade-offs. Baseten says its router was the only evaluated commercial option to exceed random-routing baselines by a statistically significant margin, although it notes that results may vary with benchmark selection and available model pools. The system was trained on verified coding-task traces and combines structured task characteristics with predictions derived from model activations to estimate response length and cost, while evaluations accounted for nondeterministic model behavior through repeated runs. Baseten also reports sub-second routing latency and is working with design partners to assess the approach in production, where incomplete requests, difficult-to-measure outcomes, multi-turn sessions, and model-switching costs present additional challenges.
Nov 04, 2026 3,142 words in the original blog post.
Baseten Grid is a public-preview enterprise platform designed to centralize AI coding model access, spending oversight, routing, and safety controls across providers including OpenAI, Anthropic, Gemini Enterprise Agent Platform, Amazon Bedrock, and Baseten. Its Baseten Router automatically selects approved open or closed models according to expected task quality, performance, and full completion cost, accounting for issues such as retries and failed tool calls; early benchmarks report up to 70% lower cost per coding task with a 0.9-percentage-point quality tradeoff compared with Opus 5.5. Grid’s gateway provides unified credentials, budgets, team-level permissions, usage analytics, and compatibility with developer tools such as Claude Code, Codex, OpenCode, and Pi, while its guardrails apply centralized configurable safety policies to model requests and responses. Built on Baseten’s inference infrastructure, the service also uses optimization techniques including caching and model-serving improvements to reduce costs, and planned features include cloud-hosted and coordinated agents, workload-specific routing, and expanded safety controls.
Nov 04, 2026 1,308 words in the original blog post.
Baseten announced an expansion of its unified AI platform with Grid, Sandboxes, and the general availability of Loops, aiming to consolidate model inference, training, routing, governance, and agent execution on shared multi-cloud infrastructure. The company argues that using separate vendors can create integration overhead, latency, and reliability risks, while customers such as OpenEvidence, Datadog, and Opendoor illustrate demand for tightly connected training, inference, and orchestration workflows. Grid, entering public preview, provides a model- and harness-agnostic gateway, intelligent routing, and guardrails intended to help enterprises control coding-agent spending, select models based on quality and cost, and improve cache efficiency. Sandboxes, also moving to public preview following Baseten’s acquisition of Blaxel, offer stateful, pause-resumable, VM-isolated execution environments designed to scale quickly and operate near inference infrastructure. Loops, a serverless platform for supervised fine-tuning and reinforcement learning on open-weight models, enables organizations to train and deploy customized models without managing underlying infrastructure while retaining ownership of their trained weights.
Nov 03, 2026 858 words in the original blog post.