August 2026 Summaries

GLM-5.3 is presented as an example of improving an existing GLM-5.2 base model through large-scale reinforcement-learning post-training rather than a new pretraining run, with gains attributed to more realistic, agent-oriented training environments, scalable rollout generation, and infrastructure optimization. Its environments emulate expert tasks such as ML systems debugging, use automated solvability checks, and apply three-stage verifier tests intended to prevent reward hacking by rewarding only correct and complete work. The model retains GLM-5.2’s 744-billion-parameter mixture-of-experts architecture, activating roughly 40 billion parameters per token, alongside latent attention, sparse attention, and multi-token prediction mechanisms designed to reduce memory and inference costs. An IndexShare technique reuses sparse-attention token selections across layers, reportedly lowering long-context computation and making RL rollouts cheaper. Training uses Single-Rollout Asynchronous Optimization with trajectory compaction to support stable, long-horizon learning, while the slime framework coordinates SGLang inference with Megatron distributed training. Additional features include multi-teacher on-policy distillation, dynamic teacher loading, and scheduling intended to keep GPUs utilized, collectively illustrating an approach to iterating on open-weight models through post-training compute and engineering.

An agentic kernel-development framework is presented to bridge the gap between isolated kernel benchmarks and production inference optimization by profiling full model workloads, identifying model-level and per-kernel opportunities, generating implementations, and validating correctness and end-to-end performance within serving engines. Its model-level layer targets changes such as operation fusion, redundant-work removal, and reduced intermediate materialization, while its kernel-level layer explores implementation variants for critical kernels; successful patches and lessons from failures are retained in a knowledge base for future runs. Applied to Qwen-Image and FLUX.2 diffusion models on B300 GPUs using SGLang, the system achieved reported end-to-end latency reductions of 42.3% and 15.2%, respectively, through changes including pre-packed FP8 scales, fused QKV projections, normalization and quantization fusion, cached classifier-free guidance modulation, fused QK normalization with RoPE, fused SwiGLU quantization, and gated residual normalization. The framework is intended to be model- and serving-engine-agnostic, and early LLM experiments on vLLM reportedly improved throughput by up to 5.5% on MiniMax M3 and GLM-5.2, supporting the goal of generating kernels specialized for particular hardware, tensor shapes, and serving workloads.

AI gateways centralize decisions that accompany model requests, including authentication, routing, rate limits, resilience, usage attribution, and auditing, reducing the fragmentation that occurs when these functions are spread across application code and operational systems. The text distinguishes between access gateways, which help applications use multiple external model providers through a unified interface, and serving gateways, which help model owners expose their own models as secure, multi-tenant customer APIs. Serving gateways are designed to connect customer identities and commercial policies with inference infrastructure, enforcing tenant isolation, quotas, capacity protections, metering, and traceability across each request. Gateway evaluation should therefore focus on alignment with a company’s customer model, billing needs, inference awareness, deployment location, portability, and remaining operational tooling rather than generic claims of routing or observability. For organizations commercializing their own models, placing gateway controls near inference can allow policies to be applied before expensive execution and improve visibility into deployment health and consumption. Baseten presents its Frontier Gateway as a managed serving-side option for models hosted on its Dedicated Inference platform, offering branded endpoints, API-key management, usage limits, consumption tracking, and inference-integrated operations, while noting that it is not intended as a universal provider-routing proxy.

LLM inference observability combines metrics, logs, and traces to identify and resolve production issues such as slow responses, errors, failed builds, and deployment problems before they affect users. Important metrics include time to first token, time per output token, tokens per second, end-to-end latency, and KV cache hit rate, with cache reuse improving speed and reducing compute costs by avoiding repeated prefix processing. Build logs diagnose container assembly issues such as missing packages, CUDA incompatibilities, and download failures; deployment and promotion logs connect operational changes with error or scaling events; and serving logs capture runtime behavior, including model-loading failures, exceptions, and GPU memory problems, often using request IDs for investigation. Traces provide a timed view of a single request through components such as the API gateway, service mesh, activator, queue, and model server, helping distinguish model-compute delays from cold starts, queue congestion, or routing failures. The approach emphasizes integrating inference monitoring with broader application observability, while platforms such as Baseten can provide dashboards and export metrics and logs to external monitoring systems.

DeepSeek Harness (DSH) is presented as a modular interface for operating multiple AI coding and agent harnesses rather than a single harness, allowing models, tools, file systems, sandboxes, loops, and subagents such as Claude Code or Codex to be swapped and combined through plugins. Its append-only event log supports tracing, forking, replaying, and modifying agent runs, making it useful for debugging failed trajectories, inspecting tool calls, monitoring performance, and collecting post-training or reinforcement-learning data. The platform is also designed to be self-extensible, enabling users to inspect its runtime and add plugins without disrupting existing workflows. The post explains how to connect DSH’s web interface to Baseten’s OpenAI-compatible Model APIs by cloning and building the repository, configuring a custom provider with a Baseten API key, fetching the available model catalog, and selecting open-weight models such as Kimi K3, GLM 5.2, and DeepSeek V4 Pro. Users can then inspect each model run through a timeline that displays event logs and metrics including latency, time to first token, throughput, and cache-hit rates.

DeepSeek V4 Pro 0813, a 1.7-trillion-parameter MIT-licensed open model, has been released with improved post-training for code generation and agentic tasks while retaining the base architecture of the earlier Preview version. Positioned near GLM-5.2 on Artificial Analysis’ intelligence index and described as less costly per task, it includes first-party benchmarks and an open-source coding harness built around a plugin-first design. Baseten has made the model available through its API and dedicated deployments with zero data retention by default, adapting its inference configuration to changing agentic-coding workloads through tuning of parallelism, KV-cache allocation, and prefill-decode workers. The release also introduces revised input and output formatting specifications and includes a DSpark speculator to support speculative decoding and improve token throughput, though specialized deployments may benefit from a custom-trained speculator. Baseten lists pricing of $1.32 per million uncached input tokens, $0.132 per million cached input tokens, and $3.96 per million output tokens.

You.com developed its Answer API to provide synthesized, fact-checked, and cited responses from live web data, addressing the limitations of LLMs that rely on static training information. Processing tens of millions of daily queries, the company partnered with Baseten to run open-source and fine-tuned models across dedicated multi-region infrastructure for tasks including entity recognition, embeddings, query expansion, reranking, intent prediction, and answer generation. Baseten serves models such as GLM-5.2 and Kimi K3, enabling You.com to combine web retrieval with lower-cost open-weight inference rather than relying entirely on proprietary frontier APIs. The production deployment reportedly achieved about a 60% latency improvement and 50% cost reduction while maintaining accuracy comparable to proprietary models, including a 93.48% Simple QA accuracy rate, 2.67-second median latency, and $5 cost per 1,000 calls for the Answer API.

NVIDIA Nemotron 3.5 Lightning is a 30-billion-parameter mixture-of-experts model with 3 billion active parameters, distilled from Nemotron 3 Ultra and designed for high-volume, always-on agentic applications such as personal assistance, financial services, cybersecurity, telecom, and retail. Available through Baseten Dedicated Inference on NVIDIA infrastructure, it supports a one-million-token context window and is presented as delivering nearly four times the throughput of similar open models while reducing task-completion time by 30% through faster reasoning and token generation. Baseten and CodeRabbit tested the model for high-volume code-review routing, using supervised fine-tuning and reinforcement learning to improve route agreement and reliability. The resulting rank-16 LoRA adapter reportedly achieved about 4% higher accuracy than the baseline, used roughly half the API cost, generated 63.4% fewer output tokens, and reached 314.82 aggregate output tokens per second across eight concurrent requests on an A100 GPU.

NVIDIA Nemotron 3.5 ASR is a speech recognition model available in the Baseten Model Library, designed for low-latency, high-accuracy transcription in both English and multilingual contexts, supporting 40 language locales. Featuring a cache-aware FastConformer-RNNT architecture with a 24-layer encoder and an RNNT decoder, it offers efficient streaming inference with NVIDIA Inference Microservices (NIM), achieving high concurrency and throughput on a single H100. The model maintains consistent finalization latency and time-to-first-token across increased real-time streams, crucial for interactive applications. It demonstrates competitive word error rates (WER) on benchmarks, such as achieving 2.32% WER on LibriSpeech Clean for English and an average of 8.84% WER across multiple languages on the FLEURS benchmark. Developers can further customize Nemotron ASR for specialized domains or specific accents using NVIDIA NeMo through Baseten Training, and the models are available under the OpenMDW-1.1 license.

Poolside Laguna S 2.1, a 118B-parameter Mixture-of-Experts model, was tested on its ability to transform the codebase of Hypersomnia, a multiplayer game, from a neon cyberpunk style to an Ancient Greek theme. The task required Laguna to navigate a complex repository, coordinate changes across files, and use external AI models without compromising the game's core functions. Laguna orchestrated specialized models, such as Step 3.7 Flash for language adaptation and Krea 2 Turbo for visual asset generation, while choosing not to employ the Cosmos 3 Nano model, as it found no fitting role for video content within the game. The result was a Greek-themed playable single-player tutorial called Hyperion, demonstrating Laguna's capacity for creative and technical judgment, though it was not ready for multiplayer production. This undertaking highlighted the model's ability to manage repository-scale transformations by balancing creative ambition with technical constraints, all within a limited timeframe of 98 minutes and 58 seconds.