June 2026 Summaries

GLM-5.2 represents a significant advancement in AI models, offering a high-quality, open-source alternative to closed-source models, with impressive speed and cost efficiency. It is approximately 4.5 times faster and 5 times cheaper than similar models like Opus 4.8, making it an attractive option for scaling operations without compromising quality. The text outlines the process of integrating GLM-5.2 into various platforms, including Claude Code, Codex, and Deep Agents CLI, by utilizing API keys from Baseten, highlighting its flexibility in different environments. This model's adaptability and performance make it a preferred choice for organizations looking to enhance their AI capabilities effectively.

The text explores the distinct phases of AI model development, focusing on training and inference. Training involves teaching a model through exposure to extensive datasets, adjusting its weights to learn patterns and relationships, and may include a fine-tuning stage for specific tasks. In contrast, inference is the phase where the trained model generates outputs in response to new data, a process characterized by different hardware requirements and cost structures. The lifecycle of a model includes pre-training, post-training fine-tuning, optimization for specific hardware, deployment, and serving, with specific metrics like time to first token (TTFT), time per output token (TPOT), throughput, and latency being crucial for assessing inference performance. The text also details how Baseten, an inference platform, optimizes and facilitates AI deployment, offering solutions for custom models and automating infrastructure management, thus allowing teams to focus on model performance without dealing with the technical complexities of deployment and scaling.

Draft models, like EAGLE-3 and DFlash, are increasingly used to enhance large language model (LLM) inference by improving throughput and reducing latency, but aligning these models with base models and dynamic traffic patterns is challenging. A solution has been developed in the form of a distributed training pipeline that uses live inference to extract hidden states and train draft models in real-time, effectively bypassing the need for offline data storage. This approach has led to a median increase in accept rates by 20%, with some traffic patterns experiencing over 100% improvement, translating to faster speculative decoding and more efficient workloads. The architecture, integrated within the Baseten Inference Stack, operates with minimal overhead by using a highly optimized inference engine, leveraging GPU execution, memory management, and networking efficiency. The system also integrates frameworks like UCXX and Trio for robust networking and concurrency management, ensuring resilience against hardware failures and network disruptions.

NVIDIA's BioNeMo Agent Toolkit is designed to transform AI agents into specialized scientific tools capable of conducting biological and drug discovery tasks such as protein structure prediction, protein design, virtual screening, genomics analysis, and target discovery. By integrating BioNeMo Skills, open models, NVIDIA NIM microservices, and agent infrastructure, the toolkit facilitates complex workflows that require reasoning, planning, and execution across multi-step scientific processes. These domain-specific models, developed specifically for applications in biology, enable research teams to execute comprehensive discovery pipelines without managing the underlying computational infrastructure. BioNeMo NIM microservices, which are available in the Baseten Model Library, provide the necessary components for deploying and scaling AI-driven scientific applications. This approach shifts the traditional model of isolated inference to one where AI agents orchestrate a series of specialized models, enhancing the efficiency and effectiveness of scientific research and drug development.

Baseten has announced a significant $1.5 billion Series F funding round, led by Altimeter Capital, Conviction Partners, and Spark Capital, among others, marking their fourth fundraising event in 18 months. The company's rapid growth, with revenue increasing 20 times and inference volume growing 40 times over the past year, underscores the vital role of inference in the AI stack. Baseten has become a preferred partner for companies integrating intelligence into their user experience, collaborating with notable clients like Notion, HubSpot, and Abridge. The company emphasizes the importance of leveraging proprietary data and custom reinforcement learning environments to build durable competitive advantages, as highlighted by industry leaders like Satya Nadella. This funding will allow Baseten to invest in resources needed to support its customers as AI becomes integral to business operations, offering a reliable platform for post-training and inference. The company is expanding its team and capabilities to enhance AI infrastructure, encouraging potential collaborators and employees to engage with their mission.

GLM-5.2, developed by Z.ai, is a highly efficient and cost-effective large language model designed for complex tasks such as coding, offering significant performance improvements over its predecessors. The model's success is attributed to a range of optimizations, including the use of NVIDIA Blackwell GPUs with NVFP4 quantization, which enhances both time to first token (TTFT) and tokens per second (TPS). By implementing disaggregated inference with NVIDIA Dynamo, GLM-5.2 separates prefill and decode processes, reducing resource competition and improving throughput. Additionally, the model benefits from Multi-Token Prediction (MTP) layers that allow for speculation, enhancing token generation efficiency without compromising performance. These advancements enable GLM-5.2 to maintain superior operational performance in production environments, while also providing considerable cost savings compared to similar models like GPT 5.5 and Opus 4.8.

This blog post evaluates eight leading open-source language models (LLMs)—DeepSeek V4 Pro, Gemma 4, GLM 5.1, GPT OSS 120B, Kimi K2.6, MiniMax M3, Nemotron 3 Ultra, and Qwen 3.6—highlighting their strengths for various use cases like agentic coding, long context reasoning, and cost-efficiency. DeepSeek V4 Pro is notable for its innovative attention mechanisms and large context window, making it ideal for complex STEM and agentic coding tasks, while Gemma 4 excels in enterprise fine-tuning and multimodal reasoning with its attention strategies. GLM 5.1 is designed for long-duration coding tasks using a Mixture of Experts architecture to enhance computational efficiency, and GPT OSS 120B offers fast, cost-efficient text generation. Kimi K2.6 is praised for its reliability in coding and multimodal input capability, MiniMax M3 stands out for frontend and UI-related tasks, and Nemotron 3 Ultra is optimized for long-running agentic workflows with a unique Mamba-Transformer structure. Lastly, Qwen 3.6 provides advanced agentic coding and repo-level reasoning with strong multimodal support, showing improved performance over its predecessor. These models are currently deployed in production at Baseten, with each offering unique features that cater to specific AI applications and workload requirements.

Inception's Mercury 2, now available on Baseten, is the first inference platform to offer production-grade diffusion LLMs (dLLMs), providing a significant speed and cost advantage over traditional autoregressive models. It achieves over 1,000 tokens per second on standard NVIDIA GPUs, without the need for specialized hardware, making it both fast and economically viable for developers. Mercury 2's innovative architecture allows for parallel token drafting and refinement, enhancing speed and opening new application possibilities in areas like real-time voice agents and efficient coding tools. Notably, Augment Code has successfully implemented Mercury 2, reducing costs by 90% and latency by 82% for critical tasks. Baseten supports Mercury 2 with enterprise-grade infrastructure, ensuring reliable, scalable, and compliant deployment, which allows Inception to focus on optimizing AI workflows while maintaining efficiency.

Rolling deployments provide a method for incrementally updating model versions without doubling GPU usage or risking downtime, addressing challenges faced by users of other inference platforms who had to choose between costly blue-green deployments and risky hard cutovers. This approach replaces replicas one at a time, shifting traffic gradually, which allows for pausing, resuming, or canceling the deployment as needed, thereby reducing operational overhead and enabling more frequent updates. Two provisioning modes, max_surge and max_unavailable, cater to different constraints, balancing latency sensitivity and compute cost, with each step in the deployment process being managed by a durable workflow engine that ensures stability and records deployment history. Customers using rolling deployments report a 50-60% increase in deployment frequency, thanks to the reduced risk and the ability to manage deployments without off-peak manual supervision. The system's design allows for efficient coordination of traffic and scaling, even amidst load spikes, by maintaining a controlled traffic split and allowing time for operators to confirm system stability before proceeding to the next step.

Gabe Stern has been appointed as General Counsel at Baseten, bringing with him extensive experience from his previous roles at Synthesia and Slack. At Synthesia, he managed legal and regulatory affairs, aiding its rise as a leading AI video platform. At Slack, he was instrumental in establishing and expanding the legal department, supporting the company's international growth, public listing, and subsequent $28 billion acquisition by Salesforce. Stern's expertise spans a wide range of legal areas, including commercial, product, litigation, and privacy matters. Baseten, which is focused on developing infrastructure for production AI applications, values Stern's ability to build robust foundations while maintaining agility. The company is eager to leverage his skills to advance their mission and is actively recruiting for their team.

Nemotron 3 Ultra is an advanced mixture-of-experts (MoE) language model developed by NVIDIA, designed to enhance the performance of long-running autonomous agents by utilizing Mamba layers instead of traditional attention mechanisms. This architectural choice allows the model to maintain consistent processing speed regardless of task length, as Mamba layers' computational cost grows linearly rather than quadratically with context length. As a result, Nemotron 3 Ultra offers up to five times faster inference and up to 30% lower costs compared to other open frontier models, making it particularly effective for tasks such as coding, deep research, enterprise workflows, and chip design, where agent efficiency over extended operations is crucial. The model is fully open-source, allowing users to operate it on their own infrastructure, and it incorporates both Mamba and attention layers to balance efficiency with precision. NVIDIA's release strategy prioritizes task completion over single-turn benchmarks, making Nemotron 3 Ultra a practical choice for real-world applications where speed and reliability are critical.

Baseten and Microsoft AI have announced the integration of Microsoft's new reasoning model, MAI-Thinking-1, with Baseten, offering a unique blend of open-source control and proprietary model convenience. MAI-Thinking-1 is designed for AI teams who seek a balance between capability and control, featuring clean data lineage, post-training customization options, and a commercial-grade design that ensures data integrity and privacy, with Microsoft having no access to user data. This model is built on 35 billion active parameters for efficiency, providing significant capability at a cost-effective scale. Through Baseten, users can optimize and customize the model in a secure enterprise-grade environment, with the assurance that their fine-tuned models remain under their control. Baseten's training SDK, Loops, supports various post-training workflows, allowing businesses to maintain ownership of their AI developments. This partnership aims to empower AI-enabled businesses to control their intelligence, contrasting with models that bind users to external infrastructure and terms.

NVIDIA Cosmos 3 is a foundational AI model designed to revolutionize robotics and physical AI by providing advanced capabilities for understanding and interacting with the physical world. Unlike traditional generative video models that focus on producing visually appealing content, Cosmos 3 is built to reason about objects, actions, and cause-and-effect relationships, addressing the challenges of generalization in robotics, such as the complex task of opening doors. The model is deployed on Baseten and offers six capabilities through a unified architecture, including text2image, text2video, image2video, forward_dynamics, inverse_dynamics, and policy generation, which are crucial for creating action sequences and training data for robots and autonomous systems. Cosmos 3 supports two primary methods of use in robotics: directly analyzing observations and suggesting actions for research and prototyping, or serving as a data factory to generate synthetic datasets for training smaller, specialized robot policies. This approach enables more efficient and cost-effective training of robots by reducing the dependence on expensive, manually collected real-world data. Cosmos 3's emphasis on producing output that adheres to physical laws makes it a valuable tool for developing reliable and generalizable AI systems in contrast to other video models optimized for aesthetics.