March 2026 Summaries

TurboQuant, a novel quantization method known as PolarQuant, aims to address the memory bottleneck in transformer-based models by employing a unique approach that combines random preconditioning and polar transformation. Unlike traditional quantization techniques such as Nvidia's FP4, which use uniformly spaced buckets and require calibration data, PolarQuant analytically precomputes everything, effectively eliminating overhead. It converts KV embeddings into polar coordinates, compressing the KV cache significantly by leveraging the properties of multivariate normal distributions, where vectors behave like Gaussian outputs after random preconditioning. By recursively transforming pairs of coordinates into polar coordinates and applying a series of quantization steps, PolarQuant achieves efficient compression with minimal precision loss, offering a 4.13x compression rate. While the performance of PolarQuant lags behind cuBLAS in certain scenarios, particularly at shorter sequence lengths, the method presents a promising alternative with its unique no-overhead, distribution-aware quantization approach, although further optimization is necessary to enhance its competitiveness.

Sameer Paranjpye has joined Baseten to lead its engineering organization, bringing extensive experience from his previous role as VP of Engineering at Databricks, where he contributed to scaling the company into a prominent player in its field. Baseten, committed to developing a robust inference platform for the AI economy, sought an engineering leader with a strong technical background and a proven track record in managing and growing developer-focused products and teams. Sameer offers a unique blend of founder mentality, technical expertise, and go-to-market instincts, making him a valuable asset to the company. His appointment follows a comprehensive search involving hundreds of candidates, and Baseten expresses enthusiasm for the future with Sameer on board.

OpenClaw, a popular always-on agent for work applications, faces security challenges like prompt injections and remote code execution, prompting the introduction of NVIDIA NemoClaw for enhanced security. NemoClaw, an open-source stack, acts as a secure intermediary between OpenClaw and infrastructure, installing the NVIDIA OpenShell runtime to provide policy-based security and enabling claws, or subagents, to operate in isolated sandboxes with fine-grained controls. This system starts with zero permissions, ensuring that actions are strictly enforced at the infrastructure level rather than within the agent process. By utilizing open-source models like GLM-5, MiniMax M2.5, and Nemotron 3 Super, NemoClaw allows developers to overcome the limitations of closed-source models in reliability and cost. Baseten supports NemoClaw by hosting these models on HIPAA and SOC 2 compliant APIs, making it suitable for regulated industries such as insurance and financial services. NemoClaw simplifies running always-on assistants with a single command, offering an efficient solution for organizations requiring secure, sandboxed environments to manage intelligent agents.

The Baseten Delivery Network (BDN) has been launched to significantly reduce cold start times for large-scale models, achieving 2-3x faster initialization through multi-tier caching and single-flight downloads, which address issues such as thundering herd problems during burst scaling. BDN is integrated with the Baseten Inference Stack and is designed to overcome common challenges associated with cold starts, including hardware provisioning and weight downloading—particularly for models with tens to hundreds of billions of parameters. By mirroring weights to infrastructure at push time, employing a multi-tier cache system, and ensuring single-flight weight downloads, BDN eliminates dependencies on third-party services and reduces bandwidth contention, ensuring consistent and rapid scaling even in high-demand scenarios. Additionally, BDN accelerates the entire inference loop by distributing various deployment artifacts, providing immediate improvements in cold start times and efficiency gains throughout the model deployment lifecycle, and is now available to all Baseten Cloud customers.

Matt Slagle has joined Baseten to lead and expand their global revenue organization, bringing a wealth of experience from his previous roles as SVP of Sales at Sigma and in sales leadership at Fauna and New Relic. Baseten aims to become a key player in powering significant AI products globally, which requires a robust customer organization capable of collaborating with cutting-edge AI teams. Slagle is praised for his ability to build high-performance, technical, and customer-focused sales teams, and he embodies the values of integrity, intensity, low ego, and pride in his work, which align well with Baseten's ethos. Baseten is actively hiring, inviting others to join their mission to create a leading inference platform for production AI.

NVIDIA's Nemotron 3 Super is a cutting-edge open-source AI model designed to optimize multi-agent systems, particularly in complex financial service applications. With a hybrid mixture-of-experts architecture, it features 120 billion parameters and utilizes 12 billion active parameters per forward pass, achieving faster token generation than its competitors. It excels in benchmarks like AIME 2025 and RULER, showcasing its efficiency and accuracy in tasks such as math reasoning and long-context retrieval. Its architecture allows for reduced communication overhead and efficient multi-token predictions, making it ideal for operations like loan processing, fraud detection, and cybersecurity. The model's design supports the orchestration of numerous simultaneous agents, offering flexible deployment options, including single-tenant and self-hosted setups to meet compliance requirements. Nemotron 3 Super, part of a family of models, serves as the central reasoning layer in agentic pipelines, with its architecture ensuring robust performance across a range of demanding tasks.

Z.ai's GLM-5, an open-weight model developed by Andon Labs, has achieved state-of-the-art results in both time to first token (TTFT) and tokens per second (TPS) with its innovative use of a mixture of experts (MoE) architecture, which selectively activates parameters based on the task at hand. This model, which is more than twice the size of its predecessor GLM-4.7, excels in tasks such as code generation and agentic reasoning, and ranks highest among open-source models in the Vending Bench 2 benchmark, which assesses a model's decision-making capabilities over a long time horizon. By leveraging custom kernels optimized for DeepSeek Sparse Attention and a low-overhead Multi-Token Prediction (MTP) speculative decoding engine, GLM-5 achieves 186+ tokens per second, making it the fastest in inference as benchmarked by Artificial Analysis. The Baseten Inference Stack enhances performance through KV-aware routing, MoE dispatch kernel optimizations, and NVFP4 quantization for compatibility with Blackwell inference. These innovations underscore GLM-5's suitability for complex systems engineering and autonomous coding tasks, offering industry-leading throughput for open-source models.

A company specializing in converting emergency department (ED) conversations into structured clinical charts faced the challenge of creating a model that could manage the complexity of emergency medicine documentation in real-time. This task required transforming ED transcripts into structured JSON and then merging this with physical exam templates to produce accurate charts. The project involved developing a model that exceeded the performance of existing models by achieving higher accuracy and speed, specifically outperforming the gemini-2.5-pro model. To achieve these results, the team implemented a simplified two-stage pipeline and used iterative SFT (iSFT) training with dense feedback, allowing the model to handle high-risk diagnoses and make precise edits to documentation templates. The model's success was bolstered by comprehensive evaluation frameworks and a focus on multi-stage robustness, enabling it to process tens of thousands of ED notes weekly and expected to expand further. This innovative approach ensures both speed and accuracy in emergency documentation, addressing the critical need for precise application of clinical rules and semantic understanding in regulated medical environments.

A company developed a model to convert emergency department (ED) conversations into structured clinical charts, addressing the complexity and real-time demands of emergency medicine documentation. The model operates in two stages: transforming ED transcripts into structured JSON with 16 sections and merging this with physical exam templates to create the final chart. It outperforms existing models like gemini-2.5-pro in both accuracy and speed, achieving 84.8% accuracy in chart generation and 67.2% in summarization while running 6-8 times faster. The model successfully handles complex routing logic, conditional inclusion, template modification, and high-risk diagnosis detection, areas where general-purpose language models typically fail. By using iterative supervised fine-tuning (iSFT) and creating comprehensive evaluation frameworks with Lumina, the model ensures high precision and robustness. The pipeline was simplified, reducing latency significantly by consolidating multiple prompts and replacing pattern-matching tasks with deterministic code. This model processes tens of thousands of emergency department notes weekly, with an expected increase in usage, demonstrating its scalability and potential impact in emergency settings.