February 2026 Summaries

RadixMLP is a technique developed to enhance the efficiency of batch inference workloads for causal transformer models by eliminating redundant computations for sequences sharing common prefixes. It leverages the position-wise nature of MLPs, LayerNorms, linear projections, and embeddings, dynamically mapping batches to a prefix trie to gather shared segments into a compressed form for computation, thus reducing redundant processing. In practice, RadixMLP has demonstrated significant speed improvements, achieving 1.44–1.59x speedups in realistic reranking workloads and up to 5x on synthetic benchmarks with longer shared prefixes, and it is integrated into platforms like Baseten Embeddings Inference. By maintaining a stateless design that operates entirely within a single forward pass, RadixMLP provides cache-like benefits without the overheads of persistent state management, making it suitable for scenarios with high prefix redundancy. Its compatibility with training and open-source release under the MIT License further enhances its accessibility and applicability for optimizing model performance in various inference tasks.

An exploration of the FLUX.2 [dev] model, a state-of-the-art diffusion model, reveals how 4-bit quantization can be leveraged to optimize image generation with minimal quality loss, achieving a 1.6x speed improvement. This process involves compressing the model's weights, which naturally leads to some information loss, but innovative techniques such as blockwise quantization and static post-training quantization help mitigate these effects. The model's architecture incorporates multiple transformer blocks and matrix multiplications, with a significant focus on optimizing the inference process by using lower precision FP4 computations instead of the standard BFloat16. By quantizing the model's weights and activations, and using a fixed global maximum for scaling, the team manages to maintain image quality while substantially reducing computation time. The effort underscores the potential of custom kernel engineering and inference optimization in enhancing model performance, exemplified by a significant reduction in latency from 2.776s to 1.81s.

MiniMax M2.5 addresses the limitations of traditional reinforcement learning (RL) by implementing a per-step process reward system that preserves intermediate signals and enhances agent performance over long trajectories. This innovative approach allows the model to achieve state-of-the-art benchmarks in tasks like coding and tool use, significantly reducing costs compared to closed-source models. By refining the Clipped Important Sampling Policy Optimization (CISPO) method and introducing token-specific rewards that optimize for both speed and quality, M2.5 tackles the credit assignment problem effectively, ensuring each action's contribution to the final outcome is recognized. The model demonstrates impressive generalization capabilities, outperforming competitors like Opus 4.6 on out-of-distribution tasks in the SWE-Bench Verified evaluation, while its affordability makes it a cost-effective solution for complex real-world tasks in finance, law, and social sciences.

Inferless, a company launched in 2022 to address serverless deployment bottlenecks for AI applications, is joining forces with Baseten to enhance their capabilities in supporting developers with mission-critical AI applications. Originally focused on creating a serverless GPU inference platform, Inferless tackled challenges such as reliable model deployment and effective scaling, particularly addressing the complex issue of cold starts in inference infrastructure. Recognizing the need for more comprehensive tooling to capture enterprise-level inference solutions, Inferless found alignment with Baseten's mission, which has concentrated on solving difficult infrastructure problems for AI systems. This merger aims to combine expertise to improve the Baseten Inference Stack, leveraging the rapid advancements in AI models and infrastructure to power scalable, reliable, and high-performance machine learning inference systems. As the AI infrastructure landscape continues to evolve, the teams look forward to making significant impacts for developers and customers in the sector.

Kimi K2.5 is a frontier-grade, open-source model boasting one trillion parameters, making it the largest of its kind on the market. Recently benchmarked by Artificial Analysis, it achieved remarkable speeds of over 340 tokens per second, emphasizing its efficiency in handling reasoning tasks through speculative decoding. Critical to this performance is the use of NVIDIA Blackwell GPUs, optimized through an INT4 to NVFP4 conversion for enhanced latency and throughput, and a custom-built EAGLE-3 speculator model trained on synthetic queries. These innovations enable Kimi K2.5 to outperform other models like Claude Opus, offering an 8x cost reduction and 4.5x speed increase, making it a compelling choice for applications in code generation and agent tasks. The model's success is attributed to a combination of inference optimizations, including the Baseten Inference Stack and KV-aware routing, which collectively ensure minimal latency and high efficiency for real-world applications.

As user expectations for AI performance evolve, developers are increasingly focusing on optimizing inference metrics that shape perceived performance rather than merely chasing benchmark numbers. The key metrics influencing user experience include time to first token (TTFT), tokens per second (TPS), and end-to-end latency, each impacting different aspects of user interaction. Developers must tailor performance improvements to their specific workloads, balancing trade-offs between cost, performance, and quality. While benchmarks provide foundational insights, real-world performance often requires fine-tuning to specific applications. Understanding user interaction patterns helps prioritize metrics that enhance user experience, as faster inference can sometimes compromise quality or increase costs. As AI models and user expectations continue to evolve, developers are encouraged to develop internal benchmarks and stay informed about new capabilities to maintain optimal performance.

OpenClaw, formerly known as ClawdBot and MoltBot, is a versatile Jarvis-like agent designed to function as a personal assistant or teammate, capable of executing a wide range of tasks on local machines or the cloud by running sub-agents with memory retention and tool usage capabilities. It can write, execute scripts, browse online, and use applications, with the added benefit of being monitored through chat platforms like Telegram and WhatsApp. When paired with Baseten’s Kimi K2.5 Model API, OpenClaw achieves performance levels comparable to closed-source models but at significantly lower costs, making it highly efficient for token-intensive applications. The setup process involves cloning a repository, installing dependencies, and integrating the Baseten API, allowing users to control OpenClaw through various interfaces, including the terminal, web pages, or messaging apps. The integration of open-source models like Kimi K2.5, GLM-4.7, and GPT-OSS 120B ensures users don't have to compromise between performance, quality, and cost, offering high throughput and low latency.

Large Language Models (LLMs) are complex tools that emulate human behavior, making their performance challenging to evaluate due to various interacting factors such as model type, hardware, and workload. SemiAnalysis has developed InferenceMAX, a benchmark focusing on inference speed across common hardware configurations, offering a reference point for the community. However, these benchmarks typically assess generic workloads, and for precise insights, users should conduct their own benchmarks tailored to their data. This article details replicating InferenceMAX on Baseten, utilizing TensorRT-LLM, and explores how the Baseten Inference Stack (BIS) can enhance performance through techniques like speculative decoding. Key patterns for effective model evaluation are provided, emphasizing the importance of server-side benchmarking to eliminate network variability and iterative benchmarking processes to refine configurations. The article highlights the significance of dataset selection, using production, public, and synthetic data for comprehensive performance metrics. While benchmarking can be complex, it is crucial for optimizing models and ensuring user satisfaction. The article suggests that a well-structured benchmarking approach serves as an early-alerting system, guiding decisions about models, configurations, and providers, and previews future exploration of realistic datasets and advanced techniques for improving inference performance.

Greg Schoeninger, founder of Oxen AI, shares insights from his journey working in AI since 2012, starting with early deep learning efforts and his experience at IBM Watson. Schoeninger discusses the development of Oxen AI, initially a version control tool designed to manage large machine learning datasets, and its evolution into a platform for fine-tuning models with integrated compute infrastructure. Oxen AI's approach allows companies to maintain control over their models, addressing issues like accuracy, speed, and privacy. The platform is particularly useful in industries like Hollywood, where pixel-perfect quality is paramount, and in advertising, where agencies use it to fine-tune models for branding. Oxen AI collaborates with Baseten to streamline the training, fine-tuning, and deployment of models, providing users with an efficient, end-to-end solution. Schoeninger emphasizes the importance of understanding specific requirements before selecting AI models and highlights promising models, particularly from Alibaba, in text and video generation.

The Baseten Inference Stack utilizes a combination of open-source tools like NVIDIA Dynamo and in-house innovations to optimize generative AI workloads with the lowest latency and highest throughput. By leveraging Dynamo, which is framework-agnostic and regularly updated, Baseten can integrate various inference engines tailored to specific models and use cases. The use of Dynamo facilitates improvements in system-level inference performance through optimizations like disaggregated serving, KV cache-aware routing, and KV cache offloading, leading to significant reductions in latency and increases in throughput. Baseten's engineers contribute to the Dynamo ecosystem by offering enhancements and new features, which were highlighted during NVIDIA's Dynamo Day event. These strategies enable Baseten to achieve a 99.99% reliability rate in their AI model performance, while also supporting multimodal model serving by extending the capabilities of Dynamo in handling complex AI workloads.