January 2026 Summaries

Baseten has developed a hybrid speculative decoding method that integrates a suffix automaton with multi-token prediction (MTP) to enhance the efficiency of token prediction in applications such as code generation. This approach reduces latency and increases throughput by allowing for more efficient token verification and prediction, achieving up to 40% higher throughput and lower latency compared to MTP alone. The suffix automaton offers improvements over traditional n-gram lookups by enabling predictions of arbitrarily long patterns and updating in real time, which is particularly effective in scenarios involving long, repetitive sequences. Baseten's speculation engine, part of their inference stack, switches between suffix automaton matches and MTP based on the length of the match to optimize prediction efficiency. This method has been integrated into the open-source NVIDIA TensorRT-LLM, ensuring minimal overhead and high performance in production workloads. The implementation leverages CUDA for efficient data transfer and processing, maintaining high GPU utilization with minimal idle time. Furthermore, the approach is compatible with existing MTP/EAGLE setups and offers potential for future enhancements, such as continuous model training and dynamic-length speculation.

Baseten has announced a successful $300 million fundraising round at a $5 billion valuation, led by IVP and CapitalG, with participation from several other investors including NVIDIA and Greylock. The company, founded six years ago, aims to harness specialized AI models to drive impactful real-world applications, with recent developments such as reinforcement learning enhancing model customization for specific workflows. Baseten has seen substantial growth, with inference volume increasing 100 times in the past year, as they partner with leading companies like Notion and OpenEvidence. As AI continues to integrate into various consumer and enterprise products, Baseten is focused on developing an AI infrastructure platform that prioritizes speed, uptime, and developer experience, intending to expand its capabilities to meet increasing demand for AI-powered solutions. This latest funding round, the third in the past year, will support the company's vision to amplify and accelerate the impact of AI, inviting collaboration from those interested in advancing AI infrastructure.

Baseten has developed a hybrid speculative decoding method that combines n-gram speculation and draft model speculation, such as EAGLE or multi-token prediction (MTP), using a suffix automaton to improve token prediction accuracy and efficiency in applications like code generation. This method, integrated into TensorRT-LLM, enhances performance by batching the token verification phase, resulting in up to 40% higher throughput and lower latency compared to MTP alone. By leveraging a suffix automaton for decoding, it identifies long patterns and updates in real-time, leading to higher acceptance rates on long sequences. The hybrid approach allows dynamic adaptation to workload requirements and exhibits interoperability between C++ and CUDA, thus achieving zero overhead while maintaining high GPU utilization. Future work includes continuous draft model training and dynamic-length speculation, indicating further potential for efficiency gains in speculative decoding without altering configuration parameters.

Baseten has developed an optimized runtime for the Wan 2.2 video generation model, resulting in significant performance improvements on NVIDIA Blackwell and Hopper GPUs, achieving up to 3.2 times faster inference on the former and 2.6 times on the latter compared to the default runtime. This enhanced runtime reduces costs by 67% for high-volume deployments and is achieved through a series of kernel optimizations, including improvements to CUDA kernels such as RoPE attention, LayerNorm, and RMSNorm, alongside advancements in the inference engine that maximize GPU utilization. The optimizations maintain output quality while enhancing speed, leveraging techniques such as Ulysses Sequence Parallelism and fine-tuned parameters like sample steps and frame numbers to ensure robust performance across varied video generation requests. These improvements are part of Baseten's broader inference stack that supports large-scale video generation for AI-focused companies, ensuring reliability during demand spikes, and the company continues to explore further advancements in speed-quality trade-offs, suggesting future publications on lossy quality optimizations.

Since 2024, the Whisper transcription service has been advancing in speed, accuracy, and cost-efficiency, with the latest improvements featuring real-time, speaker-aware transcription that is even more rapid and affordable. The service is engineered for flexible production applications, allowing customization for various use cases with or without streaming or diarization, and offering the ability to configure the number of GPUs used. Built on Baseten Chains, the Whisper transcription pipeline achieves significant cost savings and performance improvements over competitors, and now includes features like streaming audio transcription and speaker annotation for real-time applications. These advancements cater to industries requiring live note-taking, content captioning, customer support, and other voice-driven applications, and the system's diarization capability is particularly suited for speaker-aware conversational AI apps. The technology, which powers products like Notion's AI Meeting Notes, has been validated under heavy load, maintaining accuracy and cost-efficiency even with thousands of concurrent audio streams.

LangChain's LangSmith Agent Builder offers a streamlined, no-code solution for quickly creating production-ready agents by using Baseten's GLM 4.7 LLM to ensure fast performance. Users can develop agents tailored to specific tasks, such as meal planning or email assistance, without needing extensive technical expertise. The process involves describing the agent's purpose, selecting suitable tools, including web search functionalities like Tavily, and refining its capabilities through guided questions. While simple to use, users must have accounts with LangSmith, Baseten, and Tavily to access necessary resources. The system automatically generates detailed prompts and tool selections, reducing the complexity of agent development and enhancing the accuracy and quality of outputs. LangChain emphasizes the cost-effectiveness and high performance of its solution, encouraging users to explore its capabilities for various business functions.

A leading digital insurer partnered with Baseten to develop a state-of-the-art AI system that efficiently handles customer insurance queries while maintaining strict compliance with Financial Conduct Authority (FCA) regulations and delivering sub-3-second response times. The project involved deploying a fine-tuned open-source model that surpassed existing closed-source models, including GPT-5, in reasoning capabilities and speed. The development process was divided into several phases, starting with building a sophisticated knowledge retrieval system and evolving through a comprehensive evaluation framework that identified failure modes and refined prompts for consistency and accuracy. Genetic optimization and supervised fine-tuning were employed to create high-quality training data, with reinforcement learning further enhancing the model’s ability to perform precise retrievals and generate compliant responses. By eliminating vector embeddings in favor of grep-based text search, the system improved robustness and adaptability to document changes. After six weeks, the AI achieved a 95.7% compliance pass rate, significantly outpacing competitors in speed and accuracy, and set the foundation for scaling the system to support additional insurance products and more complex interactions.

Dentists face challenges in converting patient conversations into structured clinical documentation, prompting the development of a specialized low-latency model by Parsed in collaboration with a dental note-taking leader. This model efficiently performs three tasks: transforming ambient transcripts into structured notes, updating notes in real-time, and enhancing existing notes. It excels in handling complex dental terminology and various tooth notation systems, achieving faster performance and high accuracy compared to other models. The development incorporated a comprehensive evaluation framework using Lumina, which helped identify unique errors and improve model training through iterative supervised fine-tuning (iSFT), surpassing traditional reinforcement learning in data efficiency. Furthermore, synthetic data generation was employed to address domain-specific challenges like tooth notation systems, significantly enhancing the model's ability to internalize specific knowledge. This approach not only resulted in a model that matches the accuracy of slower models like gemini-2.5-pro but also demonstrated the potential of specialized models to outperform general-purpose systems in regulated fields like healthcare, emphasizing the importance of evaluation-driven development.

Baseten is demonstrating how small open-source models, when paired with rigorous evaluation and task-specific optimization, can outperform larger proprietary models in complex real-world applications, such as healthcare scribing. By leveraging the concept of compute-optimal training, Baseten emphasizes the importance of balanced parameter-to-token ratios and the effectiveness of smaller, task-optimized models that can deliver 60% better accuracy, lower inference costs, and faster processing times compared to larger models. Baseten's approach involves building a programmatic, domain-aligned evaluation system that breaks tasks into granular checks and integrates these into training and deployment pipelines. This evaluation-first methodology, which includes the use of mechanistic interpretability techniques, not only enhances model performance but also ensures transparency and reliability. In a healthcare use case, Baseten's approach to fine-tuning a 27B parameter model resulted in surpassing the performance of larger models such as Claude Sonnet 4, achieving significantly lower latency and cost while maintaining high accuracy and reliability. Their methodology includes a sophisticated evaluation harness that aligns with expert clinical judgment and supports continual reinforcement learning, thus providing a foundation for sustained improvement and cost efficiency in domain-specific applications.

Baseten has introduced a new training infrastructure designed for flexibility and scalability, allowing teams to run their existing code on scalable compute without complex abstractions. This infrastructure is aimed at providing a seamless transition from training to deployment by integrating with Baseten's inference stack, making it suitable for custom models, audio models, and multi-node jobs. Early adopters like OpenEvidence and Oxen have utilized the platform to enhance inference speeds and build comprehensive platforms. Unlike point-and-click solutions, Baseten's platform supports diverse model training needs, offering users the ability to experiment and optimize their models with hands-on control over the code and parameters. The infrastructure is designed for easy migration from existing setups, offering on-demand compute and persistent storage to streamline the iteration process. Baseten's focus on inference expertise enhances its training offering, ensuring that models transition smoothly into deployment. The development process involved close collaboration with early customers to refine the product, leading to features like private Docker registry support based on feedback. Baseten’s training infrastructure allows for direct interaction through APIs as well as CLI, making it adaptable for various use cases and platforms.