October 2025 Summaries

Lumina is Baseten’s customer-in-the-loop evaluation framework for creating adaptive LLM-as-judge systems tailored to specific tasks and domain standards. It begins by generating model outputs across diverse models, identifying and clustering observed errors, and converting those failure patterns into precise binary pass/fail evaluators that collectively assess output quality. These evaluators are then repeatedly refined using consistency testing, separation between good and bad examples, static prompt analysis, genetic optimization, and customer feedback on ambiguous edge cases, with the goal of aligning judgments closely with expert preferences. The resulting evaluation sets can support benchmarking as well as model training, including supervised fine-tuning and reinforcement learning using evaluator scores and rationales to produce higher-quality outputs. Baseten plans to extend Lumina toward continuous production-data learning, more discriminative evaluation criteria, and specialized judge models for tasks such as error annotation, ambiguity detection, and criteria synthesis.

Baseten Training, now generally available, addresses key challenges in AI model training by providing a flexible, developer-friendly infrastructure that eliminates the complexities of managing compute resources, pricing, and operational control. This platform allows developers to train any AI model on any dataset effortlessly, offering tools like caching, checkpointing, and deployment options that enhance performance and reduce costs, demonstrated by OpenEvidence's reported 23x speed improvement and significant cost savings. The platform supports a wide range of training jobs, from small finetunes to large-scale reinforcement learning projects, and integrates seamlessly into existing workflows, allowing developers to focus on model training rather than infrastructure management. Baseten Training's enhancements since its beta launch include a collection of open-source recipes, improved UI, and more robust logging and observability features, enabling users to efficiently manage training projects and deploy models with ease.

Rationale-Guided Training (RGT) is a supervised training approach that aims to improve AI sample efficiency by pairing refined outputs with explicit strategy rationales derived from grader feedback, thereby teaching models not only which answers are correct but also the rules behind them. The method builds on iterative supervised fine-tuning, which corrects model outputs but normally discards information about where errors occurred and why, and represents this information through structured “think” tokens containing compact guidance such as avoiding unsupported diagnoses inferred from medications. In experiments on dental and emergency-department clinical-note generation using a Qwen3-32B model, RGT reportedly achieved 72–83% normalized accuracy after 25,000 examples, compared with 52–56% for standard supervised fine-tuning and 66–78% for iterative fine-tuning, while reaching a defined improvement target with substantially fewer examples. The authors argue that explicit strategies improve credit assignment at critical decision points and enable rules to generalize across varied cases, while requiring no architectural changes or reinforcement-learning pipeline; however, they note that both iterative fine-tuning and RGT incur additional evaluation and refinement costs relative to basic supervised training.

NVIDIA's Nemotron Nano 2 VL is an advanced open-source vision language model designed for high performance and scalability, particularly in the financial services sector, where it excels in tasks such as know-your-customer compliance, intelligent document processing, and fraud detection. Built on a base of the Nemotron Nano 2, a 9 billion parameter foundation model, Nemotron Nano 2 VL features a 12 billion parameter architecture with a hybrid Mamba-Transformer design, offering enhanced accuracy and efficiency for tasks like multi-image understanding, document intelligence, and video captioning. This model, available on Baseten, supports robust inference capabilities using NVIDIA NIM microservices for high throughput and low latency performance, with applications extending to various industries such as healthcare and media. Additionally, a smaller model, Nemotron Parse 1.1, provides a cost-effective solution for straightforward optical character recognition tasks. Baseten enhances the deployment of these models with enterprise-grade security, multi-cloud infrastructure, and technical support, making them suitable for building secure, reliable AI agents in production environments.

Attention-based attribution is presented as a more meaningful alternative to cosine-similarity methods for tracing which input content supports a generative model’s output, particularly in clinical documentation and other regulated settings. The discussion argues that embedding-based similarity can miss contextual relationships because it compares isolated text chunks rather than the model’s internal processing, while transformer attention can connect later statements to relevant earlier context. To make attention attribution practical, the approach uses selected middle-to-late transformer layers, sometimes from smaller proxy models, instead of storing all attention tensors from large generation models, reducing memory and latency costs while aiming to preserve correlation with fuller attention patterns. Layer choices and proxy-model performance were evaluated using chunk-level distance, divergence, and rank-correlation metrics, supplemented by clinician review. Baseten reports deploying this capability for output review, hallucination checks, and user-facing source links, and describes related work using residual-stream classifiers to identify unsupported generated content. The appendix notes that attention remains an imperfect proxy for causal influence and explores more comprehensive techniques, including attention rollout, Jacobians, and normalization methods for high-attention sink tokens.

Baseten reports that final training loss is a strong predictor of LLM-as-judge performance for non-verifiable, domain-specific tasks, potentially allowing teams to optimize fine-tuned models without repeatedly running costly evaluation suites. In experiments on clinical scribing, 15 Qwen-3 model configurations spanning 1.7B to 32B parameters and three dataset sizes showed that loss explained 74% of variation in pass rates, while controlling for dataset size raised prediction accuracy above 90%. The relationship held across Gemini-2.5-Flash and GPT-5-mini judges despite substantial differences in their absolute scoring strictness, suggesting loss tracks underlying capability rather than evaluator-specific behavior. Larger models and more training data generally improved results, though gains diminished at larger scales and one anomalous training run showed that more data did not always reduce loss. The authors state that similar findings have been observed in larger production deployments across medical, legal, and insurance tasks, but emphasize that the approach depends on carefully calibrated, low-variance LLM judges. They propose using loss to identify promising configurations and reserve expensive evaluations for strategically selected points, while investigating performance ceilings, reinforcement learning versus iterative supervised fine-tuning, and whether loss correlations can help assess judge quality.

Low-KL-divergence prompt mutations offer a low-cost way to improve the robustness and performance of fine-tuned language models used repeatedly with prompt templates. Rather than collecting additional labeled outputs or paying for costly reinforcement learning rollouts, the approach rewrites the static instructions in a prompt while preserving their meaning, then pairs these variations with the original variable inputs and gold-standard outputs to expand the training dataset. This reduces models’ dependence on the wording of a single template and encourages generalization, while avoiding the high distributional mismatch that can cause catastrophic forgetting when training without contextual instructions. The approach has reportedly doubled a 5,000-example dataset and produced above-trend results at minimal cost, and it may also improve resilience in on-policy distillation and reinforcement-learning settings. Future research will examine selecting or optimizing contexts that minimize KL divergence and determining how performance gains scale as more prompt mutations are added.

Baseten's advancements in speculative decoding, particularly with EAGLE-3, have significantly increased the performance of their GPT-OSS 120B inference API, achieving over 650 tokens per second, as verified by Artificial Analysis. As a launch partner for GPT-OSS, Baseten has utilized powerful NVIDIA hardware, including B200 GPUs, TensorRT-LLM, and NVIDIA Dynamo, to establish itself as the fastest NVIDIA-based provider, rivaling custom hardware providers. The implementation of techniques such as tensor parallelism and KV-aware routing have further optimized performance, while the potential for additional improvements through PD disaggregation and advanced speculation remains under investigation. Benchmarks from Artificial Analysis and OpenRouter highlight Baseten's superior performance, showcasing their ability to match or exceed custom hardware solutions while maintaining flexibility and scalability with NVIDIA GPUs. Baseten continues to innovate in model performance engineering and offers opportunities for further exploration and employment in this field.

DeepSeek-OCR is an innovative Optical Character Recognition model that revolutionizes data processing by utilizing a unique compression technique, reducing the need for visual tokens by tenfold compared to traditional text tokens, with a decoding precision of 97%. This efficiency not only allows the model to process vast amounts of data quickly and cost-effectively but also suggests a broader impact on AI intelligence by improving data representation for downstream tasks. The model's implementation on Baseten, using Truss and vLLM, demonstrates its scalability and reliability, even when faced with challenging inputs like doctors' handwriting. This approach highlights a shift in AI data processing from text to visual tokens, underscoring the potential for developing real-time AI agents and advancing document retrieval and question answering systems. The deployment process on Baseten, involving specific configurations and dependencies, illustrates the ease of integrating DeepSeek-OCR for various applications, offering a pathway to harness its capabilities for scalable training data generation and enhancing AI contextual understanding.

Baseten collaborates with NVIDIA to enhance model performance through the adoption of NVIDIA Dynamo, an open-source inference framework designed for large-scale LLM serving across distributed GPU clusters. A key feature of NVIDIA Dynamo is its KV cache-aware routing, which optimizes inference speeds by directing requests to model replicas with cached contexts, significantly reducing redundant computations and improving system performance. This routing approach, which balances cache hit rates and workload distribution, leads to substantial improvements in metrics such as time to first token (TTFT) and time per output token (TPOT), as demonstrated in benchmarks with models like Qwen3 Coder. Baseten has observed notable reductions in latency and increases in throughput, processing more requests per second and outputs per second. Looking ahead, Baseten plans to further leverage NVIDIA's tools, exploring features like KV cache offloading to enhance resource utilization and concurrency, and they will co-host a technical workshop with NVIDIA to share insights on maximizing AI inference workloads using NVIDIA Dynamo.

Iterative supervised fine-tuning (iSFT) is presented as an alternative to conventional SFT, which requires ideal demonstration data, and reinforcement learning, which often relies on sparse, brittle reward signals. The method generates an initial model response, evaluates it with a reliable grader that explains failures, repeatedly refines the response until it passes, and then uses the approved result as a supervised training example. Experiments on dental and emergency-department medical scribe tasks found that iSFT substantially outperformed standard distillation-based SFT and approached or trailed a more advanced method called RGT, while requiring far fewer examples to achieve comparable improvements. The argument for iSFT emphasizes that corrected sequences provide dense token-level supervision, whereas policy-gradient RL reduces feedback to a low-bandwidth scalar reward, making iSFT potentially more sample-efficient, stable, and economical when strong LLM judges are used. Compared with rejection sampling, iSFT can repair failed attempts rather than discard them, allowing it to create higher-quality data even when a base model rarely succeeds. Its effectiveness nevertheless depends on accurate, comprehensive evaluators, and future work may explore using more on-policy models for refinement to reduce distribution mismatch between generated targets and the model being trained.

Rank-1 LoRA adapters are proposed as a practical mechanism for making language models appear to learn continuously by applying small, immediate updates on the same GPUs used for inference, though the authors caution that this does not solve fundamental continual-learning problems such as catastrophic forgetting or modularity. Because individual SGD examples naturally produce rank-1 gradients, repeatedly accumulating rank-1 updates can build increasingly complex changes as gradient directions vary, while merging updates incrementally or in batches is approximately equivalent for small steps in supervised fine-tuning. The approach is especially memory-efficient compared with full fine-tuning and could support production systems that hot-swap tiny adapters for immediate behavioral changes, then merge them into base weights during low-load periods with rollback and provenance tracking. However, reinforcement learning introduces complications because policy updates alter the data distribution, and optimizer-state handling, particularly with Adam, remains a significant engineering challenge. The authors argue that adapter capacity is usually not the limiting factor, since LoRA functions as persistent write memory that distills generalizable feedback, unlike the transient read memory of an in-context-learning KV cache, while noting that alternative methods such as cartridges and sparse memory fine-tuning may ultimately be better suited to broader continual learning.

Autodesk's WaLa model is a groundbreaking single-view 3D reconstruction technology that transforms 2D images into detailed 3D models using wavelet-based latent diffusion, capable of producing high-quality OBJ meshes suitable for 3D printing or integration into game engines. The text outlines a tutorial on transforming this research model into a production API using Truss and Netlify, creating a scalable system that converts sketches into shareable 3D flower cards. It emphasizes the process of deploying the model as a scalable API on Baseten, which includes containerization and dependency management via Truss, and storing and viewing 3D models through Netlify's serverless functions and database integration. The resulting application offers an interactive web interface for drawing and transforming sketches into 3D models, complete with shareable URLs and a mobile-friendly, touch-responsive viewer built with Three.js. The project demonstrates the potential of turning open-source models into practical applications, with all code made available on GitHub for further experimentation and development.

Baseten’s experiments comparing full fine-tuning, LoRA, and LoRA-XS on clinical-scribing and insurance tool-calling tasks find that standard LoRA can generally match full fine-tuning quality while training far fewer parameters, particularly at ranks of 8 or higher for datasets up to 30,000 examples. Performance improved predictably with dataset size, while very low ranks such as 1 and 4 showed signs of capacity saturation as training data increased; final loss declined approximately logarithmically with rank across tested 4B and 8B models. Training loss showed a moderate, though not statistically conclusive, relationship with downstream LLM-as-judge evaluations, suggesting it may be useful for selecting checkpoints before running costly evaluations. In contrast, LoRA-XS, which trains a small matrix between frozen SVD-derived factors, did not reproduce prior claims of superior parameter efficiency: it required substantially higher ranks to approach standard LoRA performance, especially on a 32B model, and alternative initialization from early full-fine-tuning updates did not help. The authors conclude that LoRA is a practical production method for reducing optimizer memory, enabling larger-model training and multi-tenant adapter serving, while noting that rank requirements depend on task complexity, data entropy, and scale, and that LoRA-XS remains theoretically interesting but currently less effective for their workloads.

In an interview with Dax Raad, creator of OpenCode and Zen, various topics are explored, including the motivation behind developing OpenCode, the open-source philosophy, and the importance of terminal-based workflows. OpenCode was developed to address the cumbersome experience of using large language models (LLMs) in traditional environments, by integrating LLMs directly into the terminal where users can interact with their file systems seamlessly. Dax explains the necessity of open-source for OpenCode to leverage community contributions and adapt to the constantly evolving landscape of LLMs, contrasting this with closed-source solutions like Claude Code that do not benefit as much from community input. Zen is introduced as a complementary service to OpenCode, providing access to high-quality deployments of popular and new models in an economically sustainable way. Dax also critiques the over-reliance on benchmarks for evaluating AI products, arguing that real-world usability and user experience are more significant indicators of product quality, and highlights the "superstitious behavior" people develop around AI models due to their unpredictable nature.

Baseten's text-to-video system, running on Nebius using the Baseten Inference Stack, is designed to deliver predictable and efficient performance by integrating advanced technologies and infrastructure. The system's Inference Runtime utilizes custom modality-specific kernels, kernel fusion, and attention kernels to optimize video workloads, along with topology-aware parallelism and continuous batching to manage request prioritization and latency. Inference-optimized Infrastructure ensures reliable and scalable performance through intelligent request routing, geo-aware load balancing, SLA-aware autoscaling, and active-active reliability. These components work together to maintain consistent latency and throughput across varying traffic levels, enabling efficient scaling without compromising quality. The use of Nebius's large GPU pools and low-friction capacity growth complements Baseten's sophisticated runtime and infrastructure, ensuring a seamless experience even during demand spikes. The system's ability to adapt to real-time changes and manage multi-cloud capacity efficiently makes it robust enough to handle complex video generation tasks, turning a demo into a fully-fledged product offering.

Healthcare organizations are increasingly integrating AI into their operations to reduce costs, improve diagnostics, and enhance patient experiences, necessitating infrastructure that supports low latency, robust security, and scalability. AI teams are focusing on deploying and refining open-source models or developing custom models tailored to specific healthcare tasks, such as document processing, clinical assistance, and diagnostic image recognition, which require high availability and cost-effectiveness. To meet these demands, Baseten and Vultr provide a secure, HIPAA-compliant infrastructure using NVIDIA HGX B200 systems, enabling efficient AI model inference with flexible access to GPUs. This collaboration supports healthcare AI engineering teams in overcoming production challenges and achieving rapid market deployment with controlled costs, ensuring that AI solutions can handle high traffic volumes while maintaining performance and security.