December 2025 Summaries

Fireworks RFT introduces a method for fine-tuning large language models (LLMs) using Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO), two techniques that enhance model responses by comparing preferred and dispreferred outputs. The blog post explores a simplified approach to GRPO, aligning it closely with DPO through theoretical and practical analysis, thus enabling the creation of a continuous model training pipeline that mimics Reinforcement Learning (RL) processes. This approach involves using a dataset where each prompt has two responses with one preferred over the other, allowing the model to learn by increasing the probability of preferred responses. The blog suggests that this methodology can be applied in real-world scenarios, such as customer support bots, to improve model performance over time through user feedback and continuous training. Fireworks.ai supports this process with tools and APIs that facilitate the implementation of recurring training workflows, promoting ongoing model enhancement.

Eval Protocol introduces an innovative integration with GEPA to enhance prompt optimization for open-source models without modifying model weights. This unified evaluation interface, which also supports reinforcement learning (RL), allows users to convert failure signals into actionable prompt improvements, improving model accuracy efficiently. In a case study involving a Text2SQL agent, GEPA's prompt optimization led to a significant increase in test and validation set accuracy, demonstrating the potential for substantial gains from reflective prompt adjustments. By using Eval Protocol, users can systematically write evaluations that not only assess but also enhance performance. This approach allows for a continuous improvement cycle where evals serve as both a diagnostic tool and a mechanism for performance enhancement, culminating in a seamless transition to techniques like RFT for even greater accuracy improvements.

NVIDIA Nemotron 3 Nano, an advanced reasoning model, has been launched with Day-0 support on Fireworks, promising to enhance next-generation AI agents with its cutting-edge hybrid Mixture-of-Experts (MoE) architecture. This model, building on the Nemotron 2 Nano, combines a new MoE design with a hybrid transformer-mamba architecture, optimizing compute efficiency and accuracy, especially for applications like financial fraud detection and cybersecurity threat triaging. The Nemotron 3 Nano features 30 billion parameters but activates only 3 billion for inference, ensuring streamlined performance with a long context length of 1 million. Fireworks, known for its high-performance AI Inference Cloud powered by NVIDIA's latest GPU architectures, provides proprietary optimizations and custom kernel techniques to maximize throughput while maintaining model quality. The platform supports the deployment of Nemotron 3 Nano, enabling developers to efficiently handle tasks such as code summarization with a hands-on cookbook to guide them through setup and use. This model is particularly suitable for edge deployments and interactive workflows, offering a robust solution for extracting structure from code and improving the efficiency of internal tools and documentation systems.

The blog post explores the evolution of AI agents from simple single-turn interactions to complex multi-turn, tool-heavy tasks, emphasizing the limitations of Supervised Fine-Tuning (SFT) and the advantages of Reinforcement Learning (RL) in these scenarios. It outlines the anatomy of a multi-turn RL system, highlighting the need for careful reward design and the challenges of training agents to effectively plan, call tools, and recover from mistakes. The post stresses the importance of trajectory-level rewards, environment stability, and using a strong base model to ensure success. It provides practical insights and strategies, including environment engineering, exploration encouragement, and training close to production, to make multi-turn RL systems work effectively. Through a case study of a deep research agent, the post demonstrates how RL can transform a weaker language model into a specialized agent that surpasses frontier models in specific workflows, underscoring the significance of RL in advancing AI capabilities for complex decision-making tasks.

Large language models (LLMs), traditionally used for free-form text generation, can be effectively adapted for classification tasks by leveraging their inherent ability to model token probabilities. This adaptation does not require altering the model's architecture; instead, it involves mapping each class to specific tokens and using the model's next-token probabilities as class probabilities, which can be used in applications like safety moderation, routing, and intent classification. This method is cost-effective and maintains compatibility with standard fine-tuning and inference APIs, making it suitable for small to medium label sets. Fine-tuning naturally calibrates these probabilities to reflect real-world likelihoods, eliminating the need for explicit renormalization. Empirical validation using the AG News dataset demonstrated that fine-tuning on a platform like Fireworks can achieve accurate and well-calibrated class probabilities at a low cost, confirming that LLMs can be adapted for classification tasks without significant modifications.

NVIDIA's Nemotron Nano 2 models, launched on the Fireworks AI platform, represent a significant advancement in efficient reasoning capabilities for AI models. These models leverage a hybrid Mamba-Transformer architecture, allowing them to maintain accuracy while reducing computational demands, particularly for tasks requiring long-context processing. The models excel in scientific research and code understanding by processing dense information and generating hypotheses with expert-level reasoning, achieving up to 62% accuracy on the GPQA Diamond benchmark, which surpasses the performance of models like GPT-4. Available in two sizes, Nemotron-Nano-9B-v2 and Nemotron-Nano-12B-v2, these models offer developers the ability to scale workloads efficiently on Fireworks' global infrastructure, benefiting from optimized speed, capacity, and reduced costs. This innovation is poised to enhance complex agentic applications, providing reliable decision support where simpler models fall short.