Step Rejection Fine-Tuning: Squeezing More Signal from Noisy Agent Trajectories - The JetBrains Blog

Step Rejection Fine-Tuning (SRFT) is a novel approach developed by JetBrains Research to enhance the training of large language model (LLM) agents by leveraging unsuccessful trajectories, which are traditionally discarded in standard practices like Rejection-sampling Fine-Tuning (RFT). The method involves using a "critic" model to analyze training data and identify specific steps within failed attempts that are either productive or harmful, allowing the model to learn from the useful actions while ignoring the mistakes. This approach addresses the inefficiency of discarding entire runs due to final outcome failures, as research shows that even unsuccessful trajectories contain a significant amount of correct and valuable steps. Results from experiments using the SWE-smith and SWE-bench datasets demonstrate that SRFT improves model performance by selectively masking loss calculation on erroneous steps, thus extracting the beneficial elements of failed trajectories without reinforcing errors. This technique, while straightforward and cost-effective, requires careful tuning of the critic's strictness to maximize its effectiveness, making it a promising strategy for optimizing agent training outcomes.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.