A practical guide to hill climbing

A team used a method known as hill climbing to improve the performance of the Cline coding agent by running it against the Terminal Bench's 89 real-world coding tasks, which allowed them to increase their success rate from 47% to 57%. Hill climbing is an iterative process that involves running an AI agent on standardized tasks, making incremental changes, and keeping improvements if the score increases. The setup involves using tools like Harbor, a framework for managing and monitoring agent evaluations, which facilitates running these tasks in parallel for efficiency. The process also leverages Modal for faster execution by parallelizing tasks that would otherwise take much longer if done sequentially. Through systematic evaluation and adjustments, the team was able to surpass benchmarks set by other coding agents such as Claude Code. The guide emphasizes the importance of analyzing failures, A/B testing code changes, and using techniques like Pass@k for reliable results in noisy datasets, while iterating continuously to refine the model's performance.