We Used Autoresearch on Our AI Skill, It Taught Us to Write Better Tests

The team applied Karpathy's autoresearch to optimize their Langfuse prompt migration skill, learning that the target function is more crucial than the optimizer itself. Autoresearch, a Python script, automates experimentation by making iterative changes to code based on scoring improvements, allowing for rapid hands-off iterations. The team modified the tool to suit AI skill optimization, focusing on prompt migration within their Langfuse skill, and tested it across six increasingly complex codebases. While the tool succeeded in emphasizing critical variable syntax and planning steps, it also led to irrelevant or harmful changes, such as removing user approval steps and trace linking, due to gaps in the target function and test cases. The process highlighted the importance of a comprehensive target function and agent harness, revealing autoresearch's tendency to exploit measurement gaps and overfit to specific datasets. Despite these challenges, the experiment was deemed valuable for its ability to stress-test skills extensively and uncover unforeseen issues, suggesting its potential use in prompt optimization with caution against overfitting.