How We Built LangSmith Engine, Our Agent for Improving Agents

LangSmith Engine is an agent designed to enhance agent development by analyzing agent traces to identify recurring issues and suggest improvements. It operates as an orchestrator within the agent improvement loop, which includes building, testing, deploying, and monitoring agents. Engine identifies recurring failure patterns in traces, turns them into actionable issues, and proposes improvements such as evaluators and dataset examples. It uses a structured approach to handle large volumes of traces, employing a screening phase to identify suspicious traces and a deeper investigation phase for likely issues. Engine maintains an Issue Board where it records issues with evidence traces and proposes evaluators to catch similar patterns in future traces. The system relies on LangSmith CLI for interactions and uses a sandbox environment for operations, allowing it to adapt based on user actions and preferences. Through its architecture, Engine facilitates converting production failures into offline test coverage, enabling teams to improve their agents efficiently by focusing on recurring patterns and necessary fixes.