How to Build a Model Router in the Harness

LangChain describes building a task-aware model router for its Open SWE coding agent to reduce the cost of using frontier language models without lowering task quality. After analyzing a week of agent traces, the team found that requests varied substantially in complexity, with feature work generally requiring more turns and expense than testing or routine tasks, and selected three models representing fast, balanced, and performance tiers. The router, implemented within the agent harness rather than a generic gateway, classifies a thread’s first user message and assigns the least expensive model expected to complete the work, using task-specific criteria and a decision model for rapid classification. In an A/B test across 973 threads against an always-frontier-model baseline, routed threads had similar merged-PR and PR-open rates while median cost fell from $2.61 to $0.94, a 64% reduction; most requests were assigned to the balanced or fast tiers. A separate fast-only comparison was stopped early after users reported quality problems, illustrating the need to preserve stronger models for demanding work. The team proposes future improvements including benchmark-based evaluation, routing for subagents, possible mid-thread rerouting, and refinement using user sentiment, while recommending that developers analyze real task traffic, choose models along the cost-capability curve, integrate routing into agent-specific context, and track outcomes through evaluations, experiments, or feedback.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.