Sourcegraph MCP server and a cheaper model beat a Mythos-class model alone
Blog post from Sourcegraph
On the CodeScaleBench tasks aimed at assessing agent performance in large codebases, Claude Sonnet 4.6 integrated with the Sourcegraph MCP server outperformed Fable 5 by succeeding in six out of nine tasks, while also being more cost-effective in terms of quality points. Claude Sonnet 4.6, which does not store source code on disk and instead uses search, symbol resolution, and reference following, scored 0.698, whereas Fable 5, which processes the entire repository locally, scored 0.568 and incurred nearly double the cost per quality point. The results particularly highlighted Sonnet 4.6's superiority in cross-repository discovery tasks, such as vulnerability tracing and cross-organization dependency following, though one task did favor Fable 5, indicating genuine comparative assessment. These findings suggest that a more affordable model with efficient code retrieval capabilities could outperform a more expensive, locally-operating model in certain cross-repository tasks, though further evaluation is needed for tasks requiring deep single-file analysis.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| MCP | 4 | 6,026 | 689 | 188 | -15% |
| Observability | 1 | 3,430 | 674 | 183 | +0% |