Evaluating OpenWiki with WikiBench

OpenWiki is an open-source agent for generating and maintaining codebase documentation, and WikiBench was created to assess both the quality of its generated wikis and their practical value to coding agents. Built on the Harbor agent-evaluation framework, WikiBench gives an agent a repository at a pinned commit, has it generate a wiki, and uses a reader agent to answer automatically generated coverage and retrieval questions about the codebase. Answers are scored against JSON rubrics of required facts, with LLM judges checking both factual inclusion and grounding in wiki pages the agent read. Tests found that OpenWiki 0.3.0 outperformed OpenWiki 0.2.5 and a general-purpose DeepAgents setup, largely because improved planning produced broader coverage, while retrieval improvements were smaller. Model comparisons showed significant tradeoffs among quality, cost, and runtime, with stronger models spending more effort reading repository files rather than producing substantially more pages. Experiments also indicated that using both a wiki and source code yields more accurate and cost-efficient answers than source alone, while a wiki without source performs considerably worse, positioning the wiki primarily as an index and guide for navigating codebases.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.