Building Trust in the Harness Knowledge Graph with AI Evals | Harness Blog

Harness describes a multi-layered approach to building trust in its SDLC Knowledge Graph, which translates natural-language software delivery questions into Harness Query Language and answers them using connected operational data. The company found that plausible answers could still rely on unavailable entities, unpopulated relationships, stale ingestion data, or incorrect rankings and counts, so it evaluates the full path from question and query through schema, relationships, data freshness, authoritative APIs, and the product interface. Initial schema checks validated 19 of 36 prompts, while later API- and product-backed validation covered 22 of 27 scenarios, illustrating that entity availability alone does not guarantee usable or correct results. Harness combines repeatable AI evaluations measuring correctness, groundedness, relevance, and actionability with direct graph validation and comparisons against APIs or product views, and it automates much of this work through a /validateKG Claude Code skill. The approach uses production data to expose edge cases such as retries, deleted resources, null values, and ingestion delays, while integrating expected behaviors and validation methods earlier in feature development. Looking ahead, Harness plans to broaden graph coverage, assess anomaly explanations and recommendations, and run continuous evaluations to detect regressions as its data model and platform evolve.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.