How Numan built an eval-first AI health coach with LangGraph and LangSmith

Numan describes how it developed evaluation methods for Nu, an AI wellness coach for weight, hormone, and sexual health support, while maintaining strict boundaries against diagnosis, prescribing, or clinical decision-making under UK medical-device regulation. Early static single-turn and scripted multi-turn evaluations proved inadequate for coaching because they could not capture conversational context, flexible pacing, or responses that depend on what the agent actually says. Nu now runs through a LangGraph system of a supervisor and specialist sub-agents, evaluated using themed datasets scored by category rather than aggregate metrics, with all tests executed through the same production graph to prevent test-production drift. Safety is enforced through release-blocking boundary probes, graph-based guards that regenerate or replace unsafe responses, and daily human review by health coaches. A prototype simulation layer uses model-generated patient personas to hold adaptive multi-turn conversations, including adversarial and jailbreak scenarios, but its results will not govern safety claims until clinicians approve the rubrics. Numan emphasizes that clinician-owned evaluation criteria, rather than prompts alone, are the critical asset for ensuring both useful coaching and regulatory compliance, and it plans to expand simulated testing, online monitoring, and rubric-driven automation through LangSmith.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.