Testing LLM Responses You Cannot Predict [Testμ 2026]

Gil Zilberfeld’s Testμ Conf 2026 session argues that testing LLM-based systems requires replacing fixed pass-or-fail assertions with a layered evaluation funnel because identical prompts can produce different responses across runs. He recommends thoroughly testing deterministic scaffolding, APIs, integrations, error handling, and guardrails with conventional tests, then applying low-cost sanity checks to model outputs for required sections, relevant entities, timing, and other baseline criteria before deeper evaluation. Semantic quality should be defined through small human-authored golden data sets that specify what acceptable responses contain, enabling scorecards that rate multiple criteria and track quality trends over time rather than relying on a single snapshot. The session emphasizes that prompt changes alone are not reliable bug fixes, illustrated by an example in which a request for concrete test data led to credit-card-like values; durable fixes require code-level enforcement, updated privacy requirements, and automated checks. Zilberfeld also highlights token costs, model drift, prompt injection, jailbreaks, safety, bias, and the need to make any unacceptable failure deterministic through logic outside the model or protective guards. He concludes that testers’ work is increasingly about quality architecture: understanding risks, defining “good enough,” and designing systems that can safely use non-deterministic AI outputs, especially when agents act on responses without a human reader’s judgment.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.