How to build a remote evaluator in Arize AX with Jev

The tutorial explains how to connect TypeSafe’s Jev to Arize AX through a hosted remote evaluator that scores agent trace spans after responses are generated, focusing on whether an agent’s reply resolves a user’s request. Jev answers bounded questions with typed outputs rather than explanations; in this example, its Noul question type returns a probability that a yes/no criterion is true, which the service maps to a “yes” or “no” label using a configurable threshold. The integration uses OpenInference-traced agent inputs and outputs, a FastAPI translation service that forwards span data to Jev’s System One API, and an Arize AX evaluator configured with an endpoint, JSON schema, span mappings, and optional continuous or historical evaluation tasks. Tests on a customer-support agent found that answerable questions generally received scores above 0.8, while requests requiring unavailable account-changing tools scored roughly 0.06 to 0.11, with little variation across repeated runs. The authors note that probability scores indicate confidence rather than degree of resolution, require validation against human labels for accuracy and calibration, and may be useful for scalable, low-cost evaluations, while more complex assessments still benefit from LLM judges that can provide reasoning, explanations, or additional context.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.