NVIDIA NeMo Evaluator is a microservice with an easy-to-use API that simplifies the end-to-end evaluation of generative AI applications, including retrieval-augmented generation (RAG) and agentic AI. It supports evaluation for a wide range of custom tasks and domains, including reasoning, coding, retrieval, and instruction-following, and allows developers to automatically evaluate their models against academic benchmarks or custom datasets, or score them with standard metrics such as accuracy, ROUGE, BLEU, or LLM-as-a-judge scoring. Datadog LLM Observability can be integrated with NeMo Evaluator to provide end-to-end visibility into the health and performance of LLM applications, tracing requests across RAG components and model inference and evaluation steps, collecting and visualizing key model metrics and metadata, and linking model quality metrics directly to corresponding LLM request traces for unified analysis.