Evaluating generative AI performance: When your data is “anything”

New Relic AI aims to unlock observability for all by enabling users to interact with their telemetry data using natural language, reducing reliance on New Relic Query Language (NRQL) proficiency and streamlining analysis across tech stacks. To evaluate the performance of this system, the development team identified areas where clear-cut measures of success can be applied, such as classifying user requests into different flows and performing syntax validation in NL2NRQL. They also tackled hallucinations by using retrieval-augmented generation (RAG) and reframing the task to ensure the model relies on external context rather than built-in knowledge. Additionally, they promote trustworthiness via transparency and clear communication throughout the Q&A flow, providing users with intermediate steps and explanations to assess the validity of answers. By implementing these strategies, New Relic AI can deliver real value to users while minimizing potential pitfalls.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.