From Cool Demo to Production-Ready: How We Made an AI Travel Agent Trustworthy with New Relic

WanderAI, a travel-planning startup, aims to use AI to craft personalized itineraries based on user preferences but faces challenges in proving its effectiveness beyond impressive demos. The startup uses the Microsoft Agent Framework, OpenTelemetry, and New Relic to enhance observability, traceability, and operational efficiency for its AI agents. Key measures include baseline observability integrated with OpenTelemetry, custom telemetry for business-specific insights, dashboards, alerts, and service level objectives (SLOs) for ongoing system performance monitoring. Moreover, the implementation of a three-layer quality assessment system and security guardrails ensures that the AI's outputs are reliable and secure. This comprehensive setup enables WanderAI to transition from a demo-ready to a production-ready service by ensuring that every aspect of the agent's interaction can be traced, evaluated, and improved in real-time, fostering trust and reliability in its operations.