Instrumenting an LLM router: what New Relic sees, Prometheus doesn't

Routing LLM prompts across small, medium, and large models can reduce cost and latency, but it creates a production risk if teams cannot see whether routing choices remain accurate, responses retain quality, costs rise unexpectedly, or sensitive prompt data enters telemetry. The implementation described uses a FastAPI gateway that deterministically scores prompts and routes them through LiteLLM to three local Ollama models, while LiteLLM’s New Relic callback enables AI Monitoring and distributed tracing even though Ollama is not natively listed as a supported provider. Because standard monitoring cannot explain why a tier was selected or assess answer quality, the gateway emits custom routing-decision and response-evaluation events that feed dashboards, alerts, change tracking, and governance visibility. When repeated quality or latency flags trigger an alert, a webhook starts a New Relic Workflow Automation process that combines an Autopilot investigation with a fallback query, posts findings to Slack, waits for human approval, and only then activates a circuit breaker that routes requests to the fastest tier. The project also highlights operational constraints of local inference, including long CPU-only cold starts and slow complex responses, the importance of warming models and setting realistic latency budgets, risks from tracing full prompt content, dependence on a single model provider, and deployment issues such as reverse-proxy timeouts that local testing may miss.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.