How we rebuilt provider routing for better, faster inference

OpenRouter has replaced its predominantly price-based provider routing system with a real-time algorithm designed to balance effective cost, latency, throughput, reliability, rate-limit behavior, and response quality. Rather than selecting providers mainly by input-token price, the new system estimates the practical cost of obtaining a usable answer by accounting for actual token pricing, cache rates, time to first token, generation speed, retries, errors, and malformed tool calls, while shifting traffic away from degraded endpoints and managing capacity through traffic floors, backoff, and caps. Conversation sessions remain pinned to their initially selected provider to preserve prompt-cache benefits, and users can still override the algorithm through provider ordering, filtering, or explicit price, throughput, or latency sorting. In a week-long switchback experiment covering more than 24 million requests and 103 billion input tokens, OpenRouter reported lower median time to first token, higher throughput, fewer rate-limit errors and retries, and faster full-request completion, alongside roughly a 6% increase in cost per request.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.