LLM routing in production: Choosing the right model for every request

For companies using large language models (LLMs), the initial appeal of deploying a single high-quality model like GPT-4 for all use cases can quickly lead to unsustainable costs and performance issues as usage scales. This situation often necessitates implementing a routing system, which intelligently directs requests to different models based on factors such as task complexity, cost, latency, and compliance needs, similar to how a hospital triages patients. Routing helps optimize expenses by assigning simpler tasks to cheaper models and reserving more expensive ones for complex, high-value tasks, improving both cost-efficiency and user experience. However, premature optimization and overcomplicating the routing process can add unnecessary complexity and latency, especially if the volume of requests doesn’t justify the effort. Effective routing requires clear decision criteria, observability, and fallback mechanisms to ensure resilience. The choice of routing solution—whether building in-house or using third-party services like Martian, Portkey, or OpenRouter—depends on a team's specific needs, such as control, ease of experimentation, or comprehensive AI infrastructure management. Ultimately, routing should be employed only when it addresses significant pain points in cost or performance, as a single well-chosen model can often suffice for simpler or early-stage products.