Switching LLMs mid-conversation is a ski-rental problem

Edgee’s Self-Learning Routing for coding agents addresses both selecting an appropriately capable language model for each request and deciding when changing models within a long session is economically worthwhile, since switching discards a model’s warm context cache and requires rebuilding it. It classifies turns into difficulty tiers, immediately upgrades for sufficiently confident higher-capability needs, and uses a ski-rental-inspired policy for cost-driven switches to cheaper models, accumulating forgone savings until they justify the one-time cache rewrite cost. The system accounts for changing conversation lengths, cache expiration, compaction, cooldown periods, and confidence thresholds, while treating its theoretical competitive guarantee as limited to stable conditions. With organizational consent, it labels turns without retaining prompt text, then weekly replays up to 30 days of labeled traffic across candidate model maps and switching settings, comparing estimated costs with a hindsight optimum while enforcing tier-based capability floors. Recommendations are subject to coverage, uncertainty, quality, usage, failure-rate, and administrative guardrails, and can be applied manually or automatically under constrained conditions. The approach differs from conventional query-level LLM routing by explicitly modeling within-session cache costs, though its replay assumes alternative models would generate outputs of the same length and does not yet measure response quality directly.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.