Designing LLM Provider Load Balancing for Agent Workflows

Lovable employs large language models (LLMs) extensively for tasks such as code writing and context assembly, processing around 1.8 billion tokens per minute. To manage the challenges of provider outages and rate-limiting, Lovable utilizes a sophisticated load-balancing system that integrates multiple fallback chains and project-level affinity to ensure reliable service and maintain prompt caching. This system distributes traffic across various LLM providers based on observed behavior and provider preferences, automatically adjusting provider weights using a PID controller to accommodate fluctuating capacities and minimize disruptions. By keeping consecutive requests for the same project on the same provider, Lovable enhances efficiency and reduces costs. Additionally, the system addresses streaming failures by employing Claude models to continue generating from where a previous response stopped, thereby avoiding duplicated or inconsistent outputs. This robust approach allows Lovable to maintain service continuity and explore capacity scenarios through a dedicated Lovable app, offering insights into load balancer dynamics and performance.