LLMs in Prod 2025: Insights from 2 Trillion+ Tokens
Blog post from Portkey
In 2024, AI transitioned from experimental to essential systems, presenting organizations with unexpected challenges in scaling their infrastructures. Portkey's AI Gateway, working with over 650 organizations and processing more than 2 trillion tokens globally, provided insights into these challenges through its "LLMs in Production" report. Key findings include a significant shift towards multi-provider strategies, driven by the need for redundancy and better performance, with adoption rising from 23% to 40% in just one year. Reliability emerged as a critical concern, with some providers experiencing failure rates exceeding 20% during peak times, while enterprises moved from primarily basic LLM usage to more complex implementations, indicated by an increase in 100-500 token requests from 10% to 37%. As AI systems grew in complexity, the report emphasized the importance of building reliable, efficient, and secure AI infrastructures for the future, presenting detailed reliability benchmarks, architectural patterns, and cost optimization frameworks for enterprises aiming to enhance their AI capabilities into 2025.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.