July 2025 Summaries

Universities face significant governance challenges when providing AI access to students and faculty, including managing credentials, ensuring security, and controlling costs. The need for a centralized solution is emphasized to address these issues, enabling safe and scalable access to AI models and tools like Cursor and Claude Code. This involves using a central gateway for authentication, usage tracking, and enforcing safety guardrails, while integrating with existing identity systems for dynamic access management based on enrollment and role. Portkey is highlighted as a solution, offering a centralized AI gateway that routes all traffic through a single control point, providing built-in safeguards and customizable protections. It allows institutions to manage AI access and usage efficiently, maintaining oversight and control across campus, while supporting student interaction with AI through modern tools. As the official Internet2 NET+ AI Gateway, Portkey is trusted by universities to deploy GenAI responsibly at scale, without requiring custom infrastructure.

Claude.ai introduces a 5-hour rolling session model for its users, where the session begins with the first prompt, and new weekly quotas for heavy users on the Claude Pro and Max plans will be implemented from August 28th. API usage is billed separately on a pay-as-you-go basis per token and offers consistent rates across Anthropic, AWS Bedrock, and Google Vertex AI. Portkey users can efficiently route Claude Code through different platforms while centralizing cost controls and enhancing security. The platform offers various subscription plans with differing capacities and pricing, allowing users to choose based on their needs, while API token-based pricing provides flexibility. Claude Code is positioned for both individual and enterprise use, with Portkey enabling seamless integration across different cloud services, offering governance, observability, and resilience. Anthropic plans to update quotas and expand Bedrock/Vertex regions, urging users to stay informed on future changes.

LibreChat is a favored open-source tool among developers and AI teams for creating custom chat interfaces on large language models, offering flexibility and support for multiple providers like OpenAI and Anthropic. It is ideal for initial experiments due to its ease of setup and extension without requiring a hosted service. However, as usage scales within enterprises, it presents challenges such as lack of usage tracking, cost accountability, and governance over model selection, which can lead to risks and inefficiencies. Portkey addresses these issues by integrating with LibreChat to provide control, visibility, and accountability, acting as a centralized AI gateway that logs interactions, manages costs, enforces policies, and enhances reliability with minimal setup changes. This integration transforms LibreChat into a governable and scalable enterprise tool, allowing organizations to confidently deploy it across teams while maintaining oversight and compliance.

Claude Code is popular among developers for its ability to generate and explain high-quality code, making it suitable for individual productivity but challenging for enterprise-wide deployment due to a lack of built-in logging, cost tracking, and access controls. As teams scale its use, issues arise around usage tracking, cost management, policy enforcement, and access restrictions. Portkey addresses these limitations by offering a unified AI gateway that enhances Claude Code's operational capabilities, allowing enterprises to manage costs, enforce security and governance policies, and maintain visibility and control over interactions. By routing Claude Code through Portkey, enterprises can confidently deploy AI-powered workflows, ensuring reliability, security, and scalability across their operations.

In AI applications managing numerous requests through large language models (LLMs) or agents, failures such as timeouts and latency spikes are common, often due to the reliance on external APIs. Initially, reactive strategies like retries and fallbacks are implemented to manage these failures, but they are limited to short-term issues and can inadvertently exacerbate problems when failures are persistent. Circuit breakers offer a proactive solution by monitoring failure patterns and stopping traffic to failing components, thereby preventing system overloads and maintaining performance. These strategies are not interchangeable but rather serve different roles; retries recover from minor glitches, fallbacks provide alternative pathways, and circuit breakers halt further degradation. Portkey’s AI Gateway facilitates the implementation of these strategies by allowing users to configure retry policies, fallback chains, and circuit breakers without extensive custom coding, ensuring system resilience across various providers and models.

OpenWebUI is an open-source interface designed for interacting with large language models (LLMs) and is favored for its lightweight and flexible nature, suitable for various model providers. As its use expands within organizations, the need for robust enterprise governance becomes crucial, focusing on observability, cost management, security, and compliance. To address these requirements, Portkey serves as a centralized AI gateway that enhances OpenWebUI deployments with advanced features such as cost tracking, access control, usage analytics, and reliability measures. Portkey enables platform and IT teams to monitor spending, enforce access rules, and maintain compliance effortlessly, ensuring a secure and scalable AI infrastructure without altering the user experience. It facilitates efficient deployment by including capabilities like caching, load balancing, and audit logging, making OpenWebUI enterprise-ready and suitable for shared environments by providing the necessary foundation for managing LLM usage across organizations.

Portkey has introduced unified routes for file uploads and batching inference requests, allowing users to upload a single file for asynchronous batching and fine-tuning across different AI model providers without converting to model-specific formats. Despite the complexity of handling multipart/formdata and application/octet-stream transformations, as well as navigating the differing requirements of platforms like OpenAI, AWS Bedrock, and Cohere, Portkey simplifies user interaction with a consistent API signature. The gateway, built to operate efficiently from edge locations, executes streaming transformations across various content types and automates processes like multipart file uploads and batch processing. The open-source nature of Portkey, along with its enterprise version offering immediate batching and error handling, aims to position it as the fastest AI gateway available.

The rapid adoption of generative AI tools has led to the emergence of "shadow AI," where teams use AI models without formal approval, raising significant risks for enterprises, including compliance violations, data leaks, and budget overruns. Shadow AI often bypasses traditional security, procurement, and compliance processes, making it difficult for IT and legal teams to govern its usage. Key risks involve data leaks, fragmented spending, reputational damage from unvetted outputs, and security blind spots. To mitigate these risks, organizations can implement an AI gateway, such as Portkey, which centralizes control and visibility over AI usage by routing all AI calls through a single interface. This approach ensures vetted model access, eliminates personal API keys, enforces guardrails, and allows for accurate cost allocation. Best practices to prevent shadow AI include clear AI usage policies, regular monitoring and auditing, team education on risks, and promoting responsible AI usage through shared tools and prompts. Implementing an AI gateway turns shadow AI into governed AI, fostering innovation while maintaining compliance, security, and cost management.

Shadow AI, a growing concern within enterprises, involves the use of AI models, APIs, or applications without the oversight of central IT or security teams, leading to significant risks such as security vulnerabilities, compliance violations, and financial costs. This phenomenon is driven by the rapid adoption of low-friction APIs and GenAI tools, which allow teams to bypass formal processes in their haste to leverage AI, often resulting in fragmented AI landscapes due to a lack of governance. Shadow AI can introduce reliability and debugging challenges, as applications may lack observability, making it difficult to trace and resolve issues. Additionally, the absence of centralized control can result in brand and reputational risks if unauthorized AI applications generate incorrect or offensive content. Detecting and mitigating shadow AI requires organizations to implement visibility and control mechanisms, such as routing all AI traffic through a centralized gateway, enforcing access controls, and establishing usage quotas to prevent experiments from escalating into costly risks.

As teams increasingly incorporate large language models (LLMs) into their operations, managing their use and scalability has given rise to two key infrastructure patterns: the LLM proxy and the AI gateway. An LLM proxy acts as a lightweight middleware that simplifies request routing and offers basic features like caching and logging, making it ideal for early-stage projects that prioritize flexibility over extensive governance. Conversely, an AI gateway provides a robust, production-grade infrastructure that offers centralized control, security, and observability, making it essential for organizations with complex needs involving multiple LLMs and teams. AI gateways support advanced capabilities such as access control, guardrails, audit logging, and multi-provider routing, treating LLMs as critical infrastructure requiring rigorous oversight. While LLM proxies are suitable for prototyping, AI gateways are indispensable as usage scales, offering governance and control to ensure compliance and cost efficiency in production environments.

The rapid adoption of Generative AI (GenAI) across enterprises is transforming it from an experimental tool into a foundational capability, with various teams using AI models for tasks ranging from prototyping to content creation. This surge in demand is challenging traditional centralized provisioning methods, which struggle with delays and inefficiencies, leading to insecure practices and inconsistent governance. To address these issues, a shift towards a governed self-service model is recommended, where employees can access AI models through a secure, policy-driven framework that maintains security and observability. Tools like Portkey are designed to facilitate this transition by providing a unified control plane for managing AI model access across different teams and providers, ensuring a balance between innovation, security, and compliance.

In the landscape of AI infrastructure, effective prompt management is crucial as teams scale and their prompt libraries grow, but traditional methods, which often involve scattered documents and lack version control, lead to inefficiencies and operational chaos. The text discusses the challenges faced by teams using spreadsheets and manual processes for prompt management, which result in slow iterations, inconsistent production behavior, and a lack of visibility into prompt performance. Portkey is introduced as a solution, offering structured versioning, centralized management, model-aware testing, and easy rollbacks, empowering teams to manage, test, and deploy prompts with the same rigor as coding processes. By treating prompts as first-class citizens in the AI stack, Portkey enables teams to reduce deployment times by over 90%, conduct safe experimentation, and maintain control over large prompt libraries, thus facilitating rapid, reliable, and scalable AI deployments.