April 2026 Summaries

AgentOps serves as an operational layer for AI agents in production, offering visibility and control over the autonomous systems that utilize large language models (LLMs) for planning and decision-making. Unlike traditional request-scoped infrastructures, which handle a single request-response cycle, AgentOps addresses the complexities of multi-step, dynamic agent behavior by monitoring every LLM call and tool invocation, tracking token usage, latency, and expenditure, and enforcing guardrails and policy controls. This operational layer is crucial for managing reasoning errors, silent tool call failures, and cumulative cost and latency issues that arise from the complex interactions of agents. Portkey's AI Gateway exemplifies a shared control infrastructure that centralizes agent registration, model access, and compliance checks, enabling consistent governance across diverse agent frameworks and configurations. By enforcing controls at the infrastructure level rather than per-agent configurations, AgentOps ensures that agent behavior remains visible, bounded, and compliant with organizational policies, thus preventing issues like runaway loops and policy violations.

GitHub Copilot is an AI-powered code generation tool integrated into various development environments, and while it simplifies coding for individual developers, its management becomes complex when adopted at scale across teams. As organizations increasingly rely on it, issues such as limited provider flexibility, fragmented visibility of usage data, credential sprawl, and security gaps arise, necessitating a shift towards platform-level governance. To address these challenges, best practices include implementing a control layer between Copilot and model providers, centralized logging for visibility, credential hierarchies for security, selective model access, budget and rate limits to manage costs, and guardrails to ensure compliance with internal policies. These practices aim to maintain the convenience of GitHub Copilot while ensuring reliability, control, and security as usage expands, and tools like Portkey’s AI gateway are suggested to facilitate this transition by providing monitoring and management capabilities.

Coding agents have become an essential part of the developer toolkit, with tools like Claude Code, Cursor, and Codex becoming increasingly integrated into workflows. While individual developers can quickly become productive with these tools, the quality of output can vary significantly across teams due to differences in context rather than the models themselves. Some developers excel by creating structured workflows and integrating coding agents with internal systems, resulting in better outcomes. However, this knowledge is often localized and does not spread within organizations, leading to inconsistent results and challenges as teams scale. Platform teams face the task of consolidating disparate configurations and establishing a unified infrastructure. Uber's engineering team tackled this by developing an internal CLI to unify coding agent setups and facilitate skill sharing, which many organizations can emulate with tools like Portkey. Portkey provides a centralized registry for skills and MCP (Model-Context-Provider) configurations, allowing for consistent, governed, and versioned distribution across teams. By synchronizing skills and configurations, organizations can bridge the gap between varying developer outputs, standardizing and enhancing the use of coding agents at scale, akin to how Kubernetes manages Docker deployments.

Portkey's Skills Registry addresses the issue of disparate coding agent skills by centralizing and synchronizing them across teams using a structured workflow. It allows teams to create, review, and publish markdown files called Agent Skills, which outline essential work practices like code review standards and security checklists. These skills are authored and versioned within Portkey's Prompt Engineering Studio, enabling teams to maintain consistency and transparency. The draft/publish model ensures that only approved skills are synced to agents such as Claude Code, Cursor, and Codex, while version history allows easy rollback if needed. By using Portkey's CLI, teams can efficiently distribute and update these skills across all agents, ensuring that every team member, regardless of tenure, operates with the same contextual knowledge and standards.

In scaling AI agents within enterprises, initial successes often give way to challenges as the number of agents grows, leading to issues with governance, security, and operational maturity. As teams rapidly deploy agents across various functions without a shared registry or clear ownership, they often lose track of what agents are running and their specific permissions, leading to potential security vulnerabilities. These vulnerabilities include excessive agency, where agents are granted more access than necessary, and the "lethal trifecta" of accessing private data, exposure to untrusted content, and the ability to communicate externally, which could be exploited. Additionally, chains of agents can create complex problems when errors propagate through interconnected tasks. The core issue is not negligence but rather the pace of development outstripping governance capabilities, with traditional IAM and RBAC tools unable to keep up with dynamic agent operations. The Agent Gateway platform addresses these challenges by providing a centralized system for agent registration, governance, and monitoring, ensuring that organizations can scale their AI agents securely and efficiently.

n8n is a flexible, node-based workflow automation platform that allows the integration of JavaScript or Python into workflows, making it attractive for AI automation in production environments. However, when AI Agent nodes call large language models (LLMs) across teams, operational challenges arise, such as token visibility, credential management, and provider reliability, which are not addressed by the workflow design itself. These issues highlight the need for an AI gateway as a control layer between n8n and model providers, enabling centralized management of provider credentials, routing logic, and access control. Portkey serves as this layer, offering features like centralized credential management, budget and rate limits, observability, cost attribution, provider fallback, load balancing, and guardrails to ensure a consistent and governed usage of AI models. By connecting n8n to Portkey, organizations can maintain existing workflow structures while gaining enhanced operational control, making AI workflow usage predictable and manageable across teams and environments.

Portkey has introduced the Agent Gateway, a comprehensive infrastructure solution designed to manage and govern AI agents in production, addressing the evolving needs of enterprises building autonomous AI applications. This gateway serves as a centralized production layer that offers governance, access control, observability, reliability, and security features, ensuring that all agent interactions are monitored and controlled effectively without altering the existing agent or client code. The Agent Gateway facilitates seamless integration by providing a single governed endpoint for each agent, enabling enterprises to manage budget and usage limits, apply policy changes instantly, and maintain full traceability of agent activities. It also includes PII detection, PHI filtering, content moderation, and a unified authentication model to enhance security and compliance. The service catalog within the Agent Gateway supports agent visibility, ownership, and access management, offering a native RBAC and authentication framework. Currently available in beta, the Agent Gateway aims to become the foundational production layer for AI agents, similar to Portkey's AI Gateway for LLM calls, with further insights available through an upcoming technical walkthrough.

Scaling OpenAI Codex from individual developers to enterprise teams exposes significant operational challenges, such as lack of cost visibility, credential sprawl, and unreliable access control. While a single developer may not encounter these issues, they become pronounced in organizational deployments, where shared credits and API key management are problematic. To address these, centralized operational controls are essential. Portkey's AI gateway offers solutions like centralized management of API keys, multi-provider connectivity, detailed cost tracking, input-output guardrails, and role-based access control, ensuring predictable and secure Codex usage across teams. This setup allows for seamless integration and control over provider access, enabling teams to maintain operational efficiency while preserving flexibility in their AI workflows.

In the rapidly evolving field of AI agents, traditional safeguards, designed for single large language model (LLM) requests, are proving inadequate for managing the complexities of agents that operate through multi-step execution chains. These agents autonomously chain model calls, invoke tools, and trigger external effects, which can lead to unauthorized access, runaway costs, and untraceable outputs. Effective governance for AI agents requires a comprehensive approach that includes execution control to manage decision-making, tool and action permissions to restrict unauthorized tool use, cost and resource governance to prevent overspending, and policy enforcement layers to ensure compliance at every action point. A centralized infrastructure layer, such as Portkey's AI Gateway, offers a robust solution by embedding governance controls directly into the request path, ensuring consistent application across agents without altering their code, and providing a unified view of agent activity.

Semantic caching optimizes Large Language Model (LLM) applications by reusing responses for queries with similar meanings, rather than identical phrasing, thus significantly reducing costs and improving latency compared to basic caching. This technique involves transforming queries into numerical vectors, checking them against a vector store using a similarity metric, and determining cache hits based on a configured similarity threshold. AWS reported that semantic caching could cut costs by up to 86% and improve response times by 88%, but the effectiveness hinges on setting an appropriate similarity threshold. While a more permissive threshold allows greater cache reuse with minimal impact on accuracy, domain-specific applications, such as medical or legal systems, may require stricter thresholds due to the nuances in language. Semantic caching can also introduce challenges such as stale data and issues with multi-turn conversations, which can be addressed by incorporating context-aware embeddings or query rewriting. Monitoring is crucial to prevent silent degradation of cache performance, requiring specific metrics like cache hit ratio and latency differential tracking. Various tools and platforms, from open-source libraries to managed services, are available for implementing semantic caching, each offering different levels of control and operational ease.

In March 2026, Anthropic began throttling Claude Code sessions during peak hours, causing Max plan users to quickly exhaust their session limits, a move confirmed to manage increasing demand. Conductor, a tool in the Claude Code ecosystem, revolutionizes workflow with features like git worktree isolation, diff review, and parallel session execution, built by Charlie and Jackson who are daily users. It operates above the API layer with orchestration and session handling, while Portkey manages the underlying aspects such as costs, logs, and budget limits, and both tools are designed to complement each other. Users must configure ANTHROPIC_* environment variables correctly, with specific emphasis on setting the ANTHROPIC_API_KEY to an empty string to ensure proper routing through Portkey. The setup emphasizes observability with comprehensive logging, budget controls to prevent overspending, cross-provider failovers for resilience, and support for over 1600 models through a unified interface, with caching to optimize costs and latency. Users are encouraged to consult the full setup guide and seek support through the Portkey Discord or by reporting issues on the gateway repository.

Enterprises are increasingly integrating large language models (LLMs) into critical systems, making the need for advanced AIOps platforms essential to handle unique operational challenges. Unlike traditional infrastructure, where failures are binary, LLM systems can have invisible errors, such as incorrect outputs or missed guardrails, necessitating specialized observability and control mechanisms. Portkey's AIOps platform offers solutions tailored for LLM and AI agent workloads, featuring capabilities like unified observability, routing resilience, real-time guardrails, prompt management, and governance with cost controls. As these AI systems become more autonomous, centralized control and visibility become vital, enabling enterprises to manage production-scale operations effectively while maintaining compliance and cost efficiency. Adopting such platforms involves starting with visibility into current workflows and gradually implementing routing, safety controls, and governance policies to align with operational history and system behavior.

A recent supply chain attack on an open-source LLM gateway exposed vulnerabilities in the AI ecosystem by silently harvesting credentials such as cloud credentials and SSH keys from affected environments. This incident highlighted the risks associated with LLM gateways, which serve as critical routing layers between applications and LLM providers, making them attractive targets for attackers. The breach was exacerbated by the AI ecosystem's habitual lack of operational maturity, exemplified by inadequate dependency management and credential storage practices. The attack exploited a security scanning tool in the gateway's CI/CD pipeline, emphasizing the importance of pinning dependencies and managing secrets properly. Teams that emerged unscathed had previously implemented robust architectural decisions regarding credential storage and dependency management. The incident serves as a reminder that AI infrastructure requires the same level of security and operational diligence as traditional critical infrastructure, urging a shift toward more mature practices in managing dependencies, credential storage, and CI/CD configurations to mitigate potential attack vectors.

AIOps enhances the operational management of LLM systems by focusing on how requests are executed rather than merely monitoring system health, addressing the inadequacies of traditional MLOps approaches. While standard monitoring tools indicate if a system is operational, they fail to capture nuanced issues like output quality, cost surges, or specific model decision impacts. AIOps introduces an operational layer that enables end-to-end request traceability, allowing for unified tracing of every request through its execution path, and facilitating quicker debugging and comprehensive governance. It provides a framework for centralized routing policy enforcement, access control, and real-time usage monitoring, which helps prevent unexpected costs and aligns model usage with organizational constraints. Through platforms like Portkey's AI Gateway, AIOps offers teams improved visibility and control, resulting in more predictable costs, faster issue resolution, and consistent model behavior by governing routing and policies comprehensively across services.

Portkey has open-sourced its model pricing database, encompassing over 3,500 models from more than 50 providers, to address the industry's challenge of attributing costs to AI model usage accurately. This initiative stems from the lack of a standardized method for calculating costs, a problem compounded by the complexity of modern AI pricing structures and the absence of a canonical source or comprehensive API. Portkey's AI gateway, designed to normalize provider responses and facilitate real-time budget enforcement, offers a solution by resolving model identifiers, normalizing usage, and tagging costs per team and user. The gateway architecture, which separates provider responses, pricing data, and logic, allows for adaptability to changes in provider billing dimensions. Maintaining accuracy across the vast array of models is a continual challenge, managed by an agent utilizing the Claude Agent SDK to update pricing information efficiently. This endeavor is crucial for enterprises seeking to adopt AI at scale without financial ambiguity, enabling them to track usage accurately and optimize AI budgets effectively.

Harnesses, which are essential for even the best coding models, add significant overhead that affects the efficiency of coding agents during tasks. A study comparing different agents like Pi, OpenAI Codex, and Claude Code reveals substantial disparities in harness overhead, with Claude Code sending approximately 27,000 input tokens per request compared to Pi's 2,600. This overhead, termed the "Harness Tax," involves system prompts, tool definitions, and conversation history, which consume valuable context window space and can impede the model's ability to focus on the actual task. As models evolve and improve, the assumptions encoded within harnesses quickly become outdated, suggesting the need for leaner harnesses that leverage the model's inherent capabilities. While complex work may justify the deep tooling of agents like Claude Code, the study highlights the importance of measuring and understanding harness overhead to optimize performance and resource utilization.

Cursor, an AI-first code editor built on a VS Code fork, has seen widespread adoption within enterprise engineering organizations due to its ability to integrate large language models directly into the development environment for tasks like autocomplete and multi-file edits. However, scaling Cursor's use presents challenges such as scattered API keys, invisible token spend, lack of budget controls, and compliance gaps. To address these issues, the introduction of an AI gateway like Portkey is recommended, which acts as a control plane between Cursor and model providers to enforce policies, log usage, and manage routing and budgets centrally. This setup allows for centralized credential management, budget limits, model access governance, and data security through input and output guardrails, making usage predictable and manageable across organizations while maintaining the same developer workflow.

MCP servers facilitate tool discovery and invocation for agents, but challenges arise when these servers are shared across multiple teams and environments, leading to issues in access control, credential management, and auditability. The core problem lies in unrestricted tool access, where agents can inadvertently access tools beyond their scope, resulting in over-permissioning and compliance risks. Effective governance requires a two-level tool provisioning model, with organization-level provisions controlling overall access and workspace-level provisions managing access within teams. This model ensures that only relevant tools are accessible to specific workspaces, thus maintaining security and compliance. Portkey's MCP gateway offers a centralized solution by managing tool provisioning, access control, and runtime governance at a gateway layer, integrating role-based access, authentication, and audit logs to maintain a secure and scalable infrastructure. This centralized approach prevents access sprawl and enhances traceability, crucial for maintaining trust and compliance across shared production environments.

A new LLM-powered feature experiences a surge in token consumption as usage grows, leading to increased GPU queue times, slower responses, and HTTP 429 errors. The unpredictability of LLM workloads, driven by factors like probabilistic outputs and multi-step workflows, requires effective rate-limiting strategies to maintain system reliability and prevent resource saturation. Rate limiting is essential for controlling token and request throughput, managing infrastructure demands, and ensuring fair resource allocation across users and applications. Various limits, such as those based on tokens, requests, costs, and time windows, help balance compute capacity and budget constraints. Implementing rate limits at the gateway level provides centralized management and consistent enforcement across multi-provider environments, reducing policy drift and operational overhead. Metrics and dashboards are crucial for monitoring usage patterns and preventing issues before they escalate, while unified control planes offer integrated policy management for AI infrastructure, optimizing cost and governance across deployments.

Managing API keys for large language models (LLMs) becomes increasingly complex as organizations scale, with various providers such as OpenAI, Anthropic, Google, Azure, AWS Bedrock, and Mistral each issuing their own credentials. This complexity can lead to scattered credentials across apps and environments, making it difficult for security teams to maintain control and comply with audit requirements. The proposed solution is to use an AI gateway, such as Portkey, which centralizes access through a single API, reducing the need for multiple keys and allowing credentials to be managed through existing secret management infrastructure like AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault. By using secret references, applications can securely access model providers without directly storing credentials, simplifying key rotation and access control while maintaining the existing security protocols of the organization. This approach also allows for scoped access to credentials, ensuring that different teams have appropriate access based on their roles and responsibilities.

Autoinstrumentation addresses the observability gap in AI agent systems by automatically capturing telemetry data, such as prompts, responses, token usage, and latency, without requiring developers to manually add monitoring code. By operating at various layers like the SDK, agent framework, or gateway, autoinstrumentation ensures consistent and standardized data collection across services and environments, facilitating easier debugging, cost tracking, and performance analysis as systems scale. In production environments, it provides real-time visibility into system behavior and links multi-step workflows into single traces, helping to diagnose issues like latency spikes or unexpected agent decisions. Portkey's AI Gateway exemplifies autoinstrumentation by capturing telemetry for every LLM request, centralizing observability, and simplifying debugging through consistent data formats across models and providers. This approach becomes crucial as applications transition from prototypes to production, offering unified visibility and structured data for effective monitoring and troubleshooting.