July 2025 Summaries

Organizations are increasingly investing in large language models (LLMs) to enhance customer experiences, automate workflows, and innovate, with 72% of companies planning to increase spending in the coming year. However, this rapid adoption introduces new security risks specific to LLMs, such as prompt manipulation, data poisoning, and unmonitored resource consumption. The OWASP Top 10 for LLM Applications 2025, developed by the Open Worldwide Application Security Project, provides a framework for understanding these threats. Kong AI Gateway offers a comprehensive solution for mitigating these vulnerabilities by integrating specialized AI security controls into existing infrastructures, ensuring consistent security policies across various LLM providers. The platform supports flexible deployment strategies, offers runtime observability, and integrates with third-party monitoring tools to enhance visibility and reliability. Key vulnerabilities addressed by Kong include prompt injection, sensitive information disclosure, supply chain vulnerabilities, data poisoning, improper output handling, system prompt leakage, vector and embedding weaknesses, misinformation, unbounded consumption, and excessive autonomy. The gateway's security measures include access control, prompt management, semantic understanding, caching capabilities, and resource management, making it suitable for diverse industries with specific compliance needs.

In the final part of a series on evolving AI Agents using Kong AI Gateway, the focus is on integrating multiple Large Language Models (LLMs) and implementing Semantic Routing policies to enhance AI capabilities. Utilizing Kong AI Gateway 3.11, which supports various GenAI infrastructures, the configuration involves adding new LLMs like Mistral and Anthropic alongside OpenAI, allowing selective communication based on factors such as cost and semantics. Semantic Routing is facilitated by the AI Proxy Advanced plugin, which uses a vector database, Redis, and an embedding model from Ollama to route requests to appropriate LLMs based on topic relevance. The AI Agent architecture includes a Python script with new functions and pre-built LangGraph functionalities for efficient tool integration, while Grafana Dashboards monitor request metrics. The deployment of a LangGraph Server using Minikube is outlined, leveraging Docker images and requiring a LangSmith API Key for monitoring. Overall, the setup enhances AI Agents' ability to manage diverse GenAI tasks, including text, video, and audio processing, by abstracting LLM infrastructures and applying protective policies through the Kong AI Gateway.

In part two of a series on implementing AI Agents with Kong AI Gateway, the discussion focuses on LangGraph fundamentals and the potential of integrating frameworks to enhance AI Agent complexity, security, and integration with external systems. The post explores LangGraph, a low-level, controllable framework for AI Agents, and compares it to LangChain, which provides a standard interface for interacting with LLMs. A basic LangGraph application is demonstrated, highlighting how it structures agent workflows using nodes and edges within a graph. The text explains the process of setting up an observability layer with tools like Loki, Prometheus, and Grafana to monitor AI Gateway activities. Additionally, the concept of Tools and Function Calling is introduced, allowing AI Agents to invoke external functions and APIs, exemplified through OpenAI's built-in tools. The guide further explains how to enhance AI Agents with reasoning loops using LangGraph, while integrating external functions protected by Kong AI Gateway. The post concludes by noting the use of Kong's plugins for better security and management of API Keys, and introduces the next step in the series, which will cover Semantic Routing across multiple LLM infrastructures.

Konnect Debugger, formerly known as Active Tracing, is now generally available as an advanced debugging and observability tool within Kong Konnect, providing extensive visibility into API traffic through detailed tracing and comprehensive logging. Building on the capabilities of Active Tracing, Konnect Debugger offers features such as OpenTelemetry-compatible traces, intuitive trace visualization, and a seamless interface that requires no additional instrumentation. It empowers infrastructure administrators to initiate targeted debugging sessions, capturing detailed telemetry data and logs to enhance service performance and reliability by quickly identifying and resolving issues. The tool's interface offers summary, trace, and logs views, allowing users to visualize API request-response flows, understand performance bottlenecks, and troubleshoot errors effectively, thus optimizing API strategies with confidence.

The 3.5 release of the Kong Ingress Controller (KIC) introduces significant updates, notably the general availability of combined services, support for connection draining, and the deprecation of some Ingress types as part of transitioning customers to the Kubernetes Gateway API. Combined HTTP Routes, now generally available, allow the consolidation of multiple HTTPRoute resources targeting the same backend service, thus reducing the complexity and resource usage by decreasing the number of Kong Gateway services required. This feature, which is not enabled by default due to changes in service naming, can be activated with specific flags. The release also brings support for connection draining, which ensures that connections to terminating pods are handled smoothly, particularly beneficial for session-based workloads using sticky sessions. By integrating with Kubernetes lifecycle phases and enhancing graceful shutdown processes, connection draining helps manage in-flight transactions efficiently. Users can enable this feature through specific configuration settings to improve the management of session continuity during pod termination.

Beta support for Mesh Manager in the Konnect Terraform Provider introduces infrastructure-as-code capabilities to Kong's Service Mesh management platform by allowing engineering teams to manage Konnect Mesh resources using HashiCorp Terraform. Terraform, a tool developed by HashiCorp, enables users to define and manage infrastructure through a declarative language, facilitating consistent and repeatable deployments across various service providers. Providers in Terraform serve as plugins that interface with APIs, allowing management of diverse infrastructure components. The new Konnect Mesh Manager Provider allows users to codify mesh infrastructure, enhancing version control, auditing, and collaboration. Additionally, there is a separate beta provider for managing the on-prem version of Kong Mesh. Feedback is encouraged to further improve the provider's functionality and stability.

The text delves into the first part of a series examining the integration of Kong AI Gateway and LangGraph for developing AI agents. It explains the concept of AI agents, highlighting their reliance on large language models (LLMs) for autonomous operation and decision-making. The series aims to demonstrate building AI agents using Kong AI Gateway and LangGraph, with a focus on leveraging multiple LLMs to improve response accuracy. Key topics include the implementation architecture of AI agents, the use of Kong AI Gateway to manage and protect LLM models, and the application of reasoning frameworks like ReAct for AI development. It also discusses the capabilities of Kong AI Gateway for supporting AI-related tasks such as request handling and prompt engineering, while emphasizing the need to keep AI agents separate from the policies governing model consumption. The text introduces the concept of the Model Context Protocol (MCP) and outlines the blog series structure, which will progressively cover basic to advanced AI agent development using these tools.

Retrieval-Augmented Generation (RAG) is a technique that enhances AI model performance by injecting up-to-date and domain-specific data from external sources into prompts before they reach a Large Language Model (LLM). This method helps address limitations such as hallucination and lack of transparency in LLMs by dynamically fetching relevant information without needing continuous fine-tuning. RAG consists of two main processes: the Ingest Pipeline, where documents are converted into vectors and stored in a vector database, and the Retrieve Pipeline, which fetches relevant data in response to user queries using techniques like Cosine Similarity. Kong's AI Gateway offers tools like the AI Prompt Compressor to optimize and compress prompts, reducing latency and cost, while the AI Prompt Decorator ensures that LLMs rely solely on vetted internal sources. Kong is also developing features to enhance the control and relevance of RAG-based responses, such as chunk relevance scoring and policy enforcement mechanisms, illustrating its commitment to advancing AI capabilities for various teams.

Kong AI Gateway's latest release, version 3.11, introduces a suite of new features designed to enhance the development and deployment of AI agents, emphasizing the integration of generative AI capabilities with improved efficiency and security. Key updates include multimodal capabilities, such as real-time translation and image generation, as well as tools for orchestrating complex AI workflows and optimizing language model (LLM) costs through prompt compression. The release also highlights the integration of AWS Bedrock Guardrails to mitigate risks associated with AI outputs and the introduction of an AI Manager to monitor, secure, and visualize AI traffic. These features aim to streamline AI infrastructure, offering developers the tools to build more reliable and efficient AI systems while maintaining control over application costs and ensuring adherence to safety standards.

Kong Gateway Enterprise 3.11 introduces several new features aimed at enhancing API and event stream capabilities, including advanced data orchestration via the new Datakit plugin, support for CyberArk Secrets Manager, and sticky-sessions as a new load balancer algorithm. The Event Gateway now integrates with Solace PubSub+ for event streaming and introduces Kafka schema validation to ensure data quality. The update also expands Konnect's networking capabilities with VPC Peering and Route 53 Private DNS support for Dedicated Cloud Gateways, offering simpler and cost-effective private connections. Active Tracing in Konnect is set to receive improvements, allowing users to more effectively diagnose and resolve performance issues by aligning logs with request timelines. These enhancements collectively aim to provide more powerful, secure, and efficient management of APIs and event-driven architectures.

In February 2024, Kong introduced the first dedicated AI gateway, providing a robust platform to enhance performance, observability, and policy enforcement for Generative AI (GenAI) workloads. The Kong AI Gateway offers a universal API to centrally secure and govern traffic to large language models (LLMs), AI agents, and MCP servers, helping organizations manage soaring AI costs with techniques like token rate limiting and response caching. Kong's infrastructure is built on its established API platform, ensuring performance and scalability for demanding AI projects. Benchmark tests conducted on AWS, comparing Kong AI Gateway with Portkey and LiteLLM, revealed that Kong significantly outperformed its competitors in terms of requests per second and latency. The tests showed Kong Konnect Data Planes being over 228% faster than Portkey and 859% faster than LiteLLM, with considerably lower latency. This comprehensive platform allows organizations to unify API and AI management, offering control, visibility, and scalability across diverse deployment environments, whether they are monolithic, microservices-based, or multi-cloud.

APIs serve as crucial connectors in the digital ecosystem, enabling communication between modern applications, microservices, and IoT devices by translating and transferring data across various software platforms. Ensuring the security of these APIs is essential, as it involves implementing technologies, practices, and protocols to protect them from unauthorized access, data breaches, and malicious activities, effectively acting as a digital security guard that verifies and controls access to software systems.

Kong has launched the general availability of its revamped Developer Portal and Service Catalog, which together form a unified API discovery layer within the Kong Konnect platform, aimed at addressing both consumer and producer sides of API discovery. The Developer Portal provides a centralized source for API consumers to find, test, and register APIs, enhancing their ability to build applications efficiently. Meanwhile, the Service Catalog serves API producers by cataloging internal services and providing detailed context for security, reliability, and governance, integrating with tools like GitHub and Datadog for comprehensive oversight. This holistic approach seeks to combat the challenges of API sprawl and improve organizational API management, promoting better security, ROI, and developer productivity.

Organizations face significant challenges due to API sprawl, where a lack of centralized oversight leads to shadow APIs that bypass security protocols and compliance standards, posing risks such as data breaches and inefficiencies. Kong's Konnect Service Catalog offers a solution by automating the discovery and management of APIs and services, creating a real-time, centralized repository that enhances security, compliance, and operational efficiency. The Service Catalog integrates with existing platforms like Kong Gateway, Kong Mesh, and Traceable to identify unmanaged APIs, allowing for better governance and collaboration across teams. By maintaining a comprehensive inventory of services enriched with contextual information, the platform facilitates accelerated development, improved teamwork, and consistent standards, ultimately reducing security risks and enhancing scalability. Additionally, the use of Scorecards within the Service Catalog automates compliance checks, ensuring services adhere to organizational standards and enabling swift remediation of any non-compliant issues.

Create-api.dev is a tool developed by Kong to streamline and enhance the OpenAPI specification design process, initially conceived as an internal experiment to leverage AI in API design. It offers a user-friendly platform to generate, refine, and share OpenAPI specs without the need for login or setup, catering to API designers who find traditional tools cumbersome or outdated. Users can create specs from scratch by describing an API, upload existing specs for modification, or use design documents to generate specs. The system employs Google's Gemini model for effective handling of OpenAPI specs and incorporates Kong's own design guidelines. Besides generating specs, create-api.dev allows users to share specs instantly, import them into Kong's Konnect for further policy application, and publish them for developer access. The platform is a live demonstration of Kong's AI Gateway, which enhances AI app production by managing model requests with efficiency, offering protections like prompt injection prevention, and enabling routing and A/B testing across models. Feedback from users is actively encouraged to refine the tool's capabilities.

Kong has announced the general availability of its new Konnect Dev Portal, designed to support API adoption and reuse in modern enterprises. Initially launched as a public beta, this platform aims to simplify API discovery and operationalize API reuse by addressing common challenges such as API bloat and sprawl. The portal offers a customizable interface with features like Snippets for efficient content management and supports advanced customization with CSS. API developers can leverage tools for creating OpenAPI specs with AI to enhance their API design and documentation processes. The platform emphasizes a seamless Developer Experience (DevEx) for both API providers and consumers, with streamlined publishing flows and robust search functionalities to boost API adoption. Additionally, Konnect Dev Portal incorporates analytics tools to monitor API usage metrics, helping organizations identify high-impact APIs and refine their API strategies. The portal thus aims to enhance developer productivity, reduce costs, and improve organizational agility by fostering an efficient API program focused on reuse.

Kong has launched its new Developer Portal as part of its API discovery initiative, aiming to redefine what a developer portal should be by balancing simplicity and deep customization. Built on modern frameworks such as Nuxt, Vue.js, and TypeScript, it offers features like server-side rendering, smart caching, and reusable component architecture to enhance performance and developer experience. The portal introduces Markdown Components for easily creating interactive content and Snippets for reusable custom components, promoting scalability and maintainability. Unlike traditional portals, it combines beautiful UI with integrated functionalities like self-service workflows, enterprise-grade access controls, and real-time customization through a built-in portal editor. The portal also features a custom OpenAPI renderer for improved API documentation, developer registration, smart search capabilities, and robust analytics, ensuring a comprehensive, user-friendly experience tailored to both developers and enterprise needs.

Kong Service Catalog has been introduced as part of Kong's API discovery initiative to address the gap in producer-side API discovery, transforming Kong Konnect into a comprehensive API discovery platform. Unlike traditional API management solutions that focus on consumer-side discovery, Kong Service Catalog provides a holistic view of an organization's service ecosystem, including shadow APIs, by integrating with third-party tools and enforcing governance through scorecards. This approach allows for the documentation of all services, offering real-time compliance visibility and reducing the risk of shadow APIs, unmanaged services that can lead to security issues. The Service Catalog features an object model based on integrations, resources, and services, reflecting the actual operation of services in the real world. By leveraging the existing API platform, it reduces the need for standalone internal developer platforms and avoids tool sprawl, enabling teams to manage service discovery and governance more efficiently within the Kong Konnect interface.