Announcing Kong AI Gateway 3.8 With Semantic Caching and Security, 6 New LLM Load-Balancing Algorithms, and More LLMs

Kong's latest release of AI Gateway 3.8 introduces significant advancements designed to enhance GenAI application performance, security, and scalability. This update includes the debut of intelligent semantic plugins, advanced load balancing capabilities, and expanded support for LLM providers such as AWS Bedrock and GCP Vertex. A key feature is the AI Semantic Cache, which improves processing speeds by caching semantically similar prompts, reducing latency in GenAI applications. Additionally, the AI Semantic Prompt Guard offers enhanced security by blocking prompts based on their semantic content rather than specific keywords. The release also introduces six new load-balancing algorithms tailored for LLMs, including a unique semantic routing capability that selects the most suitable model for each prompt in real-time. These features, alongside support for various LLM providers and easy integration with existing GenAI applications, position Kong AI Gateway as a comprehensive solution for organizations looking to optimize their AI and machine learning workflows.