October 2025 Summaries

AI agents, while becoming increasingly popular, face challenges in connecting to external data sources due to their inherent isolation. The Model Context Protocol (MCP) aims to address these issues by providing an open standard for secure connections between AI systems and external data, facilitating better and more relevant responses. Redpanda has introduced Redpanda Cloud Remote MCP, a managed solution that simplifies the process of building MCP servers using low-code YAML configuration, allowing seamless integration of AI applications with over 300 data connectors. This solution leverages Redpanda Connect's high-performance connectors to enable quick and reliable data integration, transforming AI system development into a more efficient process. Redpanda Cloud Remote MCP is currently available in beta, offering developers a streamlined approach to connect AI systems with the necessary data, thereby focusing on innovation rather than infrastructure management.

Redpanda has introduced the Agentic Data Plane (ADP), a unified system designed to govern AI agents' access to enterprise data, emphasizing safety and governance. ADP provides a low-latency streaming layer for events, a distributed query engine for real-time context, and over 300 connectors for comprehensive model context, alongside global governance tools that ensure access control and observability. It aims to enable AI agents to interact with data systems in a controlled and auditable manner, addressing enterprise concerns over data safety and compliance. Redpanda has also acquired Oxla to enhance its capabilities in merging real-time with historical data using a PostgreSQL wire protocol engine. This development reflects a broader shift towards AI-driven data management, emphasizing the need for centralized governance and context management to safely integrate AI agents into enterprise infrastructures.

Redpanda's new Iceberg Topics feature simplifies the integration of streaming data with analytics by allowing users to manage data flows into Iceberg tables effortlessly. This setup involves creating a local Iceberg Development Environment within Kubernetes, combining Redpanda, Apache Polaris, MinIO, Postgres, and DuckDB to facilitate a complete end-to-end data processing flow without relying on external cloud services. Users can configure Redpanda topics to write data into Iceberg with minimal effort, and the project includes three demos showcasing how to handle Avro messages, utilize the latest schema mode, and store raw bytes in Iceberg tables. The environment is designed to be easy to set up, using a single script to deploy necessary components, making it a convenient launchpad for developing custom data pipelines.

Redpanda and Snowflake combined to create a high-performance streaming data pipeline that can process 3.8 billion messages at a rate of 14.5 GB per second, achieving near real-time analytics with a P50 latency of under two seconds and a P99 latency under eight seconds. The setup was executed swiftly, transitioning from concept to production within a day, largely thanks to Redpanda's automation tools. The benchmark utilized a 9-node Redpanda Enterprise cluster on AWS EC2 instances and 12 Redpanda Connect nodes, employing the Kafka-compatible Redpanda platform and a snowflake_streaming connector optimized for high throughput and low latency. Key optimizations included using a binary format like AVRO for a 20% throughput improvement and adjusting for a delicate balance of throughput and latency. Although Snowflake's build steps introduced some latency, increasing build_parallelism and optimizing Snowpipe Streaming channels helped mitigate this. The tests demonstrated that Redpanda and Snowflake can effectively support real-time analytics in various applications, such as market surveillance and fraud detection, with insights delivered in seconds rather than hours.