July 2026 Summaries

Redpanda SQL, a Postgres-compatible analytical query engine, is designed to efficiently handle both live-streaming topics and historical data using bridge queries. Through a benchmark involving NYC Taxi trip records scaled from 100 GB to 1 TB, the study demonstrates that query latency is influenced more by the data scanned rather than the total data retained, with filtering queries taking approximately 4 to 5 seconds regardless of data size. For complex queries requiring full-database scans, performance improves significantly with additional hardware, exemplified by a heavy query's runtime dropping from 298 seconds to 43 seconds when executed on eight nodes. The study suggests that compute cluster sizing should focus on the demands of the heaviest analytical query rather than total data volume, emphasizing that Redpanda SQL's ability to prune data using Iceberg statistics ensures efficient query processing. The findings underscore the importance of distinguishing between scaling for single-query performance and scaling for concurrent query throughput, with further insights to be explored in subsequent discussions.

Redpanda Connect has introduced a visual composer tool designed to simplify the creation of complex data pipelines, enhancing user accessibility through a drag-and-drop interface while maintaining a robust backend capable of handling high-throughput streaming data. This new tool allows users to visually construct pipelines with conditional routing, error handling, and data enrichment, providing a more intuitive alternative to writing complex YAML configurations. The visual composer is complemented by Bloblang V2, a redesigned data-mapping language that offers improved customization, error handling, and editor support. Together, these innovations aim to streamline the process for both new and experienced engineers, reducing the time and complexity involved in deploying data pipelines. Additionally, Redpanda Connect continues to expand its connector offerings, such as those for SAP HANA and BigQuery, optimizing data ingestion and real-time processing capabilities for enterprise systems. The overarching goal is to modernize data workflows by efficiently integrating existing systems with real-time analytics platforms.

Organizations are facing challenges in deploying enterprise agentic systems due to a lack of trust in autonomous agents, primarily because existing governance frameworks, designed for humans, are inadequate for managing these agents. The Redpanda Agentic Data Plane offers a solution by introducing a central governance layer specifically built for agents, which facilitates trust, explainability, context, and control. This governance infrastructure allows organizations to safely scale agents by providing verified identities, auditable actions, scoped access, and the ability to deactivate agents if necessary. Redpanda's approach is characterized by its neutrality, open architecture, and built-in governance, avoiding vendor lock-in and ensuring secure operations across any model, API, or database. The Tech Talk by Redpanda CTO Tyler Akidau elaborates on these issues and solutions, emphasizing the importance of an infrastructure that enforces governance without agent interference.

Change Data Capture (CDC) is a process that allows systems to react to real-time data changes in a database, offering an efficient alternative to traditional polling methods. This tutorial demonstrates how to establish a CDC pipeline using Redpanda Connect and MySQL, enabling the streaming of row-level changes as events without polling. The tutorial details the setup process, including prerequisites like Docker and a Redpanda Connect enterprise license, and provides a step-by-step guide to creating and configuring the necessary services, such as Redpanda, Redpanda Connect, and MySQL with binary logging enabled. The pipeline captures and processes changes in the database, streams them as events to Redpanda topics, and allows users to consume these events, showcasing how changes like inserts, updates, and deletions in the MySQL database are captured and reflected in real time. The tutorial also highlights Redpanda's compatibility with Kafka APIs and encourages exploration of CDC streams with other databases supported by Redpanda Connect, emphasizing the benefits of CDC for real-time analytics, seamless integrations, and system migrations.

Redpanda Connect's v4.99.0 update significantly simplifies and enhances the process of maintaining data freshness in lakehouses by introducing row-level operations for Iceberg tables, such as insert, upsert, and delete, which allows change data capture (CDC) streams to be applied directly to Iceberg tables. This update eliminates the need for a complex setup involving multiple services and configurations, replacing it with a single, streamlined pipeline definition. This development ensures that lakehouses can now mirror the real-time status of operational databases, improving data accuracy and reducing the manual effort required to keep tables synchronized. The update transforms the Iceberg output from an append-only model to one that dynamically reflects changes in the source database, akin to directly updating values in a spreadsheet rather than manually tracking changes through a log.

The Agentic Data Plane is a crucial governance and runtime layer designed to manage AI agents' interactions with data, tools, identities, and models, filling a gap in current enterprise infrastructure that existing identity management and API gateways cannot address. As AI agents increasingly perform actions autonomously across multiple systems, a unique infrastructure is needed to provide identity, policy enforcement, and comprehensive activity trails for these agents, something not currently offered by traditional systems. Unlike traditional API gateways that manage services, the Agentic Data Plane introduces governance for agents by managing identity, tool calls, and model usage, promoting open standards to ensure interoperability across diverse environments and avoiding vendor lock-in. The Redpanda Agentic Data Plane distinguishes itself by enabling deployment within an organization's own cloud infrastructure, maintaining data sovereignty, and supporting comprehensive agent identity and action tracking through a streaming-native data integration layer. This approach ensures that agents operate under explicit governance, with a focus on compliance, security, and real-time data access, thereby closing the infrastructure gap for deploying autonomous agents in enterprise environments.

Redpanda Connect introduces a native Oracle CDC input that captures data changes from Oracle databases using LogMiner, eliminating the need for Debezium, Kafka Connect runtime, or a JVM. This update allows for the creation of an efficient two-stage data pipeline where changes are streamed from Oracle to a durable Redpanda topic and then into ClickHouse, using simple YAML configurations. The process is designed to facilitate the movement of data from a central Oracle operational system to a high-performance ClickHouse analytics environment without overburdening the source database. By streaming changes through Redpanda, users can maintain data durability, enable replay capabilities, and expand the data pipeline without additional load on Oracle. This setup uses Docker for containerization and can be initiated with a free 30-day trial license. The pipeline is designed to be scalable, allowing for the inclusion of multiple tables and the potential to fan out data to various sinks, such as a data lake, without re-engaging the Oracle database.