May 2026 Summaries

Redpanda SQL, now generally available, is a Postgres-based query engine that operates within Redpanda BYOC clusters, enabling users to query live-streaming topics and historical Iceberg tables in a single SQL statement without requiring an ETL pipeline or connector fleet. This innovation addresses longstanding challenges in the data engineering industry, where traditional methods demanded moving streaming data into warehouses before querying, resulting in delays and cumbersome processes. By integrating SQL capabilities directly into the existing infrastructure, Redpanda SQL eliminates these inefficiencies, allowing real-time operational analytics and ad hoc querying directly on streaming data. It supports Postgres-compatible tools and bridges the gap between live and historical data, offering a seamless querying experience without the need for additional infrastructure changes or new software installations. This development marks a significant shift from previous systems, facilitating quicker incident resolution and real-time data access for applications like fraud detection and personalization, while maintaining compliance by keeping data within the user's VPC. Currently available for Redpanda BYOC customers on AWS, with plans to expand support, this tool empowers users to leverage SQL for immediate data insights without the traditional lag posed by data pipelines.

Redpanda's approach to garbage collection in its Cloud Topics involves carefully managing Level Zero (L0) objects, which are temporary data batches, and transitioning them to Level One (L1) objects for historical reads. The goal is to efficiently determine when L0 objects are no longer needed and can be safely deleted without manual intervention. Redpanda utilizes a system of epochs, which act as logical timestamps embedded in L0 object IDs, to track the progression and safe deletion of these objects. Through per-partition epoch tracking embedded in the Raft log, Redpanda achieves a distributed system where epochs are updated monotonically, ensuring that once an epoch is deemed safe for garbage collection, it remains so permanently. The system allows for flexibility in handling partition leadership changes and reduces the risk of rejecting new writes due to stale epoch caches. This design ensures effective garbage collection while maintaining system safety and efficiency, with a promise of continued improvements in future updates.

Gartner's report "Predicts 2026: AI Agents, MCP and Governance are Transforming Analytics" explores how organizations are strategizing for agentic AI by 2026, emphasizing that while AI offers new opportunities, it also introduces governance challenges. Predictions include a significant reliance on general-purpose LLMs for analytics, potential failures in agentic analytics projects lacking a semantic layer, and risks of financial or reputational damage from ungoverned AI decisions. Peter Corless from Redpanda provides additional insights, suggesting that successful AI implementation requires shared leadership responsibilities, skill development among data analysts, and robust governance frameworks. He identifies emerging standards like ISO 42001 and AIUC-1 as crucial for ensuring AI safety and reliability. Corless also emphasizes the importance of a unified framework, like Redpanda's Agentic Data Plane, to address governance and operational challenges in deploying agentic systems.

A new shared skills and hooks repository has been launched for UX and cloud team members working on frontend features with Claude Code, streamlining AI-assisted workflows with built-in guardrails and automatic updates. This system enhances the development process by automatically checking for issues such as type safety, linting, formatting, testing, builds, and common anti-patterns, ensuring these are addressed before reaching a pull request. A standout feature is the lifecycle skill, which guides Claude Code through a structured development flow, from asking clarifying questions to generating shareable reports that integrate with platforms like GitHub and Jira. The repository also introduces SandCastle, a community project that allows each Claude Code instance to operate within its own Docker container, enabling independent iterations based on Jira tickets or GitHub issues. This setup, verified by a script to ensure proper installation, supports both Claude Code and Codex, offering a collaborative environment for efficient and error-free development. More information and community feedback opportunities are available on GitHub and the Redpanda Slack Community.

Redpanda, a modern data streaming platform, has evolved to handle shifting technological bottlenecks, moving from CPU constraints to high-latency I/O challenges with the introduction of cloud-based object storage in its Cloud Topics feature. Initially designed with a thread-per-core architecture using Seastar in C++, Redpanda effectively managed low-latency storage but faced performance hurdles as demand for cost-effective cloud storage reintroduced latency issues. The bottleneck arose when the new storage mode required separating metadata and application payload data, causing a significant slowdown in the upload phase to cloud storage. By introducing an additional layer of queuing during the upload phase, Redpanda increased concurrency and throughput, effectively mitigating latency and maintaining data integrity and order. This improvement was validated through performance testing, allowing Redpanda to achieve the desired throughput at scale without altering producer configurations. The development of Cloud Topics reflects Redpanda's adaptability in addressing architectural challenges and enhancing its streaming capabilities, with the feature now available in Redpanda 26.1 for enterprise trials.