April 2025 Summaries

Google's Agent2Agent (A2A) protocol aims to help AI agents collaborate by providing a shared language. However, the current transport mechanism used by A2A is point-to-point and tightly coupled, which becomes a problem at enterprise scale. To build a real enterprise agent ecosystem, A2A needs an architecture that supports loose coupling, multiple consumers, durable communication, and real-time flow of events across systems. Apache Kafka's event-driven architecture can provide this foundation, enabling A2A messages to be published and subscribed to in a scalable and decoupled manner. By combining A2A's structured protocol with Kafka's capabilities, we can shift from brittle point-to-point integrations to a dynamic ecosystem where agents publish insights, subscribe to context, and coordinate in real-time.

This blog post showcases how to explore and validate data in Apache Kafka using Tableflow, a feature in Confluent Cloud that automatically synchronizes the contents of a Kafka topic to an Iceberg table. The author builds a pipeline to ingest and analyze data from the U.K. Environment Agency's network of sensors, using Tableflow to expose Kafka topics as Apache Iceberg tables and then querying them using standard SQL tools like Trino and PopSQL. The post demonstrates how to use these tools to explore and visualize the data, including unnesting arrays and joining data from multiple sources. It also highlights the benefits of using Tableflow, which makes it easier to get answers out of the data and provides a more efficient way to build streaming data pipelines.

Tejal Bhatt, a Solution Engineer at Confluent, has leveraged her experience in tech delivery roles to advance her skills and career by transitioning into pre-sales. She now helps customers solve bigger problems and innovative use cases with data streaming, developing both technical and soft skills. With a unique approach, she addresses each customer's distinct challenges, from technology capabilities to return on investment, and has developed strong relationships with customers across various personas. Confluent's culture of collaboration and opportunities for growth have been instrumental in her success, as she continues to learn and excel in her role.

Agent Taskflow is an AI orchestration platform that uses the Confluent data streaming platform to enable multi-agent systems. The platform provides a drag-and-drop builder, real-time messaging backbone, and native memory graph to make it accessible and usable by anyone. It aims to revolutionize software development with agent-native solutions, enabling teams of AI agents to coordinate, think, reason, and act in concert like human teams. By leveraging Confluent's capabilities, Agent Taskflow unlocks the power of cloud-native Apache Kafka, connectors, Stream Governance, and more, providing real-time observability, fault-tolerant communication, and scalability. The platform is designed to make useful, affordable, and fun AI agents accessible to everyone, with a vision to become the default operating system for multi-agent orchestration.

In today's rapidly growing digital landscape, organizations are facing unprecedented challenges in managing their data efficiently. With the world expected to store about 175ZB of data by 2025, it is essential for organizations to adopt strategies that ensure data efficiency and mitigate the challenges associated with rapid data growth. Confluent and Microsoft Azure are working together to provide AI-powered innovation solutions that enable organizations to achieve data efficiency through the use of schemas, open table formats like Apache Iceberg, and no-copy solutions. These solutions help minimize data redundancy, reduce costs, and improve data quality and consistency. By adopting these modern data practices, organizations can optimize their data processes, reduce complexity, and maintain performance and efficiency in the face of exponential data expansion.

The public sector is experiencing a shift towards leveraging real-time data and modernizing applications, which ultimately leads to improved reliability of services, operational efficiency, and delivering critical insights in real-time. This shift is also enabling organizations to unlock new opportunities with artificial intelligence (AI) through the combination of data streaming, timely stream processing, and data governance. The concept of "shifting left" is being applied to power innovative solutions, including generative AI applications, and public sector leaders are asking how they can do the same. Shifting left allows organizations to address common data quality issues, reduce duplication, and cut processing costs, which are critical for building safe, secure, and trustworthy AI applications. The shift left approach also involves adopting a "what if" mindset to set the end goal for data use and working backwards to get there. Additionally, a data-first approach is being adopted, focusing on meeting the various needs across an enterprise, improving data literacy, and developing risk frameworks alongside AI development. Generative AI has been a key step in unlocking the future of AI, enabling models that can be used for multiple purposes, and agentic AI is expected to enable AI-driven agents that can reason and act. The Department of Defense's CJADC2 concept aims to connect data-centric information from all branches of service, partners, and allies into an internet of military things, making information accessible anywhere and anytime. Confluent partners are utilizing data streaming to fuel data-centric approaches to operational challenges, enabling decision superiority in the DoD, providing a "clean room" for cross-agency collaboration, predictive maintenance, and entity resolution.

The text discusses the intricacies of Apache Kafka's offset management, which determines how data is consumed, reprocessed, or skipped across topics and partitions. The consumer uses offsets to track its progress when reading through a partition, but manual offset control can be challenging due to distributed environments. KIP-1094 introduces a new constructor method with `nextOffsets` in the `ConsumerRecords` class, exposing the next offset and leader epoch, which improves precision and reliability for offset commits. This enhancement addresses challenges associated with manual offset control, ensuring accurate data processing and maintaining the integrity of data streams in distributed environments.

In Apache Flink, watermarks are used to determine when a time-based aggregation operation can be completed. Watermarks define the maximum amount of time a job should wait for out-of-order messages, and they are typically calculated by taking the maximum timestamp observed and subtracting the allowed out-of-orderness. If the watermark strategy is not set up correctly or if the out-of-orderness threshold is too large, jobs may wait forever or produce delayed results, potentially leading to data loss. Additionally, idle partitions can cause issues with watermark calculation, as Flink will wait for an inactive partition to become active before emitting results, which can result in significant latency and data loss. To address these issues, the Data Streaming Platform in Confluent Cloud provides a default watermark strategy that uses a histogram of observed timestamps to determine an appropriate watermark, dropping less than 5% of messages due to out-of-orderness. However, users may need to override this strategy if their use case requires more control over when to drop messages or operates outside the minimum and maximum out-of-orderness thresholds.

In the context of inter-organizational data sharing, organizations can benefit from increased operational efficiency, new revenue opportunities, and improved customer experiences by exchanging data across organizational boundaries. To achieve this, companies must address critical technical concerns such as security and scalability. Confluent's data streaming platform enables secure, scalable solutions that transform cross-boundary data exchange, allowing organizations to maintain control over shared data and its associated infrastructure while ensuring recipients have reliable access to well-governed streams of real-time data. The company offers multiple ways of sharing data across organizational boundaries, including Stream Sharing, Cluster Linking, and Apache Kafka or REST API client applications, each suited to different requirements. These solutions enable organizations to maximize the value of their data assets, build deeper commercial partnerships, and better meet regulatory reporting requirements.

Confluent Tableflow integrates with Trino to enable seamless integration of streaming and batch data using Apache Iceberg tables. This allows users to query and visualize Apache Iceberg tables effortlessly in Jupyter Notebooks, making it easier to derive insights from their Confluent Cloud data. With Tableflow, organizations can persist and structure streaming data into Iceberg tables stored in cloud object storage, ensuring efficient, scalable, and cost-effective analytics. The integration supports various query engines and data catalogs, including Trino, Apache Spark, DuckDB, and Amazon SageMaker Lakehouse. By enabling users to work with real-time data and aggregate insights like total net sales per user, Tableflow provides a powerful solution for businesses struggling to bridge the gap between real-time event streaming and analytical workloads.

A multi-agent system is a team of specialized agents working together to solve complex problems. In the context of meal planning, this means combining multiple agents to create a cohesive plan that caters to various preferences and dietary needs. The system uses event-driven architecture, with Apache Kafka as the backbone for communication between agents, and tools like Apache Flink and LangChain for processing and generating outputs. Each agent operates independently but collaborates through events to achieve a common goal. This approach simplifies complex tasks by breaking them down into manageable components and enables scalability, flexibility, and fault isolation. The system can be extended or modified as needed, making it suitable for various domains beyond meal planning.

A shared language for AI agents is needed to solve the problem of AI silos. AI agents perceive, reason, and act based on data, but they currently operate in isolation, creating duplicated work and lost insights. To connect these agents, a real-time event-driven communication layer is required, such as a data streaming platform like Apache Kafka or Apache Flink. Agents can self-register, publish events, and dynamically subscribe to relevant event streams without requiring explicit integrations or vendor lock-in. This enables a truly open, real-time AI ecosystem where intelligence flows freely across platforms, avoiding vendor lock-in and scaling efficiently. The solution involves an agent registry that provides a standardized framework for registering and managing AI agents, stream processing to intelligently route insights, and an embedded LLM-based event mapper to dynamically map outputs to agents.

The text discusses the importance of shifting left in data management, where tasks traditionally performed later are moved earlier to improve efficiency and reduce risk. This approach addresses data quality, structure, and governance at the point of creation, minimizing downstream errors and improving trust in data. To achieve this, organizations need to address people, processes, and technology, including establishing processes that make sharing data the standard, using data contracts, and leveraging a data platform team to provide the necessary tools and infrastructure. The text highlights the benefits of shifting left, including improved quality, reliability, and usability across teams, as well as unlocking new opportunities for AI, machine learning, and data-driven decision-making.