May 2025 Summaries

Modern companies generate large volumes of data, but often internal users find it challenging to quickly figure out answers to their questions due to the lack of specialized knowledge. Apache Kafka is a powerful tool for real-time data processing, and many organizations use Kafka to enable self-service access to data streams. However, getting specific insights from these streams often requires specialized knowledge. Recently, I explored leveraging Cursor, an AI coding assistant, and Model Control Protocol (MCP), an open standard for integrating large language model applications and data sources, to interact with a Kafka topic hosted on Confluent Cloud. This combination made it easy to query the relevant Kafka topics with natural language, making information more accessible to a variety of users. By combining AI assistants like Cursor with integration frameworks like MCP, we can build intuitive interfaces over powerful backend systems, translating natural language requests into executable queries like Flink SQL for Kafka, unlocking data accessibility and dramatically reducing the time from request to insight.

Confluent, a company that operates with a one-team culture, fosters seamless collaboration and better outcomes for its customers. The company's account executives, like Jason, help customers transform their technical roadblocks into real-world wins. Confluent supports the ambitions of its employees by providing strong leadership and opportunities for growth. Employees find the company culture to be incredibly collaborative and unique, with a focus on achieving individual, team, and company goals as one unit. The best part of interacting with customers is working with smart engineers to solve complex problems and see customers launch exciting new products and functionality that makes a tangible difference in their lives.

The General Availability (GA) of the Confluent fully managed V2 connector for Apache Kafka on Azure Cosmos DB has been announced, marking a significant milestone in simplifying real-time data streaming from and to Azure Cosmos DB using Apache Kafka. The new connector, available directly from the Confluent Cloud connector catalog, is production-ready and offers scalability, performance, and flexibility across both source and sink connectors. Key features of the V2 connector include improved scalability bottlenecks, improved performance through a pull-based change feed model, flexible write patterns, better integration with Kafka infrastructure, high-throughput change feed reads, advanced sink strategies, built-in throughput control, integrated metrics and observability, improved security and authentication, and more reliable metadata handling. The connectors can be set up via UI, CLI, or API, and are designed to simplify the integration of Azure Cosmos DB with Kafka-powered event streaming architectures.

The next generation of Confluent Control Center is now available, delivering higher partition limits, faster spin-up time, metrics freshness, and simpler operational overhead. The new version features a massive step toward delivering a next-generation user experience with faster performance and unparalleled scale. It offers unmatched performance, limitless scalability, and enhanced operational simplicity, empowering operators, developers, and architects to oversee Kafka deployments with greater efficiency, confidence, and ease. With the release of Confluent Control Center 2.0, users can scale without limits, accelerate start-up time, achieve end-to-end metrics freshness, reduce operational overhead, and simplify their architecture. The new version has been rearchitected to integrate with Prometheus alongside Confluent Telemetry Reporter, facilitating ingestion into the Prometheus open telemetry protocol (OTLP) endpoint. This allows for efficient handling of large numbers of topics and consumer groups, ensuring seamless performance in complex Kafka workloads.

This quarter’s release of Confluent Cloud brings new features that fundamentally change the dynamic between stream and batch processing. The introduction of snapshot queries in Confluent Cloud for Apache Flink enables fast, batch-style queries across Kafka topics as well as Apache Iceberg and Delta Lake tables via Tableflow. This allows users to query historical and real-time data using a unified stream and batch processing interface. Additionally, the feature supports dual materialization, allowing a single topic to populate both Delta Lake and Iceberg tables simultaneously. Confluent Cloud also introduces support for enterprise clusters on Google Cloud, cross-cloud cluster linking, WarpStream diagnostics, and schema linking. These features aim to simplify secure connectivity, improve data governance, and enhance operational efficiency in the cloud environment.

In the age of AI, real-time data is becoming increasingly important for businesses, with data streaming platforms playing a crucial role in simplifying access and reuse of this data. The 2025 Data Streaming Report highlights how these platforms are emerging as a business imperative, helping organizations solve persistent data challenges such as data silos and governance-related disjoints. A complete data streaming platform provides the necessary capabilities to turn data into a reusable asset, ensuring it is inherently usable across multiple teams, discoverable, secure, and trustworthy. As AI systems become more capable, the demand for continuous, real-time data access will only accelerate, making data streaming platforms critical infrastructure. IT leaders see data streaming platforms as key to achieving their data-related goals, with 89% citing higher investments in DSPs this year. To unlock the true value of their data, organizations must prioritize governance early and adopt a shift-left approach to data processing and governance.

Confluent has launched the next generation of its Control Center for Confluent Platform, enhancing the management of Apache Kafka with increased partition limits, faster startup times, and improved metrics freshness. Originally introduced in 2016, the Control Center now supports up to 400,000 partitions, significantly more than its previous limit, and reduces startup time to just one minute. The integration with Prometheus optimizes metrics handling, eliminating the need for a separate Kafka cluster for metrics storage, which streamlines operations and reduces costs. Fully backward compatible with versions 7.5 to 7.9, this release is part of Confluent's efforts to refine Kafka management with seamless scalability and user experience improvements. Looking ahead, the Confluent Platform 8.0, expected in 2025, will be based on Apache Kafka 4.0, with plans to phase out support for certain legacy clients. The company is also working on a UI for Apache Flink and enhancing management across cloud and on-premises environments.

The VISTA Framework is a structured approach to prioritizing AI opportunities by balancing four dimensions: Business Value, Implementation Speed, Scalability, and Tolerance for Risk. This framework helps businesses focus on AI projects that are both practical and high-impact, ensuring they deliver results without unnecessary complexity. By using the VISTA Framework, companies can identify inefficiencies in their operations, automate tasks, and improve customer experience with generative AI. The framework provides a clear way to evaluate and prioritize AI investments, allowing businesses to start small, experiment, and scale as value becomes clear. This approach helps build momentum and create lasting value by focusing on projects that deliver quick wins with minimal risk, using those successes to build expertise and confidence before expanding into more ambitious efforts.

An asynchronous work culture is gaining traction, but many corporate cultures are still blocked by synchronous workstreams. This can manifest in different ways, from closed and synchronous to open and asynchronous. An asynchronous work culture prioritizes the distribution of knowledge, creating a flywheel of institutional knowledge that benefits future employees. It allows decisions and progress to be made more quickly and saves current employees from doing extra work to acquire information. Companies like Confluent have developed commandments for async work, such as sharing work in public places by default, providing active feedback, treating everything as a shared work product, and encouraging asynchronous behavior in others. While it may feel uncomfortable at first, the switching pain is worth it, as it leads to dramatic productivity gains and enables smart companies to build high-performance, hybrid workstreams.

This article discusses the importance of event-driven architecture (EDA) in enabling autonomous agents to scale and thrive in modern enterprise systems. EDA is seen as a solution to the challenges of traditional control logic, allowing for dynamic, context-driven workflows that can adapt to unpredictable problems. The key benefits of EDA include loose coupling, real-time communication, horizontal scalability, low latency, and event persistence. By enabling agents to access and share data effortlessly, EDA enables them to make decisions and take action in real time. The article also highlights the importance of decoupling workflows and ensuring that agents can operate independently while integrating seamlessly into broader ecosystems. Ultimately, EDA is seen as a foundation for building agentic systems that are flexible, resilient, and scalable, providing businesses with a competitive edge in this new wave of AI innovation.

The text discusses the importance of decomposing complex problems into smaller, specialized units in multi-agent systems, where multiple agents collaborate to solve larger problems. The orchestrator pattern is introduced as a way to coordinate agent interactions, with a central node (the orchestrator) deciding which agent should handle a given task based on context. Event-driven architectures are highlighted as a better model for enterprise applications due to their ability to enable dynamic communication between agents without rigid dependencies. A reference architecture using Apache Flink and Apache Kafka is presented, showcasing how these technologies can be used to build an event-driven orchestrator for multi-agent systems. The system demonstrates real-time decision-making, scalability, and adaptability, with the use of a large language model (LLM) to determine which agent should handle an incoming message.