February 2025 Summaries

Confluent Cloud is now offering Cluster Linking support for Azure private networking, which enables secure data replication between Kafka clusters in private Azure environments without requiring public internet exposure. This feature allows organizations to link Kafka clusters running in private Azure networks just as easily as they can publicly networked clusters, reducing operational complexity and enabling real-time data movement across private Kafka clusters in Azure. With Cluster Linking support for both public and private networking, organizations can facilitate disaster recovery, migrations, and secure data sharing between departments or business units, while keeping clusters private and secure within Azure. The feature is currently available in several regions, including West US 2, West US 3, East US, and Central US, with additional availability to be added in the upcoming weeks.

Generative AI has significant potential to add value to the global economy, but most organizations are hindered by their data management. Despite investments in data lakes, warehouses, and analytics tools, complexity persists due to siloed systems, duplicated data, and outdated batch processes. The "data liberation problem" arises from this complexity, making it difficult for AI projects to access the right data when needed. Data streaming offers a solution by treating data as a continuously moving asset, providing real-time data flow across on-prem, cloud, and hybrid environments. This approach eliminates batch delays, improves data quality, and enables scalable AI interactions, ultimately giving AI systems the high-quality, up-to-date data they require to perform effectively.

The text discusses the importance of data governance in event-driven microservices architecture. It highlights how Apache Kafka and Confluent Cloud can help achieve this through explicit agreements between producers and consumers, formalizing a data contract that includes schema, integrity constraints, metadata, security, and evolution. The text explains how Confluent Schema Registry provides a centralized location to catalog, organize, validate, and track the evolution of data contracts. It also introduces migration rules, which enable UPGRADE and DOWNGRADE transformations between schema versions, allowing for seamless changes in event format without breaking data consistency. The text concludes by highlighting the benefits of this approach, including improved scalability, modularity, and future-proofing, as well as enhanced autonomy for individual services within a microservices architecture.

Airy has developed a framework to build copilots as a new interface to explore and work with data, bridging large language models (LLMs) and real-time data using Confluent's Data Streaming Platform. This enables users to ask questions in natural language about their data streaming across operational and analytical systems, leveraging Apache Kafka, Apache Flink, and Apache Iceberg. The team faced challenges such as data governance and security, accuracy of GenAI outputs, and infrastructure scalability, which they solved by using Confluent's platform to power real-time inference with trustworthy outputs at scale without compromising on security, accuracy, or cost. With Confluent, Airy allows users to build agentic AI copilots that can provide engineering and business teams with accurate and timely information, driving real value for businesses.

This article discusses the power of multi-agent systems in artificial intelligence (AI), where multiple autonomous agents work together to solve complex problems. The challenges of managing these systems are explored, including context and data sharing, scalability and fault tolerance, integration complexity, timely and accurate decisions, safety and validation, and overcoming these challenges requires proven design patterns tailored for multi-agent systems. Four key patterns - orchestrator-worker, hierarchical agent, blackboard, and market-based - are introduced, along with an explanation of how to adapt them to be event-driven using data streaming technologies like Apache Kafka. The article highlights the importance of a shared language, interfaces, and immutable logs in maintaining consistency and reliability across agents, enabling seamless integration into dynamic environments. Event-driven design offers a clear path forward for tackling real-world complexity, unlocking the full potential of multi-agent systems.

Singapore's Smart Nation initiative aims to utilize effective data utilization to power exemplary citizen services, with a focus on trust, growth, and community. The government has recognized AI as the next frontier of technological advancement, particularly generative artificial intelligence (GenAI) technologies that can enhance public service impact. Real-time data streaming is crucial for GenAI technologies, but many organizations struggle to get data up to speed due to fragmented ownership of data, insufficient skills and expertise in AI/ML management, and inability to seamlessly integrate new data sources. Confluent's 2024 Data Streaming Report found that nearly seven in 10 IT leaders in Singapore believe that investments in GenAI will trend up in the next two years, but AI rollouts have been slow due to these challenges. To address these issues, technology partners like Confluent can help accelerate the adoption of data streaming technologies and GenAI, enabling real-time data processing, integration, and governance, and creating a continuous system that sets data in motion for actionable insights. Confluent's accreditation by IMDA brings data streaming to the entire Singapore government, enabling agencies to enhance decision-making and citizen engagement through a modernized data infrastructure.

The Confluent Data Streaming Platform, particularly with Apache Flink for stream processing, offers a powerful tool for on-demand predictions in real-time applications such as fraud detection, customer personalization, predictive maintenance, and customer support. Remote model inference enables developers to connect real-time data streams to external machine learning models hosted on dedicated model servers and accessed via APIs, allowing for centralized model operations, streamlined updates, version control, and monitoring. This approach is ideal for high-throughput applications where latency is a trade-off for flexibility. Confluent's Flink AI model inference integrates remote AI models into data pipelines by calling AI models within Flink queries, managing remotely hosted AI models with SQL DDL statements, and invoking remote endpoints from various cloud services. The benefits of using data streaming for model inference include centralized model management, scalability and flexibility, efficient resource allocation, seamless monitoring and optimization, and reduced latency. This approach is particularly useful in hybrid cloud setups, where models might be hosted on a cloud-based infrastructure and accessed by edge or on-premises Flink applications. By separating the model server from the streaming application, developers can leverage powerful AI capabilities while keeping Flink applications focused on efficient data processing.

Suguna Ravanappa, a people manager at Confluent, has gained valuable skills in leadership and collaboration as part of the company's unique culture that emphasizes innovation, teamwork, and continuous growth. She now seeks to enhance her strategic thinking and leadership capabilities to better support her team's growth. Suguna attributes Confluent's supportive environment to its focus on creating a safe space where diverse voices are heard, respected, and valued. The company prioritizes career development through various learning opportunities, fostering an inclusive culture that encourages collaboration and open communication.

Confluent and AWS Lambda can be used to build scalable and real-time event-driven architectures (EDAs) that respond to specific business events. Confluent provides a streaming SaaS solution based on Apache Kafka and built on Kora: The Cloud-Native Engine for Apache Kafka, allowing you to focus on building event-driven applications without operating the underlying infrastructure. AWS Lambda is a serverless compute service that provides a code execution framework that abstracts the need to provision, operate, and scale any of the underlying infrastructure. Provisioned Mode for AWS Lambda's Kafka ESM allows customers to control and optimize the throughput of their Confluent-Lambda Kafka applications to achieve an enhanced throughput profile for performance-sensitive Kafka applications. This feature offers controls to optimize throughput, responsive auto-scaling, simplified networking experience and charges, and improved developer experience. By activating Provisioned Mode, customers can improve the performance profile of their Kafka applications, reduce delays for real-time applications that see sudden spikes in traffic, and simplify network configuration by removing the requirement to configure PrivateLink. The pricing for Provisioned Mode is based on the provisioned minimum event pollers and the number of event pollers consumed during automatic scaling, measured in Event Poller Unit hours.

Confluent and Databricks have partnered to simplify the integration between analytical and operational systems, enabling enterprises to deploy AI applications more efficiently. This partnership aims to bridge the divide between real-time data and analytics, governance, and AI-driven intelligence. The first capability being offered is a Delta Lake-first integration that enables seamless bidirectional data flow between Confluent's Tableflow and Databricks' Unity Catalog, unlocking real-time, governed data products from any source to power intelligent applications. This integration addresses the challenge of siloed data in enterprises, where operational systems and analytical systems often operate independently, making it difficult to deploy AI models that require fresh, real-time operational data. By transforming operational data into AI-ready assets and bringing AI-driven intelligence into real-time applications, Confluent and Databricks are working together to make data products an enterprise-wide reality, enabling enterprises to build and deploy AI that allows proactive, automated decision-making on operational systems with production-level speed and accuracy.

Confluent has achieved FedRAMP Ready status for Confluent Cloud for Government, indicating that its security capabilities have been attested to by a Third Party Assessment Organization and deemed acceptable by the FedRAMP PMO. This certification brings the cloud's advantages, including reliability and scalability, to Confluent's federal offerings, enabling governments to treat data as a stream of events rather than static snapshots, speeding time to action and creating more personalized experiences. Confluent is committed to building a secure platform for critical data and its FedRAMP certification is another step towards making Apache Kafka available as a trusted foundation for any environment. Additionally, Confluent has expanded partnerships with Databricks and announced new cohort of startups joining the Confluent for Startups AI Accelerator program.

Confluent has announced the newest cohort of early-stage startups joining its Confluent for Startups AI Accelerator program, a 10-week virtual program designed to support innovators developing real-time generative AI applications. The program provides technical mentorship, go-to-market support, and access to Confluent's expertise in real-time data streaming. This year's cohort includes an exciting mix of AI-driven innovators from around the world and across industries, including startups working on natural language workflows, continuous health monitoring devices, bespoke AI-powered wealth management, next-generation product analytics, open source billing infrastructure, fintech innovations, AI-driven legal automation platforms, digital twin technology, and AI-powered health monitoring platforms. The program aims to empower early-stage startups to scale their solutions efficiently and ensure seamless integration of AI-powered workflows with real-time data pipelines.

An AI-powered LinkedIn post generator was developed to streamline the promotion of podcast episodes by converting audio content into text and generating posts using OpenAI's GPT and Whisper models. The application is built using Next.js, Apache Kafka, and Apache Flink, operating within an event-driven architecture that decouples the front end from AI processes. This design allows for scalability and flexibility, accommodating advancements in AI technology. The system processes podcast episodes by downloading MP3 files, transcribing them, and generating LinkedIn posts, which are stored temporarily in a cache for quick access. The use of Kafka and Flink facilitates real-time data processing, enabling the application to handle large data volumes efficiently. This project not only simplifies the task of creating LinkedIn posts but also serves as an example of a modern, scalable, and decoupled architecture that can adapt to evolving AI technologies.

Don't Get Left Behind: Unlock the Secrets of Shifting Left with FLIP-304` is a webinar that focuses on transforming how failures are handled in Apache Flink, specifically through custom failure enrichers. These enrichers enable users to tag and classify errors, integrate enriched data with monitoring tools, and quickly identify root causes of failures. This approach helps manage failures effectively by providing actionable insights, reducing debugging time, and improving overall system reliability. By leveraging FLIP-304, developers can pinpoint failure triggers, support proactive alerting, and build targeted dashboards and alerts to optimize their system's performance. The webinar provides a detailed overview of how to enable FLIP-304, write custom failure enricher implementations, and integrate enriched data with monitoring tools like Grafana, Prometheus, and Flink UI.

The Ultimate Data Streaming Guide: Concepts, Use Cases, Industry Stories` is a practical guide that explores real-world use cases of data streaming across various industries. The book offers insights into successful data streaming practices and provides over a dozen real-life customer examples, including BMW Group, KOR, Lufthansa, Migros, Schiphol Group, and Uniper. Jay Kreps, Inventor of Apache Kafka, Co-founder and CEO of Confluent, praises the book as a "fantastic resource" that captures not only technology but also stories behind adoption and value delivery. The guide is designed for both experienced users and newcomers to data streaming, providing a matrix of technical use cases and industry success stories, with links for easy navigation between chapters. The book concludes with best practices for creating internal data streaming communities and accelerating the data streaming journey. Confluent plans to release new editions featuring updated content and community success stories, inviting feedback and ideas from readers.

Agentic AI, an advanced form of generative AI, is transforming how organizations automate processes by allowing AI agents to autonomously perform tasks, interact with systems, and make decisions, potentially revolutionizing industries by 2028. These AI systems engage in multi-step processes, providing benefits such as increased efficiency and cost savings, yet they also present challenges related to trustworthiness, security, and data privacy. Companies must address these issues by anonymizing data and using technologies like data streaming platforms to ensure high-quality outputs, while also investing in new infrastructure and talent to support these AI models. Despite privacy concerns and the need for significant initial investments, agentic AI is expected to proliferate, offering greater business impacts than current generative AI tools, as seen in examples like Microsoft Copilot's evolution from simple code automation to more autonomous functions.

Confluent offers a suite of self-managed and fully managed connectors that facilitate seamless integration with Apache Kafka, enabling real-time data flow and connectivity across diverse systems and applications. For HTTP API-based use cases, Confluent Cloud provides the no-code, fully managed HTTP Source V2 and HTTP Sink V2 connectors, which can be tailored to cater to diverse data integration requirements. These connectors enable real-time analytics, real-time SaaS app data synchronization, and async microservices communication and workflow automation. The HTTP Source V2 connector allows for real-time ingestion of data from HTTP endpoints into Kafka topics, while the HTTP Sink V2 connector enables real-time delivery of data from Kafka topics to HTTP endpoints. The connectors offer features such as client-side field-level encryption, ability to configure up to 15 APIs with the same hostname and authorization mechanism, and seamless experience to configure using OpenAPI Specification 3.0. Confluent Cloud users leverage these connectors for various use cases, including real-time analytics, real-time SaaS app data synchronization, and async microservices communication and workflow automation. The connectors are designed to be more robust and user-friendly, addressing the specific needs of customers and improving their overall experience with Confluent Cloud.

Freight clusters, a new cost-effective solution from Confluent Cloud, are now generally available for AWS users, enabling companies to efficiently manage high-throughput, latency-insensitive workloads such as observability data and AI/ML data ingestion. By leveraging innovations in the Kora engine, these clusters significantly reduce costs by avoiding inter-availability zone networking fees, utilizing direct write to object storage, and employing a Fetch From Follower strategy to minimize cross-zone traffic. They are particularly suited for use cases with relaxed latency requirements and offer substantial savings compared to self-managed Kafka. The Freight clusters' serverless nature, powered by Elastic CKUs, allows for dynamic scaling based on workload needs, ensuring users only pay for the resources they require. With this new offering, Confluent aims to make real-time data streaming more accessible by removing cost barriers, allowing companies to capitalize on the full benefits of data streaming.

This recognition marks the beginning of an exciting new phase of growth as Confluent pursues a $60 billion market opportunity with its partner ecosystem. The Confluent partner ecosystem enables customers to launch new use cases and accelerate their migration journeys seamlessly through innovative collaboration between partners. These partnerships allow Confluent to consistently exceed customers' evolving expectations while unlocking the transformative power of real-time data streaming. Confluent is thrilled to present the winners across global levels, extending heartfelt congratulations to the 2025 recipients for their remarkable achievements. Interested parties can learn more about becoming a Confluent partner and how they can collaborate with the company to deliver real-time data streaming solutions globally.

The Confluent partner ecosystem has recognized outstanding contributions from its global partners, who have helped businesses unlock the value of real-time data streaming. These partnerships enable customers to launch new use cases and accelerate their migration journeys seamlessly, driving meaningful impact and exceeding customer expectations. The recognition marks an exciting new phase of growth as Confluent pursues a $60 billion market opportunity, with capabilities and innovations from its partners unlocking the transformative power of real-time data streaming.