March 2025 Summaries

The Singapore government is facing pressure to provide effective public services in response to growing challenges such as supply chain disruption, pandemics, and climate change. To address this, the Infocomm Media Development Authority (IMDA) has launched its Tech Acceleration Lab (TAL), which brings together government departments and tech innovators to evaluate and test new technologies like real-time data streaming. Confluent Cloud, a data streaming technology, is being used by Singapore's Housing & Development Board (HDB) and other public sector organizations to modernize their approach to data and enhance end-user engagement with data streaming. The partnership between Confluent and the Singapore government aims to help agencies develop innovative solutions that adapt to today's unpredictable challenges, while ensuring the security and safety of end-users.

Confluent Cloud Freight clusters offer a cost-effective and high-throughput solution for ingesting and processing large volumes of data, particularly in industries with unpredictable traffic patterns. By leveraging serverless compute services like AWS Lambda, customers can build scalable and real-time event-driven applications while minimizing maintenance overhead. The integration of Freight clusters with AWS Lambda enables efficient processing of streaming data, including customer feedback and survey interactions, allowing for personalized experiences and optimized customer engagement. With its auto-scaling capabilities and private network interface, Freight clusters provide a secure and reliable environment for processing sensitive data, making it an ideal choice for e-commerce retailers and other organizations requiring high-throughput and low-latency data processing solutions.

The Model Context Protocol (MCP) is a new standard introduced by Anthropic that simplifies artificial intelligence (AI) integrations by providing a secure, consistent way to connect AI agents with external tools and data sources. Confluent has built an MCP server that connects directly to Confluent, allowing agents to interact with real-time data using natural language, eliminating the need for manual configurations. The MCP server standardizes how agents retrieve and interact with external data, providing a universal standard that simplifies connections between AI models and external data sources. This reduces integration complexity, improves security and governance, and ensures that agents operate with accurate, up-to-date information. By providing real-time access to data across disparate systems, the MCP server enables AI agents to make decisions based on the latest available context, eliminating the need for custom, one-off integrations. The Confluent MCP server integrates with other tools and frameworks, such as Tableflow, to provide a unified way for agents to interact with both real-time and stored data.

Syed, a Senior Solutions Engineer at Confluent, has transitioned from being a Solutions Architect in professional services to his current role, where he guides architectural decisions and helps businesses unlock the power of data streaming from day one. This shift has helped him grow not just technically but also in how he communicates and drives real business impact. Syed appreciates Confluent's unique company culture, which balances deep technical expertise with a collaborative mindset, as well as its "one team" mentality, where people are willing to help each other. He is motivated by the impact he can make helping customers solve challenges and seeing the value of data streaming come to life in their businesses. Syed enjoys interacting with customers, learning from their experiences, and gaining insights into different industries and architectures.

Confluent Platform 7.9 has been released, building upon Apache Kafka 3.9, with key capabilities including Oracle XStream CDC Source Connector for secure and reliable streaming of change events, Confluent for VS Code to streamline development, Client-Side Field Level Encryption (CSFLE) for securing sensitive data, and a unified security experience across Confluent Platform with OAuth support for ksqlDB and Flink. This release also includes enhancements to Confluent Control Center, the official Confluent Node.js client for Apache Kafka, KRaft mode improvements, and Ansible Playbooks for Confluent Platform support. With this release, organizations can modernize their data infrastructure, eliminate integration complexity, and unlock real-time intelligence while reducing costs and accelerating innovation.

This is an announcement from Confluent about the general availability of several new features and products in their cloud platform. They have launched Tableflow, a product that represents Kafka topics as Apache Iceberg or Delta Lake tables, making it easier to feed data into data warehouses and analytics engines. Additionally, they have introduced Freight clusters, which are cost-effective cluster types tailored to streaming use cases, offering up to 90% cost savings compared to traditional Confluent Cloud clusters. Other new features include AI enhancements for Apache Flink, a Visual Studio Code extension for Kafka development, an Oracle XStream CDC Premium Connector, and improved Cluster Linking capabilities. Confluent has also announced the general availability of Confluent for VS Code, which integrates with their cloud platform, and is now available as a Generally Available product.

The Confluent for VS Code extension is now Generally Available, offering a free and open-source tool for streaming Kafka data engineers to optimize productivity and collaboration. The extension supports all forms of Apache Kafka deployments, provides streamlined project setup with ready-to-use templates, connects to any Kafka cluster for development, management, debugging, and monitoring real-time data streams, and offers features such as schema evolution support, shareable links to topics, and manual message production. Confluent for VS Code introduces powerful new features designed to enhance workflow efficiency, expand connectivity options, and streamline schema management, making it the ultimate tool for any Kafka developer. The extension is available to install from the Visual Studio Marketplace for free and welcomes contributions to its open-source project on GitHub.

Tableflow is a product from Confluent that simplifies the process of getting Apache Kafka data into structured tables in data lakes, such as Delta Lake and Iceberg. It eliminates the complexity of mapping, converting, and cleaning data by providing a few clicks to represent Kafka topics and their associated schemas as Iceberg or Delta Lake tables. Tableflow uses innovations in Confluent's Kora Storage Layer and a new metadata publishing service to generate Iceberg metadata and handle schema mapping, schema evolution, and type conversions. It supports integrations with various analytical tools and data lakes, including AWS Glue, Databricks Unity Catalog, Snowflake Open Catalog, and Apache Polaris. By combining Tableflow with stream processing from Apache Flink, businesses can create a unified data foundation that allows for real-time analysis and decision-making. The product is now generally available for Iceberg tables and in Early Access for Delta Lake tables.

The Confluent Platform 7.9 release builds upon Apache Kafka 3.9, enhancing key capabilities for connecting and building trusted data products. New features include the Oracle XStream CDC Source Connector for secure and reliable streaming of change events, Confluent for VS Code for streamlined development and debugging in one place, Client-Side Field Level Encryption (CSFLE) for protecting sensitive data, and a new self-managed connector to the Connect portfolio powered by Oracle XStream technology. Additionally, Confluent Platform 7.9 offers improved security features, such as OAuth support for ksqlDB and Flink, and enhanced Confluent Control Center capabilities. The release also introduces official Confluent Node.js clients for Apache Kafka and supports KRaft mode enhancements.

Tableflow has been released by Confluent, a major milestone that makes data engineers' lives easier. It solves the pain points associated with streaming Apache Kafka data into a data lake, including fragile pipelines, bad data cleanup, and slow and expensive processing. Tableflow represents Kafka topics and their schemas as Iceberg or Delta Lake tables in just a few clicks, eliminating the need for custom code or pipelines to scale and manage. This innovation uses Confluent's Kora Storage Layer and a new metadata publishing service to generate Iceberg metadata and handle schema mapping, evolution, and type conversions. By combining Tableflow with stream processing from Apache Flink, businesses can transform raw Kafka events into structured tables that are instantly accessible for AI and analytics tools, creating a unified data foundation that allows them to move faster, respond smarter, and drive better outcomes.

This summary highlights several key announcements from Confluent, a leader in event-driven data processing. The Q1 launch of Confluent Cloud features the general availability of Tableflow, which simplifies the process of representing Kafka topics as Apache Iceberg or Delta Lake tables to form bronze and silver tables with automated data maintenance and strong read performance. Additionally, Freight clusters are now available on AWS, offering up to 90% cost savings by bypassing expensive replication and leveraging object storage. Confluent Cloud also introduces AI enhancements for Flink, a developer-friendly Visual Studio Code extension, and an Oracle XStream CDC Premium Connector with enterprise-grade performance and scalability. Furthermore, the company has expanded its partnership with Databricks, strengthened its partnership with Jio Platforms Limited, and released Apache Kafka 4.0 with several new improvements and features.

Apache Kafka 4.0 has been released, marking a significant milestone in the project's evolution. The new release operates entirely without Apache ZooKeeper, simplifying deployment and management, and reducing operational overhead, scalability, and administrative tasks. This change is made possible by running in KRaft mode by default. Additionally, Kafka 4.0 introduces a powerful new consumer group protocol (KIP-848) that significantly improves rebalance performance, reducing downtime and latency. The release also brings the general availability of Queues for Kafka (KIP-932), enabling Kafka to support traditional queue semantics directly. Furthermore, Kafka 4.0 requires Java 11 for clients and Java 17 for brokers, tools, and Connect. Other notable features include improved consumer group administration, enhanced Kafka Streams operator metrics, and the removal of old client protocol API versions in Kafka 4.0. The release is a result of community efforts, with over 175 contributors involved.

Apache Kafka has released its version 4.0, marking a significant milestone with several new features and improvements. The release contains many changes, including the removal of deprecated APIs and the introduction of new protocols such as KRaft mode, which simplifies deployment and management by eliminating the need for a separate ZooKeeper ensemble. Additionally, Kafka 4.0 brings improved consumer group protocol performance, enabling faster rebalancing and reducing downtime and latency. The release also introduces Queues for Kafka, allowing for traditional queue semantics directly, and enables clients to rebootstrap based on timeout or error code. Furthermore, Apache Kafka 4.0 simplifies the upgrade process by removing old client protocol API versions and providing a clear Kafka Client upgrade path. The release is accompanied by numerous bug fixes and improvements in various areas, including Connect, Streams, and MirrorMaker.

This summary discusses the challenges of maintaining a positive atmosphere for players while preventing toxic behavior in online gaming communities. Traditional methods such as post hoc reporting and keyword-based filtering are shown to be ineffective, leading to lower engagement and lost revenue. A scalable, real-time, AI- and machine learning-based detection system using Confluent's data streaming platform paired with the Databricks data intelligence platform is proposed to identify and respond to toxic messages without disrupting the natural flow of in-game chat. The system uses a specialized local model to quickly flag potentially problematic interactions and a deeper analysis service in Databricks for robust AI analysis and decision-making. The architecture allows for effective moderation while preserving enjoyable player interactions, maintaining a positive community, and ensuring long-term success for games.

The text outlines the evolution of Confluent, a data streaming platform, from its origins as a self-managed Apache Kafka solution to a comprehensive and fully managed Data Streaming Platform (DSP) that addresses modern data challenges. The document highlights the importance of DSPs in business operations, as evidenced by a survey indicating 91% of IT leaders find them critical or important. It discusses Confluent's journey through three "acts," from improving Kafka manageability and accessibility to launching cloud offerings like Confluent Cloud, which offers significant cost savings and ROI. Confluent's DSP now integrates streaming, connecting, processing, and governing data, leveraging engines like Apache Flink and offering over 120 prebuilt connectors. The platform aims to bridge the gap between real-time data applications and analytics or AI platforms, transforming data management across enterprises and highlighting its strategic importance. The text also emphasizes the challenge IT teams face in conveying DSP's business value to non-technical stakeholders and promises further exploration of this issue in a subsequent series part.

Confluent's Data Streaming Platform (DSP) is portrayed as a crucial component in driving business value by supporting revenue generation, cost savings, and risk mitigation. The platform enables companies to expedite application development and enhance real-time data accessibility, leading to faster time-to-market and new business solutions that traditional methods struggle to deliver. Hypothetical examples illustrate how DSPs can significantly contribute to revenue, such as a fintech company's real-time payments platform generating substantial income. Additionally, DSPs streamline data management and governance, reducing operational costs and improving efficiency by simplifying architectures and enabling the creation of reusable, real-time data products. While assigning precise monetary value to DSP benefits can be complex, the potential savings and revenue enhancements are supported by external studies and the platform's ability to future-proof data architecture, particularly in the context of AI and machine learning advancements. Confluent emphasizes the DSP's transformative potential and offers consultation services to organizations interested in leveraging the platform's capabilities.

Deleting a topic in Apache Kafka involves several considerations depending on the Kafka environment, such as self-managed, cloud-hosted, or fully managed services like Confluent Cloud. The process requires enabling the delete.topic.enable property in the Kafka broker configuration and ensuring permissions are set correctly to allow deletion. Reasons for deleting a topic include retiring unused topics, reorganizing data flow, removing sensitive data, correcting configuration errors, or switching to a new data model. It is crucial to consider dependencies on the topic from other applications or services to prevent disruptions. Proper deletion involves stopping active consumers, verifying permissions, backing up data, and monitoring the process to ensure the topic is fully removed. Automating the deletion process using tools like Bash scripts, Python libraries, or Ansible can help streamline operations and reduce risks. Additionally, Kafka users can manage retention policies or use topic compaction to minimize disk usage instead of deleting topics outright. For multi-cluster environments, it is important to handle replication carefully to avoid partial deletions.

Hands-on Flink Workshop: Implement Stream Processing | Register Now. Easy access to data is crucial for any successful business strategy, but many businesses face challenges accessing and using their vast amounts of data due to siloed systems and infrastructure. Data silos create a ripple effect across organizations, slowing down daily operations, hindering decision-making, and impacting long-term growth. Poor access to data also affects innovation, as teams miss big-picture insights they need to spot new opportunities or improve customer experiences. Implementing a strategic approach to data management and accessibility can help overcome these challenges by providing unified data access, fostering collaboration, driving innovation, and unleashing the full potential of an organization's data. Effective data governance is key to balancing accessibility with security and compliance, and leaders must build in place solid strategies and the right combination of data management tools to ensure data accessibility across their organizations. By creating a culture of informed decision-making, where insights aren't limited to executives or analysts but are available to anyone ready to make an impact, organizations can unlock new ideas and efficiencies, driving sustainable growth and success.

Hands-on Flink Workshop: Implement Stream Processing | Register Now is about creating a culture where data-driven decision-making is second nature at every level. It's not just about having the right tools, but also about giving employees access to reliable, actionable data and the skills to use it with confidence. Building a data-driven culture requires a fundamental shift where teams base their decisions on data insights rather than gut feelings, making data accessible and promoting transparency. Confluent makes this possible by acting as the central nervous system for your data, providing real-time data streaming, processing, and governance capabilities. A thriving, data-driven culture happens when technology, processes, and people work together, empowering teams to create and maintain systems that put decisions into action. Leadership plays a crucial role in promoting learning, development opportunities, committing resources, supporting teams through challenges, rewarding data-driven decisions, bridging the gap between technical and business teams, and highlighting wins from smart insights. By focusing equally on the technical and cultural sides of becoming data-driven, leadership can create an environment where innovation thrives and teams feel empowered to drive real change.

Hands-on Flink Workshop: Implement Stream Processing | Register Now. Relying on intuition alone isn’t enough to stay ahead of the game in today’s fast-paced business environment, requiring smart, data-driven decision-making backed by real-time insights and analytics. With stream processing and modern analytics platforms, businesses can collect, process, and analyze information as it happens, giving them a clear edge. Data-driven organizations that invest consistently in innovation significantly outperform their peers in critical business metrics, with emerging technologies like generative artificial intelligence (AI) potentially adding $2.6 trillion to $4.4 trillion in annual value across 63 analyzed use cases. Implementing real-time data analytics can completely change how an organization operates on multiple levels, enhancing operational decision-making and driving better customer experiences by allowing businesses to personalize offerings based on real-time insights. Real-time analytics also gives organizations the agility to quickly adapt to changing market conditions, spot trends, adjust strategies on the fly, and seize new opportunities faster than the competition. However, data silos, poor data quality, and inconsistent formats can hold back analytics efforts, requiring solid data integration platforms and clear data governance policies. To implement real-time analytics, organizations need to focus on setting clear goals, building a strong technical foundation with modern tools like Apache Kafka and Apache Flink, and shifting their approach to data by creating specialized "data squads" that bring together tech expertise and business knowledge.

Data-driven agility is increasingly essential for businesses to remain competitive, allowing them to quickly adapt to market changes by leveraging real-time data for informed decision-making. This approach is particularly beneficial in dynamic industries such as financial services, retail, and healthcare, where rapid responses to shifting conditions are crucial. Companies like Netflix exemplify the successful application of data-driven agility by using data insights to transform business models and maintain market leadership. The integration of data streaming platforms such as Apache Kafka and Apache Flink facilitates this agility by enabling efficient processing of massive data volumes, thus supporting rapid decision-making and operational responsiveness. Challenges such as data quality, scalability, and the complexity of analytics pipelines must be addressed through comprehensive strategies that include investing in scalable technology, focusing on data governance, and fostering a data-centric organizational culture. Ultimately, embracing data-driven agility positions businesses to innovate, personalize customer experiences, and sustain a competitive advantage.

This summary highlights how Confluent's data streaming platform enables businesses to connect, process, and govern data in motion to fuel AI-driven enterprises. With the expansion of its Connect with Confluent technology partner program, Confluent is integrating real-time data streams across various enterprise systems, simplifying Kafka management, and enabling widespread adoption of Confluent Cloud's powerful capabilities. This allows businesses to power real-time, low-latency experiences efficiently and cost-effectively, unlocking more value from their data without the burden of managing complex infrastructure. The program provides fully managed solutions that eliminate the complexity of self-managing Apache Kafka, reducing TCO by up to 60%. With each new CwC integration, customers can instantly share and access data across Confluent's vast streaming network, making real-time insights readily available across business applications.

The text discusses the use of Retrieval Augmented Generation (RAG) combined with Large Language Models (LLMs) in stream processing and data analytics. It explains how a chatbot can be built to answer user queries using RAG and LLMs, and how this technology can be used in enterprise applications such as customer relationship management (CRM) or healthcare. The text also introduces the FEDERATED_SEARCH() function in Confluent Cloud for Apache Flink, which enables searching through external vector databases like Elasticsearch, Pinecone, and MongoDB Atlas. It demonstrates how to use FEDERATED_SEARCH() with ML_PREDICT() to orchestrate the RAG workflow, converting user queries into vector embeddings and searching them against a knowledgebase stored in a vector database. The text concludes that these features enable developers to connect real-time data to external models through remote inference, paving the way for building complex agentic workflows for enterprise use cases.