February 2024 Summaries

Apache Kafka 3.7.0 has been released, bringing several new features and improvements. The release includes support for Java 11, which is now deprecated and will be removed in Apache Kafka 4.0. Additionally, the ability to migrate Kafka clusters from ZooKeeper metadata systems to KRaft metadata systems has been improved, with an early access release of JBOD (Java-based distributed object) available in KRaft clusters. The new simplified Consumer Rebalance Protocol provides better performance and reduces latency. Other notable features include support for client-level metrics via a standardized telemetry interface, transaction verification, and improved performance metrics for Tiered Storage. The update also includes changes to the Connect API, including the addition of a BooleanConverter and the deprecation of a redundant endpoint for fetching task configurations. Overall, this release aims to improve the performance, security, and resilience of Apache Kafka.

Confluent has enhanced its cloud-native, serverless Apache Flink service by introducing pre-packaged, turn-key stream processing workloads called Actions, which simplify and streamline the deployment of common workloads such as topic deduplication and field masking, allowing users to utilize Flink's capabilities with minimal configuration. These Actions reduce overhead by handling domain-agnostic requirements, enabling users to focus on core tasks without needing extensive Flink expertise. Additionally, Confluent has integrated Terraform support for Flink, facilitating automated, consistent deployments across multiple environments and CI/CD pipelines, and expanded its Flink service availability to Google Cloud Platform (GCP) alongside AWS and Azure, offering a multi-cloud deployment capability. The enhancements aim to improve flexibility and efficiency in stream processing workflows and mark a significant step in Confluent's ongoing development of Flink services, with plans for future feature expansions.

Confluent's Data Streaming Startup Challenge received nearly 100 applications from 22 countries worldwide. Three global finalists were selected for their innovative use cases with data streaming architecture. Atomic Tessallator, Busie, and TwinLabs.ai impressed the judges with their real-time simulations, transportation operating system, and digital twin platform respectively. The finals will take place at Kafka Summit London on March 19, 2024, where the grand prize winner will receive up to $500,000 investment from Confluent.

Change data capture (CDC) is a widely used technique to connect database tables to data streams but has drawbacks related to exposing internal data models to downstream consumers, which can lead to system failures. The evolution of this pattern involves using first-class data products and data contracts to decouple internal models from external data products. This approach allows for the creation of reliable data streams that can be consumed by various applications, whether operational or analytical. A data product formalizes responsibilities and includes a data contract, which defines schema, metadata, and dedicated ownership, ensuring the data remains trustworthy and easy to use. The post discusses different techniques for building such data products, including the outbox pattern and utilizing Apache Flink SQL to handle data from multiple sources. Confluent's Data Portal facilitates the discovery and management of streaming data products, enhancing collaboration and data governance. The post emphasizes the benefits of a stream-first approach to data products, which enables both real-time and batch processing while maintaining high data quality and interoperability.

With the increasing importance of real-time data in modern businesses, companies are leveraging distributed streaming platforms to process and analyze data streams in real-time. Confluent Platform allows users to connect, process, and react to all their data in real-time with a comprehensive, self-managed platform for Apache Kafka. The platform has introduced several core features that make hybrid and on-premises data streaming simple, secure, and resilient. These features include Apache Kafka Raft metadata mode, which improves stability, simplifies the software, and makes it easier to monitor, administer, and support Kafka. Additionally, Confluent for Kubernetes provides a comprehensive, declarative API to deploy and operate Confluent as a cloud-native system on Kubernetes. The platform has also introduced compaction support for Tiered Storage, allowing users to offload even more data to object storage and improve elasticity. Furthermore, the ecosystem of connectors built by Confluent, the Kafka community, or partners enables customers to integrate different ecosystems with Kafka and Confluent Platform more quickly and reliably. The platform's features have been validated in production usage with multiple backing object storage vendors. To ensure high-quality data, the platform has introduced Data Quality Rules, which serve as a formal agreement between upstream and downstream components regarding the structure, semantics, and quality of data in motion. Overall, Confluent Platform 7.6 is built on Apache Kafka 3.6 and includes several new features that simplify architecture, enhance security, and enable cost-effective scaling.

Confluent has introduced the Migration Accelerator program to streamline the migration from Apache Kafka or traditional messaging systems to its cloud-native data streaming platform, promising cost savings and enhanced operational efficiency. The program aims to alleviate the operational challenges associated with managing open-source Kafka by offering a complete, enterprise-grade platform equipped with all necessary tools for swift, reliable, and secure developer operations. The Migration Accelerator also facilitates transitions by providing expert guidance, funding credits (subject to approval), and leveraging a global network of certified system integrators, including partners like AWS, Google Cloud, and Microsoft Azure, to ensure a seamless migration process. By re-architecting Apache Kafka with its proprietary Kora engine, Confluent ensures significant improvements in cloud service performance and infrastructure cost savings, boasting a 10x enhancement in cloud operations. The program's flexibility allows deployment across on-premises, hybrid cloud, and multicloud environments, enabling organizations to focus on high-value projects instead of low-level platform maintenance.

This article discusses how companies can build GenAI-enabled applications using a combination of foundation models like LLMs, data streaming platforms, and event-driven patterns. The approach involves breaking down the application into four steps: data augmentation, inference, workflows, and post-processing, which are ideally implemented as separate event-driven services. This allows for scalability, independence, and real-time processing of data, making it possible to generate reliable results. A data streaming platform can help integrate disparate operational data across the enterprise in real-time, enabling businesses to route relevant data streams to anywhere they're needed. By embracing an event-driven methodology, companies can decouple systems, teams, and technologies, facilitating data products that are well contextualized, trustworthy, and discoverable, ultimately promoting data reusability, engineering agility, and greater trust.