February 2024 Summaries

Kafka Summit London 2024 will take place at ExCeL London on March 19–20, bringing together the Apache Kafka community for more than 90 sessions, lightning talks, technical deep dives, training opportunities, certification exams, networking events, and an expo hall. The event will retain familiar features such as expert discussions, community meetups, themed luncheons, social media activities, and a Day 1 party featuring live-coded music by Sam Aaron. A new “Current Track” will broaden the conference beyond Kafka with content on complementary real-time technologies including Apache Spark, Apache Flink, Debezium, and generative AI. Attendees are encouraged to plan their agendas in advance, with highlighted sessions covering event-modeling anti-patterns, market-data processing, and Kafka Connect. Registration includes a discount code, while remote participants can access a free livestream of keynotes and one content track, and all sessions will later be published for free.

Sergio Duran Vegas, a Global Staff Solutions Engineer at Confluent, reflects on four years of developing technical expertise, customer engagement skills, and cross-functional collaboration experience. He describes his role as applying software and data-streaming knowledge to practical customer challenges, including fraud prevention and public-service improvements, and values the opportunity to work with customers, partners, and colleagues on solutions with broad social impact. Having joined without deep Kafka expertise, he expanded from a traditional data and integration background and is now focused on learning Flink to support emerging streaming use cases. Sergio says his growth has also included learning how to prioritize effort, shape customer strategy, and collaborate with humility and empathy within a sales-oriented technical team. He attributes Confluent’s culture to a shared “#gladtobehere” mindset, mentorship from managers, and highly motivated teammates, while noting personal interests such as Jeet Kune Do sparring and a preference for a large work-from-home display.

Apache Kafka 3.7.0 has been released, bringing several new features and improvements. The release includes support for Java 11, which is now deprecated and will be removed in Apache Kafka 4.0. Additionally, the ability to migrate Kafka clusters from ZooKeeper metadata systems to KRaft metadata systems has been improved, with an early access release of JBOD (Java-based distributed object) available in KRaft clusters. The new simplified Consumer Rebalance Protocol provides better performance and reduces latency. Other notable features include support for client-level metrics via a standardized telemetry interface, transaction verification, and improved performance metrics for Tiered Storage. The update also includes changes to the Connect API, including the addition of a BooleanConverter and the deprecation of a redundant endpoint for fetching task configurations. Overall, this release aims to improve the performance, security, and resilience of Apache Kafka.

Confluent has enhanced its cloud-native, serverless Apache Flink service by introducing pre-packaged, turn-key stream processing workloads called Actions, which simplify and streamline the deployment of common workloads such as topic deduplication and field masking, allowing users to utilize Flink's capabilities with minimal configuration. These Actions reduce overhead by handling domain-agnostic requirements, enabling users to focus on core tasks without needing extensive Flink expertise. Additionally, Confluent has integrated Terraform support for Flink, facilitating automated, consistent deployments across multiple environments and CI/CD pipelines, and expanded its Flink service availability to Google Cloud Platform (GCP) alongside AWS and Azure, offering a multi-cloud deployment capability. The enhancements aim to improve flexibility and efficiency in stream processing workflows and mark a significant step in Confluent's ongoing development of Flink services, with plans for future feature expansions.

Confluent's Data Streaming Startup Challenge received nearly 100 applications from 22 countries worldwide. Three global finalists were selected for their innovative use cases with data streaming architecture. Atomic Tessallator, Busie, and TwinLabs.ai impressed the judges with their real-time simulations, transportation operating system, and digital twin platform respectively. The finals will take place at Kafka Summit London on March 19, 2024, where the grand prize winner will receive up to $500,000 investment from Confluent.

Change data capture (CDC) is a widely used technique to connect database tables to data streams but has drawbacks related to exposing internal data models to downstream consumers, which can lead to system failures. The evolution of this pattern involves using first-class data products and data contracts to decouple internal models from external data products. This approach allows for the creation of reliable data streams that can be consumed by various applications, whether operational or analytical. A data product formalizes responsibilities and includes a data contract, which defines schema, metadata, and dedicated ownership, ensuring the data remains trustworthy and easy to use. The post discusses different techniques for building such data products, including the outbox pattern and utilizing Apache Flink SQL to handle data from multiple sources. Confluent's Data Portal facilitates the discovery and management of streaming data products, enhancing collaboration and data governance. The post emphasizes the benefits of a stream-first approach to data products, which enables both real-time and batch processing while maintaining high data quality and interoperability.

With the increasing importance of real-time data in modern businesses, companies are leveraging distributed streaming platforms to process and analyze data streams in real-time. Confluent Platform allows users to connect, process, and react to all their data in real-time with a comprehensive, self-managed platform for Apache Kafka. The platform has introduced several core features that make hybrid and on-premises data streaming simple, secure, and resilient. These features include Apache Kafka Raft metadata mode, which improves stability, simplifies the software, and makes it easier to monitor, administer, and support Kafka. Additionally, Confluent for Kubernetes provides a comprehensive, declarative API to deploy and operate Confluent as a cloud-native system on Kubernetes. The platform has also introduced compaction support for Tiered Storage, allowing users to offload even more data to object storage and improve elasticity. Furthermore, the ecosystem of connectors built by Confluent, the Kafka community, or partners enables customers to integrate different ecosystems with Kafka and Confluent Platform more quickly and reliably. The platform's features have been validated in production usage with multiple backing object storage vendors. To ensure high-quality data, the platform has introduced Data Quality Rules, which serve as a formal agreement between upstream and downstream components regarding the structure, semantics, and quality of data in motion. Overall, Confluent Platform 7.6 is built on Apache Kafka 3.6 and includes several new features that simplify architecture, enhance security, and enable cost-effective scaling.

Confluent has introduced the Migration Accelerator program to streamline the migration from Apache Kafka or traditional messaging systems to its cloud-native data streaming platform, promising cost savings and enhanced operational efficiency. The program aims to alleviate the operational challenges associated with managing open-source Kafka by offering a complete, enterprise-grade platform equipped with all necessary tools for swift, reliable, and secure developer operations. The Migration Accelerator also facilitates transitions by providing expert guidance, funding credits (subject to approval), and leveraging a global network of certified system integrators, including partners like AWS, Google Cloud, and Microsoft Azure, to ensure a seamless migration process. By re-architecting Apache Kafka with its proprietary Kora engine, Confluent ensures significant improvements in cloud service performance and infrastructure cost savings, boasting a 10x enhancement in cloud operations. The program's flexibility allows deployment across on-premises, hybrid cloud, and multicloud environments, enabling organizations to focus on high-value projects instead of low-level platform maintenance.

Confluent for Kubernetes (CfK) can serve as a GitOps control plane for managing Confluent Platform resources, including topics, schemas, connectors, and RBAC bindings, even when the Confluent cluster runs outside Kubernetes. By storing declarative resource definitions in version control and deploying them through CI/CD tools such as Jenkins, GitHub Actions, Argo CD, Flux CD, or Kustomize, organizations can enforce review, approval, auditing, and repeatable deployment processes without sharing administrator credentials. The demonstration uses cp-demo for a preconfigured Confluent Platform cluster and Kind for Kubernetes, where CfK uses separately managed Kubernetes secrets for authentication and TLS, a KafkaRestClass for cluster connection settings, and custom resources to provision a topic, Avro schema, Elasticsearch sink connector, and ResourceOwner permissions for the KafkaDevelopers group. Verification through Kubernetes, Control Center, and the Confluent CLI confirms that the resources and permissions are created successfully, illustrating how this approach can support self-service topic and connector management across multiple environments while reducing administrators’ operational workload.

Deok Hwan Choi, a staff product designer at Confluent, describes his three-year experience as one of ongoing growth in understanding user needs, developing strategic thinking, and creating simple UX solutions for the company’s data streaming platform. He credits Confluent’s collaborative culture, supportive leadership, and regular teamwork with motivating innovation and helping him build skills in UX research, brainstorming facilitation, and aligning teams around design visions. Looking ahead, Choi aims to improve how he communicates design strategy to stakeholders and deepen cross-functional partnerships to strengthen product experiences. He highlights Confluent’s “One-Team” culture, maintained despite rapid growth and remote work through recognition and belonging, and notes that employee resource groups and related events support diversity, equity, inclusion, and greater empathy across the organization.

This article discusses how companies can build GenAI-enabled applications using a combination of foundation models like LLMs, data streaming platforms, and event-driven patterns. The approach involves breaking down the application into four steps: data augmentation, inference, workflows, and post-processing, which are ideally implemented as separate event-driven services. This allows for scalability, independence, and real-time processing of data, making it possible to generate reliable results. A data streaming platform can help integrate disparate operational data across the enterprise in real-time, enabling businesses to route relevant data streams to anywhere they're needed. By embracing an event-driven methodology, companies can decouple systems, teams, and technologies, facilitating data products that are well contextualized, trustworthy, and discoverable, ultimately promoting data reusability, engineering agility, and greater trust.

Confluent announced that it is welcoming the Noteable team to strengthen its efforts to make data streaming and application development more accessible for Kafka and Flink users. Noteable developed a collaborative data notebook that combines code, natural-language interaction, and visualizations to help teams move from data exploration to production while supporting open standards and collaboration. The companies share a goal of democratizing access to data and reducing silos among data practitioners. At Confluent, the Noteable team will contribute to data experiences, particularly by expanding Flink stream-processing and data-visualization capabilities, as Confluent continues to develop its broader data-streaming platform and Kora engine.

IoT data is presented as a strategic resource for wireless companies, supporting network optimization, customer experiences, operational efficiency, and new services, while creating challenges involving scale, cybersecurity, data management, and low-latency processing. The material positions Confluent Cloud, a managed Apache Kafka-compatible streaming platform, as a solution for ingesting, processing, transforming, securing, integrating, retaining, and distributing IoT data across cloud, multicloud, and edge environments. It describes a wireless company that used Confluent Cloud and bidirectional Cluster Linking across AWS regions to support cross-region replication and an active-active architecture for an IoT network of devices such as phones, cameras, robots, and sensors, targeting 99.999% availability through redundancy, failover, replication, load balancing, and disaster-recovery practices. The implementation included MongoDB Atlas source and sink connectors, real-time telemetry enrichment and aggregation, downstream analytics in systems such as Snowflake, and separate production, testing, staging, and development clusters. Claimed benefits include faster deployment, reduced infrastructure and operational costs, stronger security controls, self-service data sharing, and improved resilience for real-time IoT applications.