February 2023 Summaries

Apache Kafka is a distributed, real-time data streaming system that uses a pull-based replication model for durability and availability. It stores messages in topics, which are logical groups of one or more partitions, with each partition being an append-only log that guarantees message ordering within the partition. Kafka provides various replication topologies to support multi-geo deployments, including stretched clusters, connected clusters, read replica deployments, global write replication scenarios, fan-in and fan-out architectures, and mixed deployment strategies. These topologies offer different trade-offs between cost, business requirements, use cases, regulatory compliance, resilience to disasters, security, and fault tolerance. Choosing the right topology depends on factors such as data loss tolerance, consumer offset translation needs, clustering size, network latency, and security requirements. With various learning resources available, including Confluent Developer courses and talks, Apache Kafka provides a robust solution for building globally available systems that can handle high availability and disaster recovery use cases.

Michelin, a renowned French manufacturer, is leveraging data-driven innovations to advance its mobility solutions by utilizing Apache Kafka for real-time data streaming and event-driven architecture. Initially adopting self-managed Kafka to enhance fleet management and streamline business processes, Michelin faced challenges in operational complexity and scalability, prompting the transition to Confluent Cloud—a managed cloud-native Kafka service. This migration has significantly reduced operational burdens, improved reliability, and is projected to yield a 35% cost saving compared to on-premise operations. With Confluent Cloud, Michelin has enhanced real-time inventory management and unlocked the potential for further innovation across various departments, aligning with its mission of creating a better way forward.

Windowing in Apache Kafka Streams is a technique used to process data over time. It allows developers to define specific time-based windows within which data can be aggregated or processed. There are four types of windowing: hopping, tumbling, session, and sliding. Hopping windows advance at regular intervals, while tumbling windows do not overlap with each other. Session windows are triggered by events and have an inactivity gap, whereas sliding windows require user events to happen within a defined time frame. These techniques enable developers to analyze data over specific time periods, handle late-arriving events, and introduce grace periods for skew analysis. Understanding windowing is essential for processing and analyzing data in real-time applications using Apache Kafka Streams.

Apache Kafka 3.4 introduces numerous new features and improvements, with a key highlight being the early access capability to migrate Kafka clusters from ZooKeeper to KRaft mode without downtime, although this is not yet recommended for production use. The update includes KIP-866, which facilitates this migration by transferring cluster metadata to the KRaft metadata log, allowing brokers to restart in KRaft mode one at a time. Other notable updates include KIP-792, which enhances metadata sharing for partition assignments, and KIP-854, which optimizes producer ID cleanup to improve memory usage. Additionally, KIP-881 extends rack-awareness for partition assignments, though its practical application requires a custom assignor. The release also includes numerous configuration updates, such as the deprecation of certain parameters and the introduction of new settings for better resource management and monitoring. Overall, Apache Kafka 3.4 is a community-driven effort that aims to enhance the platform's scalability, reliability, and integration capabilities, with contributions from a large number of developers.