February 2023 Summaries

Marcus Greer, a Senior Software Engineer II in Confluent’s Cloud Manageability organization, joined the company after earning a master’s degree from Carnegie Mellon and now works on the Observability team, which develops systems that help internal and external users monitor data-system performance. Over three years, he has seen Confluent evolve from a fast-paced private startup into a public company with more formal processes for security, compliance, and product delivery, while retaining an emphasis on impactful work. His remote-first role involves coordinating across time zones, translating broad company goals into technical plans, reviewing and writing code, and collaborating on features and fixes. Greer is currently helping modernize Confluent’s telemetry infrastructure with OpenTelemetry to standardize metrics and tracing collection. His interest in distributed systems began through coursework in parallel computing, and he chose Confluent over a larger company because of its energetic culture and opportunities for junior engineers to grow. He also identifies stronger onboarding and more in-person connection opportunities as important areas for the company’s continued development as a remote-first organization.

Apache Kafka is presented as a widely used platform for helping companies process business events in real time, and Confluent’s free Apache Kafka for Python Developers course teaches Python developers how to build applications that stream data to and from Kafka clusters. Led by Confluent Senior Developer Advocate Dave Klein, the course combines instructional modules with hands-on exercises using Confluent Cloud, covering environment setup, Kafka cluster configuration, connection details, and installation of Python client libraries. Participants learn to produce events with the Python Producer class, consume them with the Consumer class, integrate applications with Confluent Schema Registry using JSON serialization and deserialization, and perform basic cluster administration through the AdminClient class. The course also introduces options for applying these skills to more complex event-streaming use cases, aiming to prepare learners to begin developing Kafka-based Python applications.

Apache Kafka is a distributed, real-time data streaming system that uses a pull-based replication model for durability and availability. It stores messages in topics, which are logical groups of one or more partitions, with each partition being an append-only log that guarantees message ordering within the partition. Kafka provides various replication topologies to support multi-geo deployments, including stretched clusters, connected clusters, read replica deployments, global write replication scenarios, fan-in and fan-out architectures, and mixed deployment strategies. These topologies offer different trade-offs between cost, business requirements, use cases, regulatory compliance, resilience to disasters, security, and fault tolerance. Choosing the right topology depends on factors such as data loss tolerance, consumer offset translation needs, clustering size, network latency, and security requirements. With various learning resources available, including Confluent Developer courses and talks, Apache Kafka provides a robust solution for building globally available systems that can handle high availability and disaster recovery use cases.

Thoughtworks developed a Kafka Streams pipeline to replicate changes from a 25-year-old IBM Db2 mainframe database into scalable, domain-oriented microservices without adding load to the source system. A commercial CDC tool emits one Avro message per changed database row, but because messages are randomly partitioned and lack reliable transaction keys, the application first repartitions them by transaction ID, aggregates all records from each transaction in a materialized state store, and releases processing only when the transaction’s final-message indicator confirms completeness. It then converts grouped changes across related tables into business-specific events, such as customer or address creation and modification events, keyed by the relevant domain identifier and sent to separate Kafka topics. The design also handles incomplete or stale transactions through scheduled state-store cleanup and alerts, uses Kafka headers to carry correlation IDs derived from transaction IDs for tracing, supports initial database loads through snapshot events, and relies on downstream consumers to process potentially duplicated events idempotently under an at-least-once delivery model.

Mission-critical Apache Kafka and Confluent Platform deployments can improve efficiency and isolation by separating the data plane, which handles business events, from a control plane that manages monitoring, security, governance, and administration. Internal services such as Control Center, Confluent Metrics Reporter, and Confluent Metadata Service can consume broker storage, CPU, memory, bandwidth, and network capacity through metrics, authorization, metadata, and internal topics, although broker-required internal traffic cannot be entirely removed. The proposed architecture moves monitoring metrics to a remote control-plane cluster and centralizes authorization, authentication, audit logging, and cluster management there, allowing one control-plane cluster to support multiple data-plane clusters. Configuration examples demonstrate connecting data-plane brokers and Control Center to remote services using bootstrap servers, SASL authentication, and JWT-based access. Confluent Cloud implements this separation by design, hosting operational services such as security, monitoring, Schema Registry, Kafka Streams, and ksqlDB separately so Kafka clusters can focus on processing business data.

Apache Kafka can provide database-like capabilities, including durable event storage, replication, ordering, transactions, and potentially indefinite retention, but it is generally not a direct replacement for specialized systems such as relational databases, document stores, search engines, or data lakes. The choice between Kafka alone and Kafka plus another database depends on requirements for retention, data structure, query complexity, transaction guarantees, and lookup performance. A conventional customer-profile CRUD architecture may write events to Kafka while using an external database for low-latency reads and updates, but this duplicates data, complicates resiliency and source-of-truth management, and adds operational cost. Kafka topics cannot efficiently perform arbitrary key lookups because a key can occur at multiple offsets, yet ksqlDB addresses this limitation by materializing the latest keyed records in RocksDB-backed state stores and exposing them through materialized views. These state stores support fast key-based reads, optimized primary-key range scans, updates through new events, and deletions through null-valued tombstone records, while changelog topics enable state recovery. This Kafka-and-ksqlDB design can simplify certain CRUD workloads and reduce dependence on an external database, although it has limitations involving complex queries, storage availability and capacity, recovery requirements, and potential single-node failure risks.

Michelin, a renowned French manufacturer, is leveraging data-driven innovations to advance its mobility solutions by utilizing Apache Kafka for real-time data streaming and event-driven architecture. Initially adopting self-managed Kafka to enhance fleet management and streamline business processes, Michelin faced challenges in operational complexity and scalability, prompting the transition to Confluent Cloud—a managed cloud-native Kafka service. This migration has significantly reduced operational burdens, improved reliability, and is projected to yield a 35% cost saving compared to on-premise operations. With Confluent Cloud, Michelin has enhanced real-time inventory management and unlocked the potential for further innovation across various departments, aligning with its mission of creating a better way forward.

Windowing in Apache Kafka Streams is a technique used to process data over time. It allows developers to define specific time-based windows within which data can be aggregated or processed. There are four types of windowing: hopping, tumbling, session, and sliding. Hopping windows advance at regular intervals, while tumbling windows do not overlap with each other. Session windows are triggered by events and have an inactivity gap, whereas sliding windows require user events to happen within a defined time frame. These techniques enable developers to analyze data over specific time periods, handle late-arriving events, and introduce grace periods for skew analysis. Understanding windowing is essential for processing and analyzing data in real-time applications using Apache Kafka Streams.

A Confluent Vice President of Sales Strategy & Operations describes her career journey from industrial engineering and technical professional services to finding a niche in sales operations, where she combines operational expertise with customer-facing business support. She oversees teams responsible for enterprise sales forecasting, planning, territory design, targets, quoting, order management, systems design, and deal booking, using data to identify growth opportunities, risks, and sales-process challenges. Her leadership approach emphasizes starting with trust, addressing mistakes without blame, collaborating directly with teams, and supporting employees’ family responsibilities and work-life balance. Looking ahead, she focuses on scaling Confluent’s business through automation that reduces repetitive administrative work, improves data flow, and allows employees to concentrate on higher-value responsibilities while adapting to new products and organizational complexity.

Apache Kafka 3.4 introduces numerous new features and improvements, with a key highlight being the early access capability to migrate Kafka clusters from ZooKeeper to KRaft mode without downtime, although this is not yet recommended for production use. The update includes KIP-866, which facilitates this migration by transferring cluster metadata to the KRaft metadata log, allowing brokers to restart in KRaft mode one at a time. Other notable updates include KIP-792, which enhances metadata sharing for partition assignments, and KIP-854, which optimizes producer ID cleanup to improve memory usage. Additionally, KIP-881 extends rack-awareness for partition assignments, though its practical application requires a custom assignor. The release also includes numerous configuration updates, such as the deprecation of certain parameters and the introduction of new settings for better resource management and monitoring. Overall, Apache Kafka 3.4 is a community-driven effort that aims to enhance the platform's scalability, reliability, and integration capabilities, with contributions from a large number of developers.