March 2021 Summaries

The Apache Kafka system has undergone significant changes with the introduction of the KIP-500 code, which replaces ZooKeeper with an internal quorum service called KRaft mode. This new feature simplifies the deployment and operation of Kafka clusters, allowing for a single-process deployment and improved scalability. The KRaft mode uses an event-driven architecture to ensure accurate replication of metadata across the cluster. With this change, Kafka can now support significantly more partitions per cluster than previously possible. The quorum controller is designed to handle a much larger number of partitions, reducing controlled shutdown times from 135 seconds to 32 seconds and recovery times from 503 seconds to 37 seconds. This improvement enables Kafka to be used for both small-scale workloads and high-throughput use cases without the complexity of managing two separate systems. The new quorum controller is available in trunk today in experimental mode and is expected to be included in the upcoming Apache Kafka 2.8 release, providing a lightweight way to get started with Kafka or use it as an alternative to monolithic brokers like ActiveMQ or RabbitMQ.

Confluent Control Center provides a UI for monitoring Kafka clusters, while Prometheus and Grafana offer more customizable dashboards for ad-hoc needs. To integrate Confluent with Prometheus, operators need to configure the server-side component and client-side configuration, including renaming attribute names and exporting MBeans in a Prometheus-compatible format. After setting up the Prometheus server to scrape Confluent metrics, operators can create beautiful dashboards using Grafana, which offers real-time visualization of Confluent components' activity. The integration allows for easy monitoring of Confluent clusters, enabling operators to quickly assess cluster health and performance, set up data flows, and more.

LiveStreams is a YouTube show dedicated to exploring Confluent and real-time data streaming technologies, offering practical insights into coding and DevOps. Each episode provides hands-on experience with tools such as Kafka and Confluent, addressing common questions from a global audience. The show covers a range of topics, including setting up Spring Boot with Confluent Cloud, processing data with Kafka Streams, and optimizing data using binary formats like Avro and Protobuf. It also delves into automating tasks with ccloud-stack, building event-driven microservices, and utilizing ksqlDB Java client for data interaction. Episodes further explore integrating Project Reactor with ksqlDB for reactive programming, generating materialized views for data access, and combining Kubernetes, Spring Boot, and Kafka Streams for advanced application deployment. The program emphasizes a transition in Spring Cloud Stream towards a function-based approach, simplifying the creation of event-driven applications. Viewers are encouraged to subscribe for upcoming episodes that focus on managing secrets with Kubernetes and HashiCorp Vault and developing event-driven microservices with Apache Kafka.

Building cross-platform solutions with Microsoft Azure and Confluent enables organizations to leverage real-time data capabilities and low-latency databases for scalable applications. Azure Cosmos DB is a fully managed NoSQL database with SLAs for latency and availability, and it integrates seamlessly with Confluent to connect large-scale applications and microservices. The collaboration between Confluent and Microsoft's Commercial Software Engineering group has resulted in a self-managed Azure Cosmos DB Connector, facilitating data movement and transformation from legacy systems to new cloud environments. The demo provided outlines the process of using Confluent to populate test data in Azure Cosmos DB, detailing steps such as setting up Docker, installing connectors, and configuring Kafka topics. The integration allows for the automation of data pipelines, ensuring the synchronization of legacy and cloud systems, which is crucial for maintaining a cohesive global information system. The tutorial also emphasizes the importance of topic mapping for database synchronization and provides guidance on configuring connectors within the Confluent Control Center for real-time data validation in Azure Cosmos DB.

Confluent Cloud has chosen HashiCorp Vault as its secure secrets management system to enable the seamless motion of data, relying on Kubernetes and Google Cloud infrastructure for high availability. Although outsourcing was considered, Confluent leveraged in-house expertise to manage Vault using Banzai Cloud's Vault operator for Kubernetes, which proved manageable without third-party services. The setup included load tests with Slapper to ensure failover efficiency and chaos resilience, while post-setup configurations were facilitated via Golang programs for simplicity. Additionally, Confluent has implemented procedures for AppRole Secret ID revocation to maintain security and compliance, emphasizing proactive measures in managing and rotating credentials.

Funding Circle employs Kafka as a central element in their event-driven microservices architecture, which offers benefits like decentralization but also presents challenges in system understanding and observability. To address these challenges, they explore distributed tracing, which profiles and monitors applications to identify failures and performance issues and provide insight into system flows. The blog post discusses the state of tracing in the Kafka ecosystem and introduces OpenTelemetry tools for enhancing telemetry collection without modifying source code. Funding Circle demonstrates a scenario utilizing OpenTelemetry for distributed tracing across Kafka-enabled services, highlighting the benefits of improved observability in debugging, incident response, and documentation. Despite challenges in stateful stream processing that hinder full end-to-end observability, the company remains optimistic about future improvements in the community's tracing capabilities. The post concludes by recommending starting with non-intrusive telemetry using OpenTelemetry to enhance productivity and gradually exploring more customizable options as familiarity with the tools grows.

Apache Kafka includes Kafka Streams, a Java and Scala client library designed for building scalable applications and microservices that process data in Kafka. Kafka Streams supports both stateless operations, such as maps and filters, and stateful operations, like windowed joins and aggregations, using local state stores backed by RocksDB for fault tolerance. RocksDB, a key-value store developed by Facebook, is optimized for high performance and is configured to use level or universal compaction to manage data size and write amplification. Kafka Streams distributes tasks across partitions, using a topology of processors connected by streams, where stateful processors utilize state stores that are fault-tolerant due to changelog topics in Kafka. By default, Kafka Streams employs universal compaction but can be adjusted to level compaction to manage disk usage. Operational challenges such as high memory, disk usage, or write stalls can be addressed by tuning RocksDB configurations, like adjusting memtable sizes, block cache, compaction styles, and the number of open files. These configurations help maintain efficient performance and fault tolerance in Kafka Streams applications.

Real-time analytics has become essential for modern internet companies to derive insights from raw logs and provide usage analytics to millions of users, exemplified by LinkedIn’s and Uber’s applications. Apache Pinot™, a distributed analytics data store, is emerging as a preferred solution for building scalable real-time analytical applications due to its low-latency performance and ability to handle high-throughput queries. Pinot can ingest data from various sources, including Apache Kafka, and processes this data through a distributed architecture involving Pinot tables, segments, and brokers. It manages real-time data ingestion efficiently by creating mutable and immutable segments, utilizing off-heap memory management to improve resource efficiency and cluster stability. Pinot's integration with Kafka allows for minimal coordination between replicas, and the system is designed to handle complex queries while ensuring data freshness and low query latency. Through advanced memory management techniques, Pinot reduces memory pressure and enhances performance, making it a robust choice for real-time data processing.

Confluent Platform 6.0 introduces the Cluster Registry, a feature designed to aid administrators in managing multiple Apache Kafka environments by centralizing cluster information within the Confluent Platform Metadata Service (MDS). This registry allows seamless integration with Confluent Control Center, a web-based tool for managing and monitoring Kafka clusters and related components like Kafka connectors and Schema Registry. By reading cluster data from the Cluster Registry, Control Center minimizes duplicate work and reduces errors associated with manual cluster configuration. The platform supports both manual and automated cluster registration through CLI commands, enhancing the efficiency of managing multi-cluster environments. Additionally, the integration of Role-Based Access Control (RBAC) ensures secure management of cross-cluster authorization data. For those preferring a managed service, Confluent Cloud offers a cloud-native alternative that handles infrastructure management tasks.

In 2021, three Kafka Summits are being held virtually across different regions to accommodate global participants. The events aim to engage the Kafka community by featuring a variety of presentations, including keynotes from notable figures like Jay Kreps and discussions on innovative topics such as Data Mesh by Zhamak Dehghani. Other highlights include Stripe's strategies for managing message processing outages, Adidas' advanced observability system handling massive data volumes, and AO's development of an enterprise-wide data governance framework. Additionally, the summit will cover technical advancements like Kafka's implementation of the Raft protocol and practical insights from experienced professionals like Simon Aubury and Jason Gustafson. The successful organization of these summits is attributed to the dedicated efforts of the Kafka Summit Program Committee, who are responsible for curating a comprehensive agenda.

The blog post explores the complexities of data collection in modern companies, particularly through the use of Kafka Connect for data pipeline integration. It expands on a Kafka Summit session by detailing how to replace a Security Information and Event Management (SIEM) vendor's data collection layer with Kafka Connect, focusing on collecting data from remote hosts and services. The post outlines the technical details of using the NettySource connector to receive data via TCP and UDP protocols, as well as the PollableAPIClient connector for pulling data from APIs. It emphasizes the importance of choosing between push and pull data collection methods and provides configurations for both connectors. The post also introduces a transformations library for sorting, filtering, and transforming data to ensure it reaches the right place in the correct format. It highlights the flexibility of Kafka Connect in managing large-scale data collection and processing, providing configurations for high availability, load balancing, and connector customization. The article concludes by stressing the ease of adapting these connectors to specific data needs and encouraging further exploration through additional resources and talks.