August 2022 Summaries

The Apache Kafka Raft (KRaft) protocol is a consensus protocol used in Apache Kafka to manage metadata, simplifying the architecture by consolidating responsibility into Kafka itself, rather than relying on ZooKeeper. This allows for improved scalability and reduced bottlenecks as the cluster grows. KRaft mode is available in the Apache Kafka 3.1 release but is not yet ready for production environments. Resources are available to learn more about KRaft, including a blog post by Guozhang Wang, which explores the rationale behind its implementation, and a module in the Confluent Developer site's Apache Kafka Internal Architecture course, led by Jun Rao, one of the original co-creators of Apache Kafka. Configuration changes related to removing ZooKeeper from Kafka are also discussed, including updates to clients and services, Schema Registry, and administrative tools. A guide is available to get started with Apache Kafka in KRaft mode, while a GitHub repo provides a Gitpod workspace definition for running an Apache Kafka KRaft mode cluster.

Confluent’s free Cloud Networking course is designed for both networking specialists and general practitioners, covering foundational concepts such as IP addresses, DNS, CIDR ranges, and cloud virtual networks across AWS, Azure, and Google Cloud. It helps learners assess what services need connectivity, where connections originate, and whether private networking is required, then explains Confluent Cloud options including secure public endpoints, VPC/VNet peering, AWS Transit Gateway, and AWS/Azure PrivateLink, with Google Cloud Private Service Connect noted as an early-access alternative. The course compares these approaches based on security, implementation complexity, address-space requirements, scalability, compliance needs, and cost, while hands-on labs demonstrate configuring clusters, generating and consuming data, and connecting AWS VPC resources. It concludes with a review of connectivity tradeoffs and an overview of Confluent Cloud’s control-plane and data-plane components.

Confluent reports that G2 named it the leader in the event stream processing category, based on verified user reviews assessing areas such as performance, reliability, ease of setup, integrations, and data modeling. G2’s category covers technologies that process data changes in real time, supporting uses including billing, fulfillment, fraud detection, and analysis of data over time, in contrast to historical batch processing. According to the post, Confluent received an overall G2 score of 85 from 47 reviews, with a customer satisfaction score of 75 and market presence score of 94, while 91% of verified users awarded it four or five stars. Reviewers highlighted its managed Kafka capabilities, cloud service features, support, and suitability for asynchronous communication and data processing.

A newly appointed VP of Engineering for India describes joining Confluent because of its people, remote-first approach, and the potential of Kafka to make distributed data available in real time. The leader is building the Kafka Connect product organization in India, focusing on simplifying data ingestion from diverse sources, enabling customers to create and run connectors in the cloud, and operating reliably at large scale in a multi-tenant environment. With most of the Connect team based in Bangalore and others working remotely across India, the organization aims to develop a shared vision centered on customer proximity, experimentation, decentralized decision-making, continuous improvement, and learning from systemic failures rather than assigning individual blame. The author also identifies opportunities to improve operational focus as Confluent grows and expresses a goal of making the company a preferred employer for engineers in India through both technical innovation and workplace culture.

Apache Kafka supports critical real-time applications but has limited security enabled by default, making protection of data in transit and at rest essential. A free Confluent Developer course introduces Kafka security planning, emphasizing corporate policies, regulatory obligations, deployment environments, performance trade-offs, encryption, and auditability. It covers authentication through principals for clients, brokers, and ZooKeeper; authorization through ACLs and, for larger organizations, external role or group systems with custom authorizers; and SSL-based encryption configurations for broker and client traffic. The course also addresses securing ZooKeeper metadata with SSL or SASL and restrictive ACLs, as well as using Log4j-based audit logging to investigate attacks, monitor unauthorized activity, assess impact, and meet compliance requirements.

The text details a project undertaken during the Confluent Hackathon '22, where the author explored using Apache Kafka and other technologies to monitor wildlife. Initially testing the concept by tracking animals in his backyard with a Raspberry Pi, camera, and TensorFlow Lite for real-time object classification, he expanded the project by integrating a webcam feed from a local zoo to capture more exotic animals. Data collected was sent to a Kafka cluster and processed using ksqlDB to analyze wildlife trends, and a dashboard was created using Kibana to visualize these trends. Additionally, a Telegram bot was set up to alert the author of rare animal sightings, such as teddy bears detected by the model. This project highlighted the potential of combining streaming platforms, machine learning, and open-source tools to innovate wildlife monitoring and address real-world challenges.

A hands-on Apache Kafka and Confluent Cloud tutorial extends a houseplant monitoring system by mirroring existing Raspberry Pi soil readings to a separate cluster through Cluster Linking, which replicates topic data, offsets, metadata, and optionally schemas while requiring source-cluster credentials and ACL permissions. It then collects San Francisco weather measurements from the OpenWeather API with a Python producer, serializes humidity, pressure, and temperature records, and publishes them to a Kafka topic at 15-minute intervals. Using ksqlDB, the project creates streams from the mirrored plant and new weather topics, adds explicit timestamps, and performs a time-based inner join to generate enriched records. A Python consumer retrieves and deserializes the joined data into a Pandas data set for per-plant batch analysis, including correlation, linear regression, and ANOVA tests. The sample results indicate only weak relationships between weather measures and soil moisture, with the tested regression not statistically significant at the 5% threshold, while illustrating Kafka’s capabilities for cross-cluster data sharing, real-time stream processing, and downstream analytics.

In the context of evolving customer expectations for fast and seamless transactions, businesses increasingly rely on stream processing to build real-time applications that continuously capture, store, and process data streams. Confluent addresses the complexity of traditional stream processing architectures, which often involve convoluted blends of separate solutions, by offering ksqlDB. This stream processor simplifies the architecture by using a declarative approach with lightweight SQL syntax, enabling developers to efficiently unlock the value of real-time data. ksqlDB provides several benefits, including the ability to process data in motion, streamline architecture by reducing infrastructure maintenance, and facilitate the creation of real-time applications with simplified SQL. It supports various technical use cases such as streaming data pipelines, materialized caches, and event-driven microservices, effectively managing real-time insights and customer experiences. Furthermore, ksqlDB is part of Confluent's broader platform, which includes a rich ecosystem of pre-built connectors and offers robust compatibility, reliability, and security for enterprises aiming to leverage data in motion.

Serverless stream processing with Apache Kafka is a powerful yet often underutilized field that can be leveraged for building GenAI apps faster using Microsoft's Azure Functions, ksqlDB, and Confluent's sink connector. These tools provide a powerful and easy-to-use set of tools that can handle even the most complex workloads. By leveraging Kafka Connect to trigger Azure Functions, developers can create stateless event stream processing applications or use ksqlDB for long-running stateful compute. The integration discussed in this blog uses ksqlDB queries to complete the application, relying on its high scalability and serverless capabilities. Microsoft's Azure Functions provide a Function as a Service (FaaS) component that accelerates serverless application development, making it easy to process and react to events. The Azure Functions Kafka extension enables customers to detect and respond to real-time messages streaming into Kafka topics or write to a Kafka topic through the output binding. By using the Azure Functions Kafka extension, developers can create stateless applications with high throughput and scalability, while also providing at-most-once processing guarantees. The integration of ksqlDB and Azure Functions offers a powerful serverless one-two punch that enables developers to build efficient event streaming applications on Confluent Cloud.

Cloud database modernization is increasingly important as organizations move workloads from on-premises systems to cloud platforms for lower costs, scalability, agility, and real-time data use, while self-managed databases can be rigid, costly, and difficult to scale. The passage presents Confluent’s three-phase approach of Connect, Optimize, and Modernize: managed Kafka Connect source and sink connectors integrate existing systems with Kafka and cloud destinations; ksqlDB enables teams to build SQL-based real-time stream-processing applications with features such as joins, aggregations, windowing, fault tolerance, and exactly-once processing; and sink connectors plus single message transformations deliver customized event data to cloud systems such as Azure Synapse and Azure Cosmos DB. Confluent positions its managed data-streaming platform as a centralized alternative to complex point-to-point integrations, claiming it reduces operational burden, development risk, and total cost of ownership while supporting real-time applications and analytics across an organization.

The author built a system to monitor their houseplants' moisture levels using Apache Kafka, Confluent Cloud, Flink, and MongoDB. They created a Raspberry Pi-based system that collects data from capacitive sensors, which are then sent to Kafka topics for processing. The author used ksqlDB to process the data, creating a stream of enriched readings with metadata about each plant. They then created a new table, `houseplant_low_readings`, to store alerts when a plant's moisture level falls below a certain threshold. Finally, they used Confluent Cloud's HTTP Sink Connector to send these alerts directly to their phone via a Telegram bot. The system was successful in sending alerts only when necessary, reducing the need for manual checks on the plants.