August 2022 Summaries

The Apache Kafka Raft (KRaft) protocol is a consensus protocol used in Apache Kafka to manage metadata, simplifying the architecture by consolidating responsibility into Kafka itself, rather than relying on ZooKeeper. This allows for improved scalability and reduced bottlenecks as the cluster grows. KRaft mode is available in the Apache Kafka 3.1 release but is not yet ready for production environments. Resources are available to learn more about KRaft, including a blog post by Guozhang Wang, which explores the rationale behind its implementation, and a module in the Confluent Developer site's Apache Kafka Internal Architecture course, led by Jun Rao, one of the original co-creators of Apache Kafka. Configuration changes related to removing ZooKeeper from Kafka are also discussed, including updates to clients and services, Schema Registry, and administrative tools. A guide is available to get started with Apache Kafka in KRaft mode, while a GitHub repo provides a Gitpod workspace definition for running an Apache Kafka KRaft mode cluster.

The text details a project undertaken during the Confluent Hackathon '22, where the author explored using Apache Kafka and other technologies to monitor wildlife. Initially testing the concept by tracking animals in his backyard with a Raspberry Pi, camera, and TensorFlow Lite for real-time object classification, he expanded the project by integrating a webcam feed from a local zoo to capture more exotic animals. Data collected was sent to a Kafka cluster and processed using ksqlDB to analyze wildlife trends, and a dashboard was created using Kibana to visualize these trends. Additionally, a Telegram bot was set up to alert the author of rare animal sightings, such as teddy bears detected by the model. This project highlighted the potential of combining streaming platforms, machine learning, and open-source tools to innovate wildlife monitoring and address real-world challenges.

In the context of evolving customer expectations for fast and seamless transactions, businesses increasingly rely on stream processing to build real-time applications that continuously capture, store, and process data streams. Confluent addresses the complexity of traditional stream processing architectures, which often involve convoluted blends of separate solutions, by offering ksqlDB. This stream processor simplifies the architecture by using a declarative approach with lightweight SQL syntax, enabling developers to efficiently unlock the value of real-time data. ksqlDB provides several benefits, including the ability to process data in motion, streamline architecture by reducing infrastructure maintenance, and facilitate the creation of real-time applications with simplified SQL. It supports various technical use cases such as streaming data pipelines, materialized caches, and event-driven microservices, effectively managing real-time insights and customer experiences. Furthermore, ksqlDB is part of Confluent's broader platform, which includes a rich ecosystem of pre-built connectors and offers robust compatibility, reliability, and security for enterprises aiming to leverage data in motion.

Serverless stream processing with Apache Kafka is a powerful yet often underutilized field that can be leveraged for building GenAI apps faster using Microsoft's Azure Functions, ksqlDB, and Confluent's sink connector. These tools provide a powerful and easy-to-use set of tools that can handle even the most complex workloads. By leveraging Kafka Connect to trigger Azure Functions, developers can create stateless event stream processing applications or use ksqlDB for long-running stateful compute. The integration discussed in this blog uses ksqlDB queries to complete the application, relying on its high scalability and serverless capabilities. Microsoft's Azure Functions provide a Function as a Service (FaaS) component that accelerates serverless application development, making it easy to process and react to events. The Azure Functions Kafka extension enables customers to detect and respond to real-time messages streaming into Kafka topics or write to a Kafka topic through the output binding. By using the Azure Functions Kafka extension, developers can create stateless applications with high throughput and scalability, while also providing at-most-once processing guarantees. The integration of ksqlDB and Azure Functions offers a powerful serverless one-two punch that enables developers to build efficient event streaming applications on Confluent Cloud.

The author built a system to monitor their houseplants' moisture levels using Apache Kafka, Confluent Cloud, Flink, and MongoDB. They created a Raspberry Pi-based system that collects data from capacitive sensors, which are then sent to Kafka topics for processing. The author used ksqlDB to process the data, creating a stream of enriched readings with metadata about each plant. They then created a new table, `houseplant_low_readings`, to store alerts when a plant's moisture level falls below a certain threshold. Finally, they used Confluent Cloud's HTTP Sink Connector to send these alerts directly to their phone via a Telegram bot. The system was successful in sending alerts only when necessary, reducing the need for manual checks on the plants.