October 2022 Summaries

We built an end-to-end motion detection and alerting system on top of Apache Kafka, ksqlDB, and MongoDB to demonstrate how easy it is to build IoT solutions by leveraging Confluent Cloud. We used a Raspberry Pi Zero W as the hardware due to its compact size, low cost, and low power consumption. The system uses motion detecting software to detect motion from camera feeds, which are then uploaded to AWS S3 and published to Kafka for processing. We chose Confluent Cloud for its scalability, availability, latency, and agility, and used Docker to containerize the software and services. The system also includes a UI built with React and Next.js for real-time monitoring and past event search, utilizing ksqlDB's push and pull queries. The solution showcases how Kafka and ksqlDB can be used to build scalable and efficient IoT systems, and demonstrates the possibilities of expanding such solutions to other IoT applications.

The text discusses an introductory course on using schemas and schema registries in data streaming systems. It covers topics such as configuring applications to connect with a Kafka cluster, working with Protobuf and Avro schemas, managing schemas, integrating client applications, understanding schema subjects, and ensuring schema compatibility. The course also includes hands-on exercises for practical application of the concepts learned. Additionally, it mentions the release of Apache Kafka 3.8.0 and highlights some of its new features and improvements.

An e-commerce GDPR opt-out workflow needed to enrich Kafka consent events with customer email addresses before Kafka Connect could reliably call a third-party retargeter’s deletion API. Using ksqlDB, the author initially applied a stream–table join between consent and account topics but found missing emails caused by incomplete source data, then used kcat to load a complete email dataset. Although the replacement data was correct, all joins returned null because kcat’s default librdkafka partitioner placed matching keys in different partitions than Java Kafka clients, breaking the co-partitioning required for the join; specifying the `murmur2_random` partitioner resolved this issue. A further failure revealed that stream–table joins are temporal: only incoming stream records trigger lookups, so table entries added afterward do not update earlier join results. Treating both sources as tables and using a table–table join allowed updates from either topic to produce enriched records, ultimately supplying complete consent and email information for the opt-out process.

ksqlDB 0.28.2 introduces Confluent Cloud and query-engine enhancements aimed at simplifying setup, improving operational control, and supporting larger streaming workloads. In Confluent Cloud, a topic import wizard can automatically create streams from Kafka topics with Schema Registry schemas, while a new CRUD REST API and Terraform support enable programmatic cluster management through infrastructure-as-code workflows. The release also offers faster provisioning for eligible small clusters, increases the maximum cluster size from 12 to 28 CSUs, and adds preview metrics for consumption, production, offsets, errors, and restarts alongside generally available storage and query saturation metrics. Core additions include EMIT FINAL for emitting one result after a window closes, using a disk-backed implementation intended to avoid memory risks from large aggregation states; PAUSE and RESUME commands for persistent queries; and fourteen trigonometric, inverse trigonometric, and hyperbolic scalar functions, with utilities for radians, degrees, and pi.

Data mesh is a contemporary approach to IT infrastructure that emphasizes agility, simplicity, and flexibility by integrating people, processes, and technology. This architectural mindset aims to reduce complexity and align data products with their owners, ensuring teams have the necessary data access and self-service capabilities. Key practices for successfully implementing a data mesh include prioritizing evolvability, establishing a hierarchical domain structure, managing complexity, refining ownership models, and defining default implementations. Additionally, it underscores the importance of thinking deeply about the temporal aspects of data, instituting feedback loops for continuous improvement, and viewing the data mesh as a dynamic process rather than a static technology. By focusing on interoperable data without relying on centralized control, organizations can effectively share data across teams, adapting the mesh to suit unique needs and evolving requirements.

Confluent’s inaugural Current 2022 conference, positioned as the next generation of Kafka Summit, emphasized the growing importance of data streaming as organizations move beyond static, data-at-rest architectures toward systems that reflect real-time business activity. CEO Jay Kreps and other speakers described streaming as a major technological shift, while examples from USPS, Pinterest, Walmart, NASA/JPL, and other organizations illustrated its use in large-scale operations, real-time inventory, scientific alerts, healthcare devices, and motorsport telemetry. Confluent announced Stream Designer to simplify streaming pipeline development, Stream Governance Advanced for enterprise governance and visibility, and Confluent for Startups, which provides eligible new companies with a year of free Confluent Cloud services. The event also introduced Data Streaming Awards recognizing organizations including NASA/JPL, Ademen, Pinterest, Walmart, Control, and LinkedIn for innovative implementations and community contributions.

Confluent has released new features in its real-time data streaming platform, including Stream Designer for building pipelines quickly and easily, an expanded suite of governance tools called Stream Governance Advanced, and Private Service Connect for Google Cloud. These updates aim to help businesses innovate faster with their real-time data streaming platforms while maintaining high levels of security and performance.

Stream Designer is a visual interface for building, testing, and deploying streaming data pipelines natively on Apache Kafka. It provides a graphical canvas for designing pipelines and translates the pipeline definitions into ksqlDB code under the hood. Stream Designer offers speed, simplicity, and flexibility, making it easier to build streaming data pipelines without requiring extensive knowledge of Kafka or Java-based Kafka Streams. The tool also provides pre-built connectors, stream processing use case recipes, and a built-in SQL editor for developers to easily reuse production-ready pipelines and collaborate with others in real-time. Stream Designer is now generally available on Confluent Cloud and can be used to build streaming data pipelines that support decoupling of data sources and sinks, continuous data processing, and triggering actions based on real-time events.

Apache Kafka 3.3 introduces KRaft mode as production ready for newly created clusters, replacing ZooKeeper-based metadata management to improve scalability, resilience, and deployment simplicity, while a planned 3.5 bridge release is intended to support migration from ZooKeeper mode. The release also improves KRaft upgrades, controller-quorum observability, metadata replication monitoring, replica safety during shutdown, and administrative access to cluster and disk information. Producer and consumer enhancements include a more balanced sticky partitioner, multi-group offset fetching, delegation tokens for other users, log-recovery metrics, and options supporting exactly-once offset queries. Kafka Streams gains finer-grained source and sink throughput metrics, topology pause and resume controls, simplified processing APIs, and faster permanent shutdown behavior, while Kafka Connect adds exactly-once support for source connectors through atomic record and offset writes and safeguards against zombie tasks.

Confluent Cloud is a fully managed data streaming platform that enables startups to build quick, easy, and cost-effective real-time data infrastructure, without the need for extensive expertise or infrastructure capacity. By leveraging Confluent's open-source and agnostic platform, startups can choose from various tools and experiment with different use cases, while maintaining flexibility in their data infrastructure. The new Confluent for Startups Program provides a streamlined experience, including access to Confluent's deep bench of Kafka and data streaming expertise, as well as advice on tackling any challenge or architectural decision. This program aims to help startups overcome the tradeoffs between building a DIY data streaming platform or managing a batch solution, allowing them to focus on their business logic and deliver product value more efficiently.