July 2021 Summaries

The text discusses the importance of data durability in Apache Kafka® and how Confluent has been working on solving difficult and innovative data durability challenges. It highlights the role of Kafka as a system of record, which gives rise to new challenges and responsibilities for maintaining data safety and integrity. The focus is on durability auditing, which involves proactively detecting data integrity issues in Confluent Cloud. The text also mentions various scenarios where there could be lapses in durability and the lessons learned from managing Kafka clusters and trillions of processed messages over the years. It explains how Confluent performs extensive durability auditing and monitoring with real-time detection and alerting, using a source of truth for validation during sensitive operations. The text concludes by emphasizing the commitment of the Kafka Data Platform team at Confluent to ensuring data safety and integrity in their cloud service.

Apache Kafka is designed to support multi-tenancy in the cloud, but its traditional on-prem usage requires manual capacity planning and over-provisioning resources to protect against unplanned increases in demand. Confluent Cloud builds upon this by providing automated capacity planning and auto-scaling while supporting a pay-as-you-go model cost-efficiently. To achieve this, Confluent Cloud uses various features such as broker-wide bandwidth quotas, request quotas, and tenant-wide bandwidth quota configurations to ensure that tenants operate within their resource limits. The cloud-native solution also enables fair sharing of resources among tenants, proportional to their original quota allocation, while automatically detecting and adjusting for changes in demand. By leveraging these features, Confluent Cloud provides a scalable and cost-effective way to deploy multi-tenant Kafka clusters in the cloud.

This article discusses the optimization of Apache Kafka for Confluent Cloud. It highlights three guiding principles that have led to incremental optimizations, as well as architectural changes that have delivered large step-changes in performance. The author explains how workload simulation is used to understand the limits of their architecture and prioritize features or improvements on their roadmap. They also discuss the importance of reproducibility and monitoring for identifying and fixing issues faster. Finally, they highlight some key architectural changes that have made a significant difference to performance in Confluent Cloud, such as Tiered Storage and improved connection service times.

Confluent Cloud's architecture leverages Kubernetes and the Confluent Operator for Kafka to achieve a serverless experience, where customers can expand their Kafka clusters automatically without service disruption or user intervention. The control plane is responsible for managing capacity, provisioning new brokers, and rebalancing data across the cluster. Infinite Storage separates the concerns of data storage from processing, allowing each layer to scale independently and reducing migration times when balancing the load. Confluent's dual-tier architecture provides a truly elastic Kafka experience with disk buffering and separate paths for fresh and historical data, isolating latency-sensitive apps from those reading older data. The Confluent Cloud platform offers a scalable, elastic, and seamless serverless experience, aligning with the "serverless" vision where users focus on client application logic while resources magically arrive when needed and disappear when no longer required.

The shift in expectations for cloud-native data systems has led to the development of new capabilities that users now expect, such as elasticity, multi-tenancy, scalability, resiliency, cost-effectiveness, and global replication. Confluent Cloud's approach to building a high-performance cloud-native data system involves understanding user workloads well, optimizing for their needs, and aligning infrastructure constraints across the entire stack. The company has developed techniques such as Self Balancing Clusters, tiered storage, and performance optimization principles to deliver better performance on cost-effective infrastructure. By designing systems from the ground up to be cloud-native software-as-a-service offerings, Confluent Cloud provides a multi-tenant system that introduces economies of scale to the service, allowing for automatic and transparent response to new workloads and isolating tenants from noisy neighbors.

The latest release of ksqlDB, version 0.19.0, introduces a significant enhancement with the inclusion of foreign-key joins, a highly anticipated feature that allows for many-to-one relationships between tables, previously only achievable through workarounds. This update does not require new syntax and enables users to specify any left-hand table column in the join condition, facilitating more flexible data modeling without a co-partitioning requirement, while still supporting INNER or LEFT OUTER join semantics. Additionally, the release includes a community-contributed NULLIF function, which returns NULL when its arguments are equal, offering a utility for transforming specific values into NULL. These advancements are poised to broaden the use cases for ksqlDB, making data processing more versatile and efficient, while also building on the enhanced stateful processing capabilities introduced in Kafka Streams 3.5, which allow for storing multiple record versions per key.

This blog post focuses on the critical step of retrieving data for data science projects, specifically how to create data pipelines from Apache Kafka into RStudio using two methods: one involving MongoDB as an intermediary layer and the other directly consuming data with the rkafka package. By leveraging Python and Jupyter Notebooks for descriptive analytics and R for its statistical capabilities, the tutorial explains the setup process, including the use of Docker and docker-compose, to simulate a data-producing environment with a Kafka producer. The post outlines the pros and cons of each method; using MongoDB offers the benefit of data aggregation and ease of querying through MongoDB Compass, whereas directly consuming data via rkafka provides a simpler setup but with less flexibility in querying. The choice of method ultimately depends on project requirements and personal preferences. All relevant code is available on GitHub, and the post concludes by mentioning a future exploration of applying the defined models on real-time data.

The blog post elaborates on the complexities and solutions involved in implementing foreign-key joins in Kafka Streams, a distributed stream processing system. It highlights the limitations of primary-key joins, which were previously the only join type available, and the challenges faced in supporting foreign-key joins due to distributed data partitioning. The post details the journey from initial proposals to the final implementation, which required innovative approaches to data partitioning and message passing between distributed tasks. The solution involves using composite keys and a subscription/response message passing system to ensure efficient and scalable foreign-key joins while maintaining data integrity and performance. The implementation not only simplifies the application code but also supports real-time materialized views, paving the way for advanced use cases in event-driven applications and future integration with ksqlDB. The post underscores the importance of this feature for enhancing the capabilities of Kafka Streams, enabling more complex data processing tasks previously restricted to relational databases.