August 2020 Summaries

Confluent Cloud is a cloud-based event streaming platform that requires careful consideration of security measures when using sensitive data, and involves multiple layers of security controls designed to protect and secure user data. The company's security philosophy centers around layered security controls, including access management, authentication, encryption, logging, monitoring, vulnerability management, and bug bounty programs. Confluent Cloud builds on open source Apache Kafka, inheriting security capabilities like wire encryption, authentication, and access control lists, and adds new features such as single sign-on, bring your own key encryption, and upcoming Role-Based Access Control and audit logging. The company prioritizes transparency in its internal processes and security controls, providing a white paper to help users understand the security mechanisms of Confluent Cloud.

Kafka Summit 2020 was a significant virtual event, drawing over 35,000 registrations from 143 countries, showcasing the global reach and enthusiasm for Kafka-related technologies. Despite the challenges posed by the COVID era, the summit successfully recreated the excitement of in-person events through online sessions, interactive discussions, and keynote speeches. Highlights included presentations on streaming data pipelines, reactive Apache Kafka, and multi-region Kafka Streams applications, with notable talks by industry experts like Jay Kreps and Sam Newman. Kreps discussed the fusion of cloud-native systems with event streaming platforms, emphasizing Project Metamorphosis, while Newman argued for building scalable distributed applications using a distributed log like Kafka. The event underscored Kafka's growth and maturity as a platform, highlighting its evolution from its initial capabilities in 2016 to its current focus on features like cloud region operation and platform-native SQL stream processing. The event demonstrated Kafka's continued relevance and development in the tech industry, leaving attendees with a sense of optimism and anticipation for future innovations.

Cluster Linking is a feature in Confluent Cloud that enables the creation of a single "virtual Kafka cluster" that spans globally, allowing event streams to travel seamlessly between different clusters around the world, accurately preserving the exact order, partitioning, and offsets of the original data streams. This feature addresses the challenge of running Apache Kafka in a globalized environment, providing a zero-bottleneck, highly available architecture and allowing for buffering indefinitely to support high latency and unreliable networks. Cluster Linking can be used to create hybrid cloud solutions, where critical datasets are made available as event streams, and can also be used in multi-cloud strategies, unifying the movement of data among thousands of microservices across different cloud providers. The feature is currently in early access on selected regions of Confluent Cloud and will be rolled out globally in the near future.

Kafka Summit 2020, the first-ever virtual edition, attracted over 33,000 registrants and featured a range of engaging sessions and keynotes, including an opening address by Gwen Shapira, who discussed Apache Kafka's growth and future innovations such as Kafka Tiered Storage and the replacement of ZooKeeper with a Self-Managed Metadata Quorum. Jun Rao, co-creator of Apache Kafka, highlighted how various industries like online travel, global banking, and retail are adopting Kafka, often starting with small use cases and transitioning to cloud-native data ecosystems. The event also included practical sessions, such as a deep dive into messaging systems by Ben Stopford and Michael Noll, and other notable presentations from Lyft, Confluent, and Microsoft, providing valuable insights into using Kafka for real-time event streaming. Day one was complemented with entertaining activities and set the stage for more discussions in the following days, with keynotes from Jay Kreps and Sam Newman anticipated.

This article discusses a comprehensive performance test comparing Apache Kafka, RabbitMQ, and Pulsar on the latest cloud hardware. The results show that Kafka delivers the best throughput while providing the lowest end-to-end latencies up to the p99.9th percentile. At lower throughputs, RabbitMQ delivers messages with significantly lower latency than Kafka and Pulsar but only when lightly loaded. The cost tends to be an inverse function of performance, making Kafka the most efficient system in terms of value per byte written due to its efficient design.

Bolt, a leading European on-demand transportation platform, has transitioned from batch data workloads to stream processing by adopting Change Data Capture (CDC) and implementing the Confluent Platform for event streaming. This shift enables Bolt to replicate data from operational databases to a data warehouse more efficiently, addressing previous limitations of periodic polling by using log-based CDC, which minimizes load on source databases. Bolt utilizes Debezium connectors to translate database changes into JSON, which are then streamed through Apache Kafka, ensuring strict event ordering. Kafka Connect orchestrates data movement between systems, while Schema Registry optimizes storage by managing Avro schemas. The system's reliability is enhanced by high-availability setups, allowing for scalable data replication and real-time stream processing. Bolt also uses ksqlDB and Kafka Streams for flexible data transformation, promoting cross-service communication and event processing. This infrastructure supports Bolt's goal of leveraging event streaming benefits and migrating from batch to real-time data processing.

As businesses navigate the challenges posed by the pandemic, the shift toward digital transformation, particularly event streaming, is widening the digital divide between companies. Research by Enterprise Management Associates highlights that the pandemic tested digital business models, emphasizing cost-saving initiatives where event streaming plays a crucial role. Confluent Cloud emerges as a significant enabler, offering cost savings and efficiency by providing a managed, scalable platform for event streaming, exemplified by Walmart's agile response to changes in retail demands. The use of event streaming and cloud-centric strategies allows businesses to adapt quickly to shifting market conditions, making it a valuable tool for organizations of all sizes. The EMA white paper underscores the importance of real-time event streaming for innovation and efficiency, further supported by technologies like IoT and AI/ML, enhancing the resilience and adaptability of companies in a rapidly changing environment.

The TopologyTestDriver tool is essential for building Kafka Streams applications, allowing developers to write fast and deterministic tests that cover serialization/deserialization issues automatically. However, it has limitations, particularly in handling asynchronous processing and state stores. To overcome these challenges, developers can use the Awaitility library to simplify asynchronous testing and reduce test running time. It's recommended to utilize TopologyTestDriver for unit tests and reserve TestContainers-based tests for more complex scenarios where additional guarantees are required.

Kafka Summit 2020 is an upcoming virtual event designed to explore the future of event streaming and engage the Apache Kafka community in a series of educational and networking opportunities. Attendees are encouraged to prepare by taking advantage of free training sessions, including a fundamentals tutorial and certification bootcamp. The summit offers over 60 sessions and lightning talks, with a platform that helps personalize schedules and suggests relevant topics. Keynotes will feature prominent technology leaders sharing their insights, and there are opportunities for participants to engage in Birds of Feather sessions, expert Q&As, and social networking events such as the Virtual Confluent Girl Geek Networking Event. Participants can also earn swag by completing tasks, and they are encouraged to take breaks to avoid fatigue. All sessions will be available on-demand post-event, ensuring attendees do not miss any critical content.

Confluent reaffirms its dedication to supporting customers, employees, and communities during the COVID-19 pandemic by offering 12 months of Confluent Cloud services to eligible nonprofits and NGOs involved in COVID-19 relief efforts. This initiative aims to enable these organizations to leverage digital transformation and event streaming technologies for real-time applications such as inventory management and resource tracking. Confluent's commitment extends to its workforce, providing options to work from home and introducing programs to support employee well-being and minimize health risks. Additionally, the company emphasizes community engagement by encouraging volunteerism and supporting philanthropic efforts, while also maintaining a focus on customer satisfaction and innovation. Confluent's broader efforts include transforming the Kafka Summit into a virtual event for community safety and achieving FedRAMP Ready status for its government data streaming services, underscoring its commitment to security and industry standards.

The blog post explores the intricacies of implementing a multi-threaded consumer model for Apache Kafka, highlighting both the potential benefits and challenges compared to the commonly used thread per consumer model. By delving into the mechanics of Kafka's consumer internals, it discusses how records can be processed in parallel without compromising processing order guarantees and at-least-once delivery semantics. It emphasizes the importance of understanding Kafka consumer behavior, particularly regarding poll loops, offset management, and consumer group rebalancing. The post provides insights into using Java's thread pool for efficient task execution and stresses the need for manual offset commits to maintain data integrity during parallel processing. Additionally, it offers practical coding examples and links to further resources, including a GitHub repository and a Kafka Summit talk, for those interested in exploring the described multi-threaded model in more detail.

Operating cloud infrastructure efficiently is crucial due to the direct cost implications of processing time, as highlighted by an incident involving ksqlDB on Confluent Cloud. A node relocation led to a failure in query processing, revealing complexities in how ksqlDB manages state stores and schemas. The issue arose from a bug in Kafka Streams' optimization, which mistakenly used a changelog topic name instead of the source topic name during serialization, resulting in schema ID mismatches that prevented proper data recovery. The solution involved registering the correct schema under the appropriate subject to enable successful deserialization. Moving forward, ksqlDB plans to improve its integration with Confluent Schema Registry to enhance schema management flexibility and prevent similar incidents.

This blog post discusses the integration of databases that span across both on-premises and cloud deployments using Kafka Connect and Confluent Cloud. It highlights how this approach provides a more seamless way to build a flexible, future-proof multi-cloud architecture with a single source of truth for data. The post also demonstrates an automatable workflow to integrate a cloud database and on-prem database into an ETL pipeline using Kafka Connect API. Additionally, it introduces the general availability of Confluent Platform 7.7 and its latest key features such as enhanced security with OAuth support, Confluent Platform for Apache Flink® (LA), a new Connector, and more.

Confluent Cloud, a managed event streaming service powered by Apache Kafka, now supports AWS PrivateLink for secure network connectivity, enhancing its existing options like VPC peering and AWS Transit Gateway. This move enables users to establish one-way secure connections from their AWS VPCs to Confluent Cloud Dedicated clusters without the complexity of IP address coordination required by VPC peering. PrivateLink offers a streamlined approach to secure connectivity, allowing enterprises to minimize network attack vectors and protect resources. While VPC peering allows mutual network access, PrivateLink ensures a one-way connection, reducing the network's surface area. Confluent has updated its infrastructure to facilitate the self-service setup of PrivateLink endpoints, making it easier for customers to quickly deploy and connect clusters securely. The announcement coincides with the release of Confluent Platform 7.7, which includes features like OAuth support and integration with Apache Flink, alongside a new connector and OpenSearch Ingestion for Amazon OpenSearch integration.

Confluent Cloud is now available on major cloud infrastructure providers such as AWS, Microsoft Azure, and Google Cloud, allowing organizations to easily adopt and scale Apache Kafka for their business needs without having to manage the complexity of running it themselves. The platform provides a seamless experience, unified billing, portability across multiple clouds, and support for existing agreements with cloud infrastructure providers. With Confluent Cloud, users can access features such as event streaming, real-time data processing, and scalable storage, all through a simple sign-up process that leverages their existing billing accounts or enterprise agreements. This makes it easier for organizations to leverage the power of Apache Kafka without being bogged down by operational aspects of its deployment.

ksqlDB 0.11.0 introduces significant enhancements aimed at improving production stability, which is crucial as its usage in mission-critical applications grows. This release addresses several bug fixes, such as resolving issues with stranded transient queries and schema management, and introduces features like HTTP client caching to enhance pull query performance across multi-node clusters. The update also fixes a bug related to windowed table retention in Apache Kafka® topics, ensuring proper expiry of outdated windows. The Java client sees expanded functionality, allowing for the creation and management of streams, tables, and persistent queries through new methods, while also enabling users to list streams, tables, topics, and queries. Additionally, pull queries now support filtering using both WINDOWSTART and WINDOWEND keywords, offering more flexibility in querying session windows. Users are encouraged to explore these features via the standalone distribution or Confluent Cloud and engage with the community through various channels.

Tencent, a leading internet-based platform company, has significantly advanced its use of Apache Kafka to enhance the scalability and fault tolerance of its data infrastructure supporting its various digital products, including WeChat. The Platform and Content Group (PCG) at Tencent has developed a federated Kafka solution that incorporates a proxy layer to manage multiple Kafka clusters, thereby overcoming limitations in scalability, customization, and service level agreements inherent in using a single Kafka cluster. This setup allows Tencent to handle trillions of messages daily across hundreds of clusters, with improvements in real-time analytics and feature engineering. Despite challenges such as message ordering issues and the need for frequent updates to the proxy broker interface, Tencent's approach aligns with ongoing developments in the Kafka community, particularly around Tiered Storage and the removal of ZooKeeper. The company's successful implementation demonstrates both its commitment to pushing Kafka's capabilities and its potential contributions to the broader Kafka ecosystem, as it continues to explore further innovation and integration.