May 2021 Summaries

Confluent's connectors portfolio plays a critical role in liberating siloed data from on-premises legacy technologies, enabling companies to build modern cloud-based applications and accelerate their efforts to modernize their data infrastructure. With over 120 expert-built and tested connectors, Confluent helps customers easily transition to new cloud-native ecosystems and applications, unlocking data and divesting from expensive on-premises vendors. The company's goal is to boost the productivity of Apache Kafka developers by delivering capabilities that help them spend more time building real-time applications that drive business forward. By providing a robust portfolio of connectors across both legacy and cloud-based applications and ecosystems, Confluent enables companies to modernize their data infrastructure cost-effectively, while de-risking their business and freeing their developers to work on higher-value activities.

The blog post explores methods for handling errors and implementing retry mechanisms in Apache Kafka applications, which operate in distributed environments where failures are common. It outlines various patterns for managing event streaming processes, emphasizing the importance of maintaining event order in scenarios such as change-data-capture streams. The post describes a specific use case involving an application that consumes, transforms, and produces events, and discusses the complexities of managing event order and retries when dependent conditions, like missing data, arise. A retry topic is introduced to handle such conditions, although this may disrupt the original event order, which can be critical in applications like inventory management. To address this, the post suggests tracking events using a local in-memory structure to ensure events for specific items are processed in sequence. Additionally, the use of tombstone events in a redirect topic aids in managing successful retries and maintaining system consistency. The article concludes by highlighting the release of Apache Kafka 3.8.0, featuring new developments, and invites readers to explore further resources through Confluent Developer.

Confluent Cloud has introduced several new features to enhance security and compliance for Apache Kafka deployments, including Cluster Role-Based Access Control (RBAC) and Audit Logs, designed to streamline data management and improve access control. Cluster RBAC allows setting role-based permissions, ensuring users have access only to necessary resources, while Audit Logs provide real-time tracking of user interactions with data, helping identify anomalies and potential threats. To further secure data, Confluent supports Bring Your Own Key (BYOK) encryption for data at rest on major cloud services and offers private networking options like VPC/VNet peering, AWS Transit Gateway, and Private Link for secure data transit. These features aim to reduce the complexity and time spent on managing security controls, allowing businesses to focus on creating impactful customer experiences. The blog also introduces the general availability of Confluent Platform 7.7, highlighting new features like OAuth support and integrations with tools such as Apache Flink and Amazon OpenSearch.

Confluent for Kubernetes enables customers to realize the benefits of Confluent Platform with additional control and customization available from self-managing clusters on their private infrastructure. This provides a complete, declarative API-driven experience for deploying and managing Confluent Platform as a cloud-native system. By using Confluent for Kubernetes, users can achieve simplicity, flexibility, and efficiency in managing Kafka-related infrastructure operations without the headaches of complex infrastructure management. The platform offers high-level declarative APIs to deploy and manage Confluent services, including Kafka, ZooKeeper, Confluent Schema Registry, Kafka Connect, ksqlDB, and Confluent Control Center. This allows users to focus on higher-value activities that drive business value rather than managing low-level infrastructure. Confluent for Kubernetes also provides a GitOps approach, enabling users to express the state of their infrastructure and application state as code, leading to consistency across organizations.

Confluent has announced the general availability of MongoDB Atlas Source and Sink Connectors within its Confluent Cloud, allowing users to easily link Apache Kafka® with MongoDB for real-time data streaming. The fully managed connectors eliminate the need for custom integration development and reduce operational burden when connecting data between Kafka and MongoDB across major cloud providers like AWS, Microsoft Azure, and Google Cloud. These connectors enable seamless integration of modern applications built on MongoDB Atlas with Apache Kafka®, simplifying real-time event streaming data processing.