September 2021 Summaries

Apache Kafka's operational burden can quickly become a limiting factor for adoption and developer agility due to its distributed architecture. To address this, cloud-native, fully managed Apache Kafka as a service like Confluent is essential. Confluent offers a serverless experience with self-serve provisioning, elastic scaling, and usage-based billing, protecting data using industry-standard security features and providing an enterprise-grade uptime SLA. Additionally, Confluent provides fully managed components such as Schema Registry, connectors to popular cloud services like Amazon S3 and Redshift, and ksqlDB, enabling the harnessing of real-time events without operational burden. Confluent integrates seamlessly with AWS Lambda, allowing for a serverless architecture where streaming data from Kafka topics is triggered by AWS Lambda functions, processing data downstream into Amazon DynamoDB tables. With Confluent Cloud, customers can easily deploy and manage their Kafka clusters, as well as access various services such as Data Integration, Connectors, and ksqlDB, to focus on building applications instead of managing infrastructure. The integration with Confluent Cloud enables a real-time event streaming solution that can be tried out using AWS Marketplace, providing an additional $60 credit for Confluent Cloud usage.

The latest release of ksqlDB 0.21.0 includes significant upgrades to its foreign-key joins, the introduction of the BYTES data type, and a new ARRAY_CONCAT function. With these enhancements, developers can now use arbitrary expressions to define foreign keys, handle binary data, and concatenate arrays in a more flexible way. The new features are now available in Confluent Cloud and can be accessed via the standalone distribution or with Confluent. Additionally, ksqlDB is part of a broader effort to enhance stateful processing capabilities through versioned key-value stores, which allow users to store multiple record versions per key.

Confluent provides a cloud-native messaging infrastructure that enables businesses to modernize their existing messaging architecture in incremental steps. Confluent's platform offers standardized connectivity, metadata management, and stream processing capabilities through its Kafka Connect framework and ksqlDB database. The platform also includes schema registration, governance, and data lineage capabilities to ensure the context of the data travels with it. By integrating Confluent with messaging middleware, businesses can unlock new use cases such as real-time customer applications and cloud-based applications. Confluent's solution for messaging modernization enables incremental adoption, allowing businesses to keep their existing messaging middleware at the center of their architecture while leveraging stream processing capabilities. The platform also includes a data in motion solution that allows businesses to send aggregate data to any connected destination, including third-party systems.

Apache Kafka 3.0 introduces significant updates including new features, breaking API changes, and advancements in the KRaft consensus mechanism, set to replace Apache ZooKeeper. While KRaft is not yet recommended for production, it now supports exactly-once semantics and partition reassignment. The release also brings improved delivery guarantees by default in the Kafka producer and enhanced capabilities in Kafka Connect and Kafka Streams, such as task restart enhancements and new API methods for offset management. Notable deprecations include support for Java 8 and Scala 2.12, as well as older message formats, with a transition towards newer implementations for features like Exactly Once Semantics (EOS). The update includes a shift in default settings, requiring users to configure SerDes explicitly and aligning the replication factor with broker defaults. Moreover, the release emphasizes a streamlined migration from ZooKeeper to KRaft and a focus on MirrorMaker 2, allowing users to configure internal topics for offset storage separately from the source Kafka cluster. These enhancements are part of a broader effort to improve efficiency, flexibility, and future readiness of the Kafka platform.

The GraphQL schema for this project is created manually and fits the client use case. The types are declared in Avro, and the Confluent Schema Registry is used to register the schema. The main advantage of this approach is that it's very easy to set up, especially if you're already using ksqlDB. It uses a setup with Docker Compose to bring up a data cluster, create some topics and schemas, and define a table and some streams in ksqlDB. The GraphQL endpoint spins up a GraphQL Playground to easily interact with the GraphQL endpoint. Queries can be issued by sending a query message to the Projector, which awaits a response from the Projector. Mutations are sent to the command handler, and subscriptions can be used to consume messages from Kafka. The main advantage of this approach is that it's easy to set up, especially if you're already using ksqlDB.

Confluent is launching its Stream Governance suite, which enables businesses to manage and govern data in motion across their organizations. The suite aims to strike a balance between data guardians and users, allowing companies to find harmony between protecting and democratizing access to data. Confluent Cloud's Stream Governance features include schema management, security, and business metadata tagging, enabling self-service data discovery and lineage tracking. This allows businesses to harness the full value of their real-time data while maintaining compliance with regulations. The suite is designed to be scalable, cloud-native, and interoperable across environments, aiming to unlock a powerful self-service platform for data in motion.

This webinar explores how to build Real-time Anomaly Detection (RAG) enabled GenAI with Confluent, Flink, and MongoDB. It discusses the importance of online machine learning for real-time fault detection in IoT sensors and the benefits of leveraging Apache Kafka to drive cutting-edge machine learning solutions. The presentation showcases SymetryML, a streaming machine learning software that extracts statistical information from new data tuples and builds predictive models or anomaly detection models. With Confluent and Kafka, users can create predictive models that continuously learn on the fly, leading to more efficient and effective solutions. The webinar also highlights the potential of real-time machine learning in various industries, including telecoms and finance.

Confluent has partnered with VMware Tanzu to provide a complete, cloud-native experience for application developers and platform operators, enabling them to set data in motion everywhere they choose to run VMware Tanzu. Confluent for Kubernetes provides a cloud-native experience but relies on an underlying Kubernetes API and runtime, whereas VMware Tanzu offers a consistent enterprise-ready Kubernetes runtime that streamlines operations. The two companies have jointly validated Confluent for Kubernetes atop VMware Tanzu Kubernetes Grid, ensuring reliable deployment and operation of the intelligent automation built into Confluent for Kubernetes. By leveraging Confluent for Kubernetes with VMware Tanzu Kubernetes Grid, platform operators can deploy Confluent to multiple Kubernetes clusters and manage their clusters at scale. Developers can also build applications that harness data in motion using Spring, running them alongside Confluent for Kubernetes on a common runtime for maximum efficiency. The partnership aims to make it easier to procure, deploy, and manage Confluent for Kubernetes as organizations adapt to real-time, event-driven world.

Apache Kafka serves as a crucial data infrastructure component for modern enterprises, acting as a distributed log system that efficiently manages data flow. Kafka Connect, a key feature within the Kafka ecosystem, simplifies data integration by providing a pluggable, declarative framework that connects various data sources and sinks to Kafka, allowing for seamless event streaming and transformation. Confluent offers both fully managed and self-managed connectors, enabling users to configure, monitor, and scale their data integration efforts. With features such as change data capture, Kafka Connect enhances real-time data analytics and message brokering, ensuring flexibility and fault tolerance in data processing. Users can experiment with Kafka Connect using Confluent Cloud, which offers a range of tools and resources for hands-on learning, including courses, webinars, and API documentation. Additionally, the Kafka Connect REST API facilitates cluster management and error handling through various methods, such as fail-fast or dead letter queues, enhancing the reliability and transparency of data pipelines.