April 2022 Summaries

Confluent's data streaming platform enables real-time data movement across multiple cloud platforms, eliminating the need for point-to-point connections and providing a singular, global, and real-time data plane. This allows organizations to unlock valuable data from systems, applications, and data stores that run across any number of global environments and stream that data in real time to their cloud database. Confluent's Cluster Linking technology allows users to reach into on-premises Kafka clusters from Confluent Cloud, creating an easy-to-use bridge to bring data into Confluent Cloud in their cloud service provider. By moving to real-time streaming and pulling data from any environment across cloud and on prem, businesses can provide application teams with the data they need faster and deliver new features and capabilities to customers at scale. Confluent offers five pre-built, fully managed CDC connectors for popular databases such as Oracle, Microsoft SQL Server, and PostgreSQL, allowing source databases to stay online and operating throughout migration. Choosing a cloud-first approach opens up additional challenges that must be considered, but Confluent can help organizations realize TCO savings of nearly $2.58 million by providing a cost-effective solution for real-time data streaming and ETL pipelines into cloud-based systems.

The latest update to the Confluent CLI integrates all functionalities from the cloud-specific ccloud CLI into the existing confluent CLI, marking a significant step towards a unified platform that caters to both cloud and on-premises users. This update, part of the Confluent Platform 7.1 release, introduces a single CLI that allows seamless switching between cloud and on-prem deployments, offering features like multi-org support and improved command syntax for cloud users, while enhancing usability for on-prem users. An example workflow is provided for hybrid users who need to link and mirror clusters across Confluent Platform and Confluent Cloud, underscoring the CLI's ability to manage both environments efficiently. With Confluent CLI v2, the previous ccloud CLI will be deprecated, with its sunset scheduled for May 2022, and users are encouraged to update to enjoy the streamlined functionalities and new features. The update also includes enhanced security with OAuth support and integration capabilities with Amazon OpenSearch, reflecting Confluent's commitment to evolving its platform.

The Confluent Q2 ‘22 cloud bundle has been launched, offering new features designed to make securing data and connecting systems easier. This includes role-based access control (RBAC), which allows users to set permissions per user for key resources like production environments, sensitive clusters, and billing details. The enhanced RBAC enables granular yet intuitive controls to manage users and applications within a single cluster, providing better alignment and faster time to production. Additionally, Confluent has launched new connectors, including Oracle CDC Source Premium Connector, which democratizes access to Oracle CDC data with no operational overhead. Other features include Stream Processing Use Case Recipes, powered by ksqlDB, which packages the 25+ most popular real-world use cases validated by experts, and a new 99.99% uptime SLA for Confluent Cloud deployments. The bundle also includes improvements to ksqlDB, including access to Kafka record headers, JSON functions, and pull query limit clause. Furthermore, Confluent has expanded its metrics API with two new capabilities: Principal ID breakdown for organizational chargebacks and consumer lag metric for performance monitoring.

This blog post demonstrates how to build an RAG-enabled GenAI model using Confluent, Flink, and MongoDB. It outlines the need for integrating data from different sources and destinations in real-time, leveraging NoSQL database technology for agile development, reduced operational costs, and scalable operations. The authors discuss the benefits of using Confluent Cloud, Confluent for Kubernetes, and Apache Kafka to build a streaming data pipeline that facilitates real-time data transfer between an on-premises relational database and a document-oriented NoSQL database like MarkLogic. They provide step-by-step instructions on deploying a self-managed Oracle CDC connector and a MarkLogic Sink connector using Confluent for Kubernetes, leveraging the Control Center UI or CLI to configure the connectors. The post also highlights the importance of security, scalability, and ease of use in building end-to-end data pipelines consisting of both relational and NoSQL databases.

The text discusses a comprehensive course on Apache Kafka's internal workings, taught by Jun Rao, one of Kafka's original creators. It delves into various advanced topics such as data replication, consumer group protocols, transactional processing, topic compaction, and tiered storage, offering insights into Kafka's architecture and configuration options. The course also covers Kafka's transition from ZooKeeper to KRaft, the implications of replication and partitioning strategies, and features like the Idempotent Producer for maintaining ordering guarantees. Additionally, it highlights the release of Apache Kafka 3.8.0, which includes new features and improvements, and touches on a playful project using Kafka to settle the "tabs versus spaces" debate by producing and consuming vote counts with a new JavaScript client.

Confluent Platform 7.1 introduces significant enhancements aimed at improving data streaming capabilities across hybrid and multicloud environments, offering a comprehensive feature set that supports mission-critical use cases. Central to the release is Schema Linking, which simplifies maintaining consistent and trusted data streams by allowing real-time schema syncing across different clusters. The platform also expands storage options with Tiered Storage, enabling long-term data storage with reduced costs, and introduces Health+ for enhanced monitoring and intelligent alerting to prevent downtime. Additionally, ksqlDB 0.23 brings support for pull queries over streams, simplifying debugging processes for developers. The release continues to build on previous innovations, such as Cluster Linking and Confluent for Kubernetes, offering a more cloud-native experience and expanded support for multi-region clusters. Overall, Confluent Platform 7.1 strengthens its capacity to deliver a seamless, real-time data fabric, crucial for modern business operations.

Confluent has been working to seamlessly connect Apache Kafka topics to Google BigQuery, earning the Google Cloud Ready – BigQuery validation designation, which ensures its fully managed connector for Confluent Cloud meets functional and interoperability requirements. This collaboration with Google will continue to improve the integration between Confluent Cloud and BigQuery, making it easier for customers to analyze data from multiple environments. Confluent is also expanding its Microsoft Azure partnership, being named 2024 OSS on Azure Global Partner of the Year, reflecting its commitment to delivering outstanding open source-based applications and infrastructure solutions on Azure. This recognition comes as data streaming is becoming a critical foundation of modern business, with Confluent poised to play a key role in this shift.

Confluent and Microsoft are expanding their partnership to accelerate enterprises' journey to cloud data streaming on Azure, with joint technical, marketing, and sales investments to make it easier to build a modern data stack. Confluent's cloud-native service helps migrate and connect data to Azure services with pre-built connectors, making it easy for enterprises to start their first data in motion use case. The partnership aims to reduce the operational burden of managing Kafka and provides joint customers across industries with easy ways to move data to the cloud and access real-time data. With 175 ZB of data being generated by 2025, this new investment will help every enterprise learn about the future of data in motion and its impact on their business.

As digital experiences evolve to become more interactive and real-time, industries like gaming and betting are leveraging event-driven architectures to meet customer expectations for immediacy and reliability. This blog post outlines how Confluent Cloud and Ably can be integrated to create a robust real-time betting platform that processes and streams data at an elastic scale, ensuring seamless user experiences even during large events like the Super Bowl. Confluent Cloud provides a scalable backend through Apache Kafka, while Ably acts as an edge messaging platform, connecting the real-time data to end users via low-latency WebSockets. The Ably Kafka Connector facilitates smooth data flow between these systems, enabling real-time updates and interactions for betting scenarios, such as dynamically updated odds during live sports events. This integration supports the creation of highly scalable and reliable systems, extending beyond betting to various sectors requiring real-time data transfer and processing. The post emphasizes the ease of setting up and managing these serverless solutions, highlighting their potential to deliver low-latency, scalable digital experiences across industries.