June 2021 Summaries

Confluent has been recognized with the Databricks ISV Momentum Award, highlighting the significant integration progress between Confluent's data streaming platform and Databricks since their partnership began in 2019. This collaboration aims to address the challenges enterprises face with real-time data analytics, scalability, and legacy IT architectures by providing a comprehensive solution that enhances scalability, performance, and data reliability. An example of this integration's impact is illustrated by a financial services customer who overcame the limitations of legacy systems to implement real-time fraud detection, leading to a deeper understanding of customer behavior and risk reduction. The partnership between Confluent and Databricks embodies the digital transformation journey, breaking down data silos and improving efficiency, productivity, and data availability. Additionally, Confluent's commitment to open-source solutions has been recognized with the Microsoft OSS on Azure Global Partner of the Year award, underscoring the strategic importance of data streaming in the modern business landscape.

Saxo Bank has adopted a data mesh architecture to address integration challenges and deliver data-driven insights across the organization. The company has leveraged Confluent, Flink, and MongoDB to build a scalable and flexible data fabric. By federating domain ownership and centralizing oversight, Saxo aims to ensure that data is accessible, understandable, and consumable by various stakeholders. The data mesh architecture is technology-agnostic, with Protobuf serving as the foundational layer for data domains. To address discoverability, Saxo has implemented a self-service platform that provides a consistent user experience across domains, utilizing metadata and metrics to surface data assets and their relationships. By embracing Protobuf's strong message types and field-level annotations, Saxo aims to improve data product usability and drive continuous improvement through ongoing curation of domain models. The company also emphasizes the importance of standards, referencing external coding schemes and business terms to ensure consistency and accuracy in data representation.

The webinar focuses on guiding developers and architects through the process of building a RAG-enabled GenAI system using Confluent, Flink, and MongoDB. It highlights the challenges of incorporating tasks such as stream processing, connectivity to third-party systems, and scaling to production when working with data in motion, and suggests using Confluent for a comprehensive solution. The tutorial explains how to set up a data pipeline using Confluent Cloud and Platform, MongoDB Atlas, and ksqlDB for joining streams and enriching data. It emphasizes the advantages of using managed cloud services like MongoDB Atlas for flexibility and quick scalability, and demonstrates how to replicate Kafka topics between environments using Confluent Replicator. The tutorial also covers the setup of a self-managed environment with Docker to run the same data processes locally, bridging on-premises and cloud systems seamlessly. Finally, it introduces the availability of Confluent Platform 7.7 with enhanced features and discusses integrating Confluent with Amazon OpenSearch using OpenSearch Ingestion.

The text delves into the evolution of stream processing, emphasizing the pivotal role of Apache Kafka and its component, Kafka Streams, in addressing challenges of consistency and completeness in data streaming. Initially perceived as a real-time, low-latency, yet potentially transient architecture, stream processing has advanced to offer robust correctness guarantees through mechanisms like exactly-once semantics and speculative processing. Kafka achieves this by integrating stream processing with persistent logging, utilizing idempotent and transactional writes for consistency, and revision-based approaches for completeness, even with out-of-order data. This approach allows Kafka to manage the trade-offs between latency, throughput, and correctness more flexibly compared to other frameworks that often incur higher latencies. The discussion also highlights a white paper, presented at the ACM SIGMOD International Conference, detailing these innovations and their applications in large-scale deployments by companies like Bloomberg and Expedia.

Confluent Platform 6.2 introduces Health+, a new tool designed to enhance the reliability of data-in-motion infrastructures by offering intelligent alerts, cloud-based monitoring, and a streamlined support experience, crucial for minimizing business disruption. Health+ provides real-time insights into cluster performance and health through extensive rule-based analyses and offers recommendations for issue resolution, leveraging over a million hours of Kafka expertise. The platform also simplifies disaster recovery with its new failover command in Cluster Linking, allowing seamless data mirroring between clusters, and includes ksqlDB 0.17 enhancements for real-time application development. Confluent Platform 6.2, built on Kafka version 2.8, supports KRaft for improved partition management and security, enhancing the overall data-in-motion capabilities for enterprises.