September 2022 Summaries

In the rapidly evolving landscape of data management, the integration of real-time data and event-driven systems has become crucial across industries, with 97% of organizations globally utilizing streaming data technology, as highlighted in Confluent’s State of Data in Motion Report 2022. This shift necessitates robust data governance strategies to ensure data quality and compliance, moving away from restrictive measures to enabling broader access within organizations. Confluent has been at the forefront, developing Stream Governance on Confluent Cloud, the only fully managed governance suite for Apache Kafka® and data in motion, to facilitate the discovery, understanding, and trust of data streams. The journey of BT Group, a major UK telecoms company, in creating a 'Smart Event Mesh' illustrates how real-time data streams can be effectively governed across hybrid cloud environments. This transformation aligns with regulatory demands, such as those imposed by the Markets in Financial Instruments Directive II (MiFID II), which mandates transparency in European financial markets.

The text discusses the launch of the Confluent Terraform Provider, which enables businesses to manage cloud and on-premise resources using Infrastructure as Code with HashiCorp Terraform, facilitating automated infrastructure management and data streaming initiatives. It details the essential role of Terraform resources, which describe infrastructure objects like Environment and Kafka Cluster, and explains how the Confluent Terraform Provider can manage these resources. A guided walkthrough demonstrates how to apply Terraform configurations to build and manage infrastructure, including setting up environments, Kafka clusters, and role bindings. The text also highlights upcoming investments in Terraform to reduce complexity and enhance data streaming infrastructure deployments, encourages new users to sign up for a free trial of Confluent Cloud, and offers additional resources and support for users to explore further integrations, such as with Amazon OpenSearch.

Kafka Connect is a tool used to integrate Apache Kafka with external data sources, such as databases, and provides a flexible way to bridge data silos in microservices architectures. It uses Change Data Capture (CDC) tools to read from the database's transaction log, extracting events at low latency and impact. This allows for real-time synchronization of data across multiple databases, enabling features like three-way database syncs for complex use cases. Kafka Connect solves problems such as handling different data types, destination database data models, and referential integrity issues, providing a flexible solution for integrating Kafka with external data sources.

Current 2022, The Next Generation of Kafka Summit, will take place on October 4-5 in Austin and will gather data streaming experts, influencers, and business leaders from the Kafka community and beyond. This two-day event aims to help industry leaders and technologists learn and connect, accelerating innovation across the data streaming ecosystem. The summit will feature presentations from companies such as Netflix, Snowflake, Shopify, Uber, Microsoft Azure, and Nasdaq on how Apache Kafka is central to the rise of real-time use cases and how data streaming is used as a whole. Attendees can expect to learn about industry best practices, new technologies, and opportunities for innovation and growth in data streaming.

Financial services organizations are leveraging modern data architectures and streaming platforms to stay competitive in a rapidly evolving digital marketplace projected to reach $28.5 trillion by 2025. These companies, which include large banks, fintech firms, and insurance providers, are increasingly using Apache Kafka through Confluent Cloud to enhance real-time data processing, improve customer experiences, and streamline operations. By adopting event-driven architectures, financial institutions can address technical challenges posed by legacy systems, meet regulatory requirements, and innovate in areas such as fraud detection, personalized banking, and trading strategies. The use of data streaming allows for the creation of scalable, efficient applications that integrate data silos, enabling better decision-making and risk management. Notably, Confluent Cloud facilitates the development of custom software solutions that support diverse use cases, from consumer banking and online payments to insurance underwriting and risk assessment, by providing an enterprise-grade experience that includes tools for data integration, real-time analytics, and enhanced security measures.

This blog post discusses a strategy for migrating data from on-premise relational databases to Amazon Aurora using Confluent technology. The process involves setting up real-time applications with legacy or existing data sources and building a streaming data pipeline that integrates with the cloud database. By moving to real-time streaming, organizations can provide application teams with faster access to data and deliver new features and capabilities at scale. Amazon Aurora is a fully managed relational database service designed for running in the cloud, offering high performance and availability while automating time-consuming administration tasks. The migration process involves setting up Confluent connectors to synchronize data between on-premise databases and Amazon Aurora, followed by cutting over event-driven applications to source data from Aurora. This approach helps organizations modernize their database strategy and improve agility, elasticity, and cost efficiency in the cloud environment.

Confluent Cloud offers a highly resilient and reliable data streaming service that surpasses the capabilities of open-source Apache Kafka, delivering a 99.99% uptime availability SLA, which significantly reduces potential downtime. This is achieved through extensive investments in redundancy, proactive monitoring, and self-healing automation, ensuring data integrity and service continuity even during major cloud outages affecting providers like AWS, Azure, and GCP. The service's design includes features like Cluster Linking and Schema Linking, allowing seamless multi-region disaster recovery and enhanced data durability, which benefits organizations by reducing the operational burden traditionally associated with maintaining such high availability and durability with Kafka. Confluent Cloud's architecture, which simplifies the deployment of multi-region disaster recovery strategies, is credited with enabling companies like 10x Banking to deliver hyper-personalized and flawless financial products to their clients.

Health+ is a cloud-based monitoring solution for Apache Kafka and Confluent Platform that helps teams prioritize and manage telemetry data efficiently, offering a significant reduction in monitoring costs. Built from experience running thousands of clusters, Health+ includes new dashboards that provide key insights into the health of Kafka and Confluent Platform clusters, automatically highlighting critical metrics and alerts. The tool is designed with security in mind, sending encrypted telemetry data to Confluent Cloud while allowing for easy setup and integration with existing infrastructure. Health+ continuously evolves with user feedback, adding features and metrics that enhance its utility for both new and power users, while ensuring seamless updates with minimal user intervention. The blog also highlights the release of Confluent Platform 7.7, featuring enhanced security and new integrations.

Apache Kafka's rich ecosystem of connectors necessitates managing multiple connectors in a consistent, infra-as-code, automated manner. Confluent for Kubernetes (CFK) enables this by providing a declarative API to manage connectors through an infrastructure-as-code approach. CFK uses CustomResourceDefinitions (CRDs) and Kubernetes APIs to deploy and self-manage Confluent Platform components as well as application resources like topics, schemas, and connectors. With CFK, connectors can be managed efficiently, and their lifecycle can be automated. The platform also supports the use of init containers to download and install connector plugins, making it easy to provision Kafka Connect with connector plugins through a one-step operation.