September 2022 Summaries

In the rapidly evolving landscape of data management, the integration of real-time data and event-driven systems has become crucial across industries, with 97% of organizations globally utilizing streaming data technology, as highlighted in Confluent’s State of Data in Motion Report 2022. This shift necessitates robust data governance strategies to ensure data quality and compliance, moving away from restrictive measures to enabling broader access within organizations. Confluent has been at the forefront, developing Stream Governance on Confluent Cloud, the only fully managed governance suite for Apache Kafka® and data in motion, to facilitate the discovery, understanding, and trust of data streams. The journey of BT Group, a major UK telecoms company, in creating a 'Smart Event Mesh' illustrates how real-time data streams can be effectively governed across hybrid cloud environments. This transformation aligns with regulatory demands, such as those imposed by the Markets in Financial Instruments Directive II (MiFID II), which mandates transparency in European financial markets.

Confluent outlined its participation in the primarily virtual Google Cloud Next event on October 11–13, 2022, highlighting four sessions focused on data streaming integrations with Google Cloud. Topics included securing Confluent Cloud workloads through Google Cloud Private Service Connect, using Confluent Cloud and AlloyDB to support database modernization and real-time analytics, a panel with MongoDB on open-cloud approaches to data-driven transformation, and a demonstration of streaming at scale with Confluent Cloud and Dataproc Serverless Spark. The company also planned to announce additional updates during the event and offered a promotional package for attendees who started a Confluent trial through Google Cloud Marketplace.

The text discusses the launch of the Confluent Terraform Provider, which enables businesses to manage cloud and on-premise resources using Infrastructure as Code with HashiCorp Terraform, facilitating automated infrastructure management and data streaming initiatives. It details the essential role of Terraform resources, which describe infrastructure objects like Environment and Kafka Cluster, and explains how the Confluent Terraform Provider can manage these resources. A guided walkthrough demonstrates how to apply Terraform configurations to build and manage infrastructure, including setting up environments, Kafka clusters, and role bindings. The text also highlights upcoming investments in Terraform to reduce complexity and enhance data streaming infrastructure deployments, encourages new users to sign up for a free trial of Confluent Cloud, and offers additional resources and support for users to explore further integrations, such as with Amazon OpenSearch.

Microservices emerged as a way to avoid the deployment and coordination challenges of monolithic applications by allowing smaller services to be changed, scaled, and deployed independently, but synchronous HTTP, REST, or gRPC communication can recreate tight dependencies and lead to distributed monoliths. Event-driven architecture using Apache Kafka offers an alternative in which services publish durable events to topics and independently consume relevant events, reducing design-time coupling, supporting replayable data, and allowing new consumers to be added without disrupting existing workflows. The example builds a Python and Flask random pizza-ordering system where a client submits an order to PizzaService over HTTP, then separate sauce, cheese, meat, and vegetable services sequentially enrich pizza events through Kafka topics before PizzaService collects completed orders. Although event schemas still create some coupling and unavailable processing services can prevent an order from completing, tools such as Confluent Schema Registry can manage schema evolution, while independent consumers such as a cheese-reporting service can continue serving potentially stale results during upstream failures.

Kafka Connect is a tool used to integrate Apache Kafka with external data sources, such as databases, and provides a flexible way to bridge data silos in microservices architectures. It uses Change Data Capture (CDC) tools to read from the database's transaction log, extracting events at low latency and impact. This allows for real-time synchronization of data across multiple databases, enabling features like three-way database syncs for complex use cases. Kafka Connect solves problems such as handling different data types, destination database data models, and referential integrity issues, providing a flexible solution for integrating Kafka with external data sources.

Current 2022, The Next Generation of Kafka Summit, will take place on October 4-5 in Austin and will gather data streaming experts, influencers, and business leaders from the Kafka community and beyond. This two-day event aims to help industry leaders and technologists learn and connect, accelerating innovation across the data streaming ecosystem. The summit will feature presentations from companies such as Netflix, Snowflake, Shopify, Uber, Microsoft Azure, and Nasdaq on how Apache Kafka is central to the rise of real-time use cases and how data streaming is used as a whole. Attendees can expect to learn about industry best practices, new technologies, and opportunities for innovation and growth in data streaming.

A collection of Confluent blog posts from January through September 2022 covers Apache Kafka, ksqlDB, Confluent Platform, and real-time stream processing applications. Topics include infrastructure for large-scale gaming, wildlife monitoring, serverless processing with Azure Functions, and stream-processing use cases such as fraud detection, predictive analytics, cybersecurity, and customer experiences. The posts also announce several ksqlDB releases, highlighting enhancements to multi-schema topic access, aggregate functions, joins, schemas, complex data types, record headers, and pull queries on streams. Additional announcements describe Confluent Platform 7.1 and 7.2 improvements in data quality, cloud-native security, geo-replication, and Kafka operations, alongside discussion of the growing importance of real-time processing compared with batch processing.

Gaming companies increasingly use real-time event streaming to support player engagement, monetization, analytics, and fraud prevention at large scale, with Apache Kafka and services such as Confluent Cloud providing managed data ingestion, processing, governance, and security. Because Kafka is designed primarily for backend-system communication rather than millions of direct web-client connections, WebSockets can serve as a complementary last-mile delivery layer by maintaining persistent bidirectional connections that reduce latency, server load, and bandwidth overhead compared with HTTP polling. The described gamification demo, inspired by Disney+ Hotstar’s live cricket interaction features, combines Kafka, MigratoryData, and WebSockets to send timed questions to players, collect answers, return scores, and update a top-10 leaderboard in real time. MigratoryData automatically maps WebSocket subjects to Kafka topics and keys, while backend processors handle question generation, answer validation, scoring, and leaderboard requests; ksqlDB can simplify this architecture by replacing several custom processing components with stream-processing queries. Benchmark results cited by the post indicate that one MigratoryData instance on commodity hardware can support one million concurrent players and that clustering provides linear horizontal scaling, positioning the Kafka-WebSocket combination as an approach for interactive gaming experiences involving millions of users.

A year after its IPO, Confluent’s EMEA Sales Operations team described supporting the company’s rapid growth by managing forecasting, quarterly business reviews, sales planning, reporting, account issues, analytics, and dashboards. Ian Phillpot leads EMEA go-to-market operations, Leanne Moylan partners with sales leaders across central and southern Europe, and Harry Marsh provides regional strategy and data analysis. They cited Confluent’s collaborative culture, accessible leadership, onboarding materials, training resources, and hands-on learning as important to their development, while noting that rapid scaling requires redesigning processes, balancing stakeholder needs, and creating solutions where established procedures do not yet exist. The team views these challenges as opportunities to build expertise, improve sales visibility, solve operational problems, and advance their careers alongside the expanding business.

Financial services organizations are leveraging modern data architectures and streaming platforms to stay competitive in a rapidly evolving digital marketplace projected to reach $28.5 trillion by 2025. These companies, which include large banks, fintech firms, and insurance providers, are increasingly using Apache Kafka through Confluent Cloud to enhance real-time data processing, improve customer experiences, and streamline operations. By adopting event-driven architectures, financial institutions can address technical challenges posed by legacy systems, meet regulatory requirements, and innovate in areas such as fraud detection, personalized banking, and trading strategies. The use of data streaming allows for the creation of scalable, efficient applications that integrate data silos, enabling better decision-making and risk management. Notably, Confluent Cloud facilitates the development of custom software solutions that support diverse use cases, from consumer banking and online payments to insurance underwriting and risk assessment, by providing an enterprise-grade experience that includes tools for data integration, real-time analytics, and enhanced security measures.

Confluent’s Current 2022 conference, held October 4–5 in Austin, Texas, was presented as a gathering for developers, data scientists, architects, and industry leaders working with real-time data streaming and Apache Kafka. Interviews with presenters Jay Patel of Snowflake, Sanjana Kaundinya of Confluent, and Viktor Gamov of Kong emphasized the growing demand for immediate data access, the importance of collaboration beyond the Kafka community, and the value of learning how organizations deploy streaming systems in production. The event program included Kafka training, technical breakouts, expert sessions, networking, and presentations from companies such as Netflix, AWS, Databricks, and USPS. Speakers highlighted opportunities to exchange practical lessons across organizations, explore topics including change data capture, Apache Iceberg, multi-region replication, real-time analytics, and availability, and build professional relationships that can support career growth. The conference also aimed to help newcomers gain foundational knowledge while giving experienced practitioners updates on a rapidly evolving data-streaming landscape.

This blog post discusses a strategy for migrating data from on-premise relational databases to Amazon Aurora using Confluent technology. The process involves setting up real-time applications with legacy or existing data sources and building a streaming data pipeline that integrates with the cloud database. By moving to real-time streaming, organizations can provide application teams with faster access to data and deliver new features and capabilities at scale. Amazon Aurora is a fully managed relational database service designed for running in the cloud, offering high performance and availability while automating time-consuming administration tasks. The migration process involves setting up Confluent connectors to synchronize data between on-premise databases and Amazon Aurora, followed by cutting over event-driven applications to source data from Aurora. This approach helps organizations modernize their database strategy and improve agility, elasticity, and cost efficiency in the cloud environment.

Confluent Cloud offers a highly resilient and reliable data streaming service that surpasses the capabilities of open-source Apache Kafka, delivering a 99.99% uptime availability SLA, which significantly reduces potential downtime. This is achieved through extensive investments in redundancy, proactive monitoring, and self-healing automation, ensuring data integrity and service continuity even during major cloud outages affecting providers like AWS, Azure, and GCP. The service's design includes features like Cluster Linking and Schema Linking, allowing seamless multi-region disaster recovery and enhanced data durability, which benefits organizations by reducing the operational burden traditionally associated with maintaining such high availability and durability with Kafka. Confluent Cloud's architecture, which simplifies the deployment of multi-region disaster recovery strategies, is credited with enabling companies like 10x Banking to deliver hyper-personalized and flawless financial products to their clients.

Health+ is a cloud-based monitoring solution for Apache Kafka and Confluent Platform that helps teams prioritize and manage telemetry data efficiently, offering a significant reduction in monitoring costs. Built from experience running thousands of clusters, Health+ includes new dashboards that provide key insights into the health of Kafka and Confluent Platform clusters, automatically highlighting critical metrics and alerts. The tool is designed with security in mind, sending encrypted telemetry data to Confluent Cloud while allowing for easy setup and integration with existing infrastructure. Health+ continuously evolves with user feedback, adding features and metrics that enhance its utility for both new and power users, while ensuring seamless updates with minimal user intervention. The blog also highlights the release of Confluent Platform 7.7, featuring enhanced security and new integrations.

Apache Kafka's rich ecosystem of connectors necessitates managing multiple connectors in a consistent, infra-as-code, automated manner. Confluent for Kubernetes (CFK) enables this by providing a declarative API to manage connectors through an infrastructure-as-code approach. CFK uses CustomResourceDefinitions (CRDs) and Kubernetes APIs to deploy and self-manage Confluent Platform components as well as application resources like topics, schemas, and connectors. With CFK, connectors can be managed efficiently, and their lifecycle can be automated. The platform also supports the use of init containers to download and install connector plugins, making it easy to provision Kafka Connect with connector plugins through a one-step operation.

A back-to-school-themed data streaming syllabus introduces learners to Apache Kafka, ksqlDB, streaming pipelines, modern data architectures, managed services, practical use cases, and community learning resources. It recommends beginning with Kafka’s core concepts, including events, topics, partitions, producers, and consumers, before exploring how ksqlDB processes event data within streaming pipelines to reduce batch-processing delays. The syllabus situates Kafka within decentralized data architectures such as data mesh, which emphasizes domain ownership, data as a product, self-service platforms, and federated governance. It also explains that managed offerings such as Confluent Cloud can support mission-critical streaming through managed connectors and scalable storage, while illustrating real-world applications such as keeping retail inventory and customer orders synchronized. Learners are encouraged to develop hands-on skills through the 100 Days of Code for Apache Kafka challenge, engage with peers online, explore related technologies, and attend Current 2022 for further education and networking.