January 2023 Summaries

Confluent Cloud has introduced new features that enable businesses to build a secure shared services platform for data streaming, maximizing efficiency. These features include OAuth support, which allows customers to integrate with their own third-party identity provider and centralize account management across all of their cloud services. Additionally, RBAC (Role-Based Access Control) has been expanded to now include all Confluent Cloud resources, enabling better security and faster time to production. Furthermore, Cloud Client Quotas on Confluent Cloud enable businesses to easily deliver enterprise-wide access to highly performant Apache Kafka while reducing operational complexity and cost.

The text discusses the challenges and solutions associated with building and managing data streaming systems, particularly focusing on the use of Confluent Cloud’s Stream Governance tools. It narrates the evolution of a data streaming application from a simple microservice setup to a complex system that became difficult to manage and share across teams due to issues with data discoverability and outdated documentation. The text highlights how Confluent Cloud’s tools, such as the Schema Registry, Stream Catalog, and Stream Lineage, can address these challenges by ensuring data compatibility, enhancing stream discoverability, and providing real-time visualization of data flow, thus maintaining data integrity and encouraging broader adoption within an organization. It emphasizes the importance of automated tooling over manual documentation to maintain up-to-date and reliable data streams, suggesting that these tools could transform how teams interact with and benefit from data streaming architectures.

Treehouse Software addresses the complexities faced by enterprises seeking to modernize their legacy mainframe systems, which often include disparate databases and data stores accumulated through mergers and acquisitions. These organizations find themselves in complex, multi-cloud environments, making integration challenging due to the brittle nature of point-to-point connections that do not scale well. A growing number of firms are interested in establishing mainframe-to-Confluent data pipelines, using Confluent as an event streaming platform that provides a scalable and reliable data propagation system without a predetermined final target. This approach allows companies flexibility in subscribing data to various ETL toolsets and target datastores. Treehouse Software’s tcVISION facilitates real-time data synchronization from mainframes to Confluent, enabling integration with cloud services like AWS, Azure, and Google Cloud, and offering support for a wide range of mainframe data sources. The combination of tcVISION and Confluent's architecture allows for efficient data aggregation and management in hybrid and multicloud environments, enhancing data pipeline oversight and governance.

Change data capture (CDC) is an increasingly popular data pattern that converts database changes into events, enabling real-time data streaming for analytics and operational use cases without the need to replace existing relational databases. CDC can address the limitations of traditional databases by transforming them from static repositories to dynamic, reactive systems. It allows businesses to leverage event-driven architectures while maintaining their current infrastructure, offering solutions like reverse ETL and real-time data analysis. The implementation of CDC can vary, with techniques such as polling connectors, triggers, materialized views, and outbox tables being used to handle denormalization and event streaming. These methods help isolate the internal data model of a source system, minimizing dependencies and enhancing data utility. Tools like Debezium and platforms like Confluent Cloud support CDC, facilitating the development of data pipelines that integrate seamlessly into modern, real-time applications. As businesses increasingly rely on data-driven insights to inform their products, the role of CDC becomes more pivotal in bridging the gap between operational and analytical data needs.

Confluent has announced its acquisition of Immerok, a startup known for its managed Apache Flink service, which will be integrated into Confluent Cloud to enhance its stream processing capabilities. This move aligns with Confluent's strategy to make streaming data the default in modern data architectures by simplifying both the operational deployment and development of data streaming applications. Apache Flink was chosen for its multi-language support, robust processing model, and active community. Confluent plans to launch Flink on its platform later this year, starting with SQL support, and aims to create a seamless integration with its existing Kafka services. This strategic expansion not only solidifies Flink's role as a standard for stream processing but also highlights Confluent's commitment to open-source community contributions, mirroring its efforts with Kafka. Additionally, Confluent's recognition as Microsoft's 2024 OSS on Azure Global Partner of the Year underscores its dedication to delivering top-tier open source-based solutions.