July 2022 Summaries

The blog post explores the transformation of data integration and pipeline practices, drawing parallels with the evolution of service development over the past two decades. It highlights the shift toward treating data as a product, encouraging decentralization and better data sharing among organizations. The post outlines the challenges with traditional pipelines, such as fragility and redundancy, and introduces trends like declarative transformation models and ELT (Extract, Load, Transform) that aim to improve the pipeline experience. It also discusses the emergence of developer-friendly tools and the significance of stream processing in creating a network of real-time data flows. The post emphasizes the importance of adopting modern software practices, like agile and DevOps, for building resilient and scalable streaming data pipelines. These pipelines are crucial for enabling organizations to derive value from data efficiently, fostering collaboration, and ensuring timely, reusable, and composable data capabilities.

The Q3 '22 Launch of Confluent Cloud brings several key features designed to enhance developer productivity when working with real-time data streams, including the automation of infrastructure management and separation of networking from cluster workflows. The new Terraform provider allows businesses to accelerate development while avoiding operational costs by managing infrastructure through code. This approach gives developers safe, reliable access to infrastructure resources while operators can offload day-to-day provisioning requirements to version-controlled automations. Additionally, Confluent Cloud now offers 70 expert-built connectors ready to launch with just a few clicks, including source connectors for popular data systems like Google Cloud Storage and Amazon S3. The platform also includes new user login monitoring to spot potential breaches, flexible topic naming, and improved Cluster Linking capabilities. These features aim to simplify the setup of common use cases and improve the overall developer experience.

Confluent Platform 7.2 offers several enhancements to existing features, including improved management tasks for Apache Kafka operators, flexible options to build globally available hybrid architectures, and boosted developer productivity. The platform now supports source-initiated links in Confluent for Kubernetes, enabling the creation of cluster links that originate from the source cluster without requiring firewall access or storing security credentials in the cloud. Additionally, Confluent Platform 7.2 introduces flexible topic naming for Cluster Linking, simplifying the setup of common hybrid and multi-cloud use cases. The platform also includes several enhancements to ksqlDB, such as support for aggregate functions with complex data types and RIGHT JOINs for stream-stream and table-table-joins. Furthermore, Confluent Platform 7.2 is built on Apache Kafka version 3.2, which simplifies the security architecture by introducing a Kraft-based authorizer.

The text discusses the Confluent Stream Catalog, which is a centralized view of metadata for data-in-motion in modern data stacks. It helps organizations scale and govern their Kafka and real-time data streaming by providing features like tracking all metadata about data-in-motion, tagging or classifying entities, and augmenting entities with business metadata. The Confluent Stream Catalog uses Apache Atlas for its entity-relationship model and supports REST API and newly released GraphQL API for searching schemas and their components. It is one of three pillars among the Stream Governance offerings at Confluent, along with Stream Lineage and Stream Quality. The text also mentions the general availability of Confluent Platform 7.7 and its latest key features, including enhanced security with OAuth support and integration with Amazon OpenSearch using OpenSearch Ingestion.

The text discusses the development of Confluent Cloud's storage system, which is designed to be more scalable and performant than Apache Kafka. It explains how Confluent built an intelligent storage management system that allows for quick scaling up or down, addressing various operational challenges faced by customers. This new storage engine separates compute and storage resources, enabling users to store as much data as they want without worrying about capacity issues. The text also highlights the adoption of Apache Kafka as a source of truth for businesses due to its ability to provide real-time and historical events from any time period. Confluent Cloud's Infinite Storage ensures companies meet regulatory requirements for data retention, while resource isolation prevents latency spikes and improves throughput. The text concludes by encouraging users to try out the 10x better storage offering with a free trial of Confluent Platform 7.7.

Confluent and Databricks are integrated technologies being used by capital markets and trading operations businesses to gain a real-time understanding of their current risk, informing their trading strategies. This is achieved through the use of Confluent's streaming data technology, which enriches portfolio and market data in real time and feeds it into mathematical models and simulations performed within Databricks' Lakehouse. The result is that key risk metrics can be available near real-time to traders, enabling them to make informed decisions.

The text discusses the challenges of data governance in modern businesses, where data flows across different layers of an organization's application and data stack without proper controls and visibility. Apache Kafka is presented as a central nervous system for real-time data, connecting, storing, and analyzing continuous streams of data. Confluent's Stream Lineage tool is introduced to address the lack of data visibility in Kafka and event-driven architectures, providing an interactive map of all data flows that enable users to visualize and track critical data flows across their organization immediately as they occur. The tool allows users to navigate the flow of data, inspect and learn more about producers, consumers, or topics, and troubleshoot issues quickly. Stream Lineage is demonstrated through example scenarios, showcasing its ability to help developers build and visualize data pipelines, identify issues, and support audits and compliance questionnaires.