June 2022 Summaries

In the era of data-driven decision-making, maintaining high data quality is paramount, especially with the rise of real-time data streaming, where poor data quality can quickly propagate and disrupt systems. Confluent's Stream Quality solution addresses this challenge by offering Schema Registry and Schema Validation tools that ensure trustworthy real-time data streams within Kafka ecosystems. Schema Registry serves as a central repository for schemas, facilitating seamless data serialization and deserialization while supporting formats like Avro, Protocol Buffers, and JSON. Schema Validation, available on Confluent Cloud and Platform, enforces adherence to these schemas, discarding data that does not comply, thereby preventing system failures due to invalid data. The blog highlights the importance of schema-driven development, illustrating through a case study how a lack of schema enforcement led to significant disruptions in a business's operations. By using these tools, organizations can enhance data integrity and reliability across services, ensuring seamless evolution of data structures without service breakdowns. The announcement coincides with the release of Confluent Platform 7.7, introducing features such as enhanced security, integration with Apache Flink, and new connectors, emphasizing the platform's commitment to robust, secure, and efficient data streaming solutions.

Confluent, a leading provider of data streaming platforms, has won Microsoft's Commercial Marketplace Partner of the Year Award (Global) in 2022. This award recognizes companies that excel in innovation and delivery of unique solutions through Microsoft's commercial marketplace. Confluent's marketplace solution is integrated with Azure security, management, billing, and data analytics services, offering a more agile and reliable approach for customers to procure and deploy their data streaming solution. The company has made significant efforts to streamline the process of getting started with Confluent on Azure, including free trials and SSO capabilities through third parties like GitHub. Microsoft's VP and GM Commercial Marketplace, Jake Swenson, praised Confluent for helping businesses make the most of real-time data and partnering effectively to reach customers globally. This recognition highlights the strong partnership between Microsoft and Confluent, with both companies committed to innovating and bringing new data streaming solutions and use cases to market.

Confluent for Kubernetes has added declarative API support for Cluster Linking, allowing users to connect public and private cloud environments with a declarative API. This feature provides a complete, declarative API-driven experience for deploying and self-managing Confluent Platform as a cloud-native system. With the release of Confluent Platform 7.0, Confluent announced the general availability of Cluster Linking in Confluent Platform and Confluent Cloud. The declarative API enables users to express the state of their infrastructure and application state as code in YAML files, which can be checked into a Git repository for collaboration. This approach is referred to as GitOps. Confluent for Kubernetes provides a declarative ClusterLink API to configure and manage Cluster Linking, allowing users to define the desired state of their cluster links using a CustomResource based on the ClusterLink CustomResourceDefinition. The feature also includes authentication and TLS information, which can be configured directly via secrets or HashiCorp Vault. With this feature, users can create hybrid cloud environments with simplicity, flexibility, and efficiency without the complexities of Kafka-related infrastructure operations.

Autonomous networks are becoming increasingly critical for communication service providers as they aim to create rich customer experiences by integrating legacy operations support systems with modern 5G networks. This integration requires new approaches and methodologies to network operations, driven by technologies such as NFV, MEC, AI/ML, SDN, and cloud-native computing. Confluent offers a data platform that enables real-time, interconnected application services and provides a self-healing, self-service infrastructure for CSPs to deliver a unified, real-time experience for customers and clients. By leveraging Confluent's data platform, companies like BT Group have been able to create well-governed, real-time streams of data across hybrid cloud environments, enabling them to navigate the complexities of modern 5G networks and drive innovation in customer experiences.

Elasticity is a fundamental feature of mature cloud services, enabling users to adjust capacity based on demand fluctuations, and Confluent Cloud now supports this capability for dedicated Apache Kafka clusters. Users can expand or shrink Kafka clusters in terms of Confluent Units (CKUs) through a self-serve model available via Confluent Cloud's UI, CLI, and public APIs. This allows for alignment with business needs and cost optimization while maintaining performance. Monitoring cluster load is essential before resizing, as high loads can increase latency and throttling, while low loads may suggest a cost-saving opportunity. The process involves adding or removing CKUs and rebalancing data, which takes a few hours depending on the size adjustment. Safeguards prevent adverse impacts during resizing, and billing adjusts according to the actual capacity used. The blog post emphasizes user experience, covering key aspects such as the control plane's role in resizing operations and introducing features like cluster load metrics. Future plans include enabling autoscaling based on user-defined policies to further ease capacity management.

ksqlDB 0.26 has been released with several key improvements, including enhancements to aggregate functions, JOINs, and schemas. The new release extends the MIN and MAX aggregates to natively support DATE, TIME, and TIMESTAMP types, making it easier for users to perform aggregations without extra casting. Additionally, ksqlDB now supports Stream-Stream and Table-Table RIGHT OUTER JOINs, allowing for more flexible data joining capabilities. The release also improves schema management with enhanced Protobuf schema support, enabling the definition of multiple schemas within a single schema definition. Furthermore, the update includes bug fixes for null-handling improvements and dereferencing null structs. Overall, these enhancements aim to improve the user experience and expand ksqlDB's capabilities in data processing and analysis.