June 2022 Summaries

In the era of data-driven decision-making, maintaining high data quality is paramount, especially with the rise of real-time data streaming, where poor data quality can quickly propagate and disrupt systems. Confluent's Stream Quality solution addresses this challenge by offering Schema Registry and Schema Validation tools that ensure trustworthy real-time data streams within Kafka ecosystems. Schema Registry serves as a central repository for schemas, facilitating seamless data serialization and deserialization while supporting formats like Avro, Protocol Buffers, and JSON. Schema Validation, available on Confluent Cloud and Platform, enforces adherence to these schemas, discarding data that does not comply, thereby preventing system failures due to invalid data. The blog highlights the importance of schema-driven development, illustrating through a case study how a lack of schema enforcement led to significant disruptions in a business's operations. By using these tools, organizations can enhance data integrity and reliability across services, ensuring seamless evolution of data structures without service breakdowns. The announcement coincides with the release of Confluent Platform 7.7, introducing features such as enhanced security, integration with Apache Flink, and new connectors, emphasizing the platform's commitment to robust, secure, and efficient data streaming solutions.

Keerthana Srikanth, a computer science graduate from BITS Pilani, joined Confluent’s Observability India Team as a software engineer after completing a six-month internship, where she became interested in Kafka and experienced the company’s collaborative culture. As part of the Notifications Service Team, she helps build a centralized, customizable platform for customer notifications across Confluent products, while contributing to feature planning, technical design, coding, testing, onboarding internal users, and debugging. She describes Confluent as a remote-first but highly connected workplace that encourages engineers, including junior developers, to consider product improvements and business needs beyond their assigned coding tasks. Keerthana highlights mentorship, meaningful internship work, approachable colleagues, and a culture of ownership as important parts of her professional development. Her advice to new developers is to recognize the differences between academic and professional engineering, weigh practical trade-offs, learn efficiently without needing to master every technology immediately, ask questions, listen closely, and take initiative beyond daily responsibilities.

Confluent, a leading provider of data streaming platforms, has won Microsoft's Commercial Marketplace Partner of the Year Award (Global) in 2022. This award recognizes companies that excel in innovation and delivery of unique solutions through Microsoft's commercial marketplace. Confluent's marketplace solution is integrated with Azure security, management, billing, and data analytics services, offering a more agile and reliable approach for customers to procure and deploy their data streaming solution. The company has made significant efforts to streamline the process of getting started with Confluent on Azure, including free trials and SSO capabilities through third parties like GitHub. Microsoft's VP and GM Commercial Marketplace, Jake Swenson, praised Confluent for helping businesses make the most of real-time data and partnering effectively to reach customers globally. This recognition highlights the strong partnership between Microsoft and Confluent, with both companies committed to innovating and bringing new data streaming solutions and use cases to market.

Confluent for Kubernetes has added declarative API support for Cluster Linking, allowing users to connect public and private cloud environments with a declarative API. This feature provides a complete, declarative API-driven experience for deploying and self-managing Confluent Platform as a cloud-native system. With the release of Confluent Platform 7.0, Confluent announced the general availability of Cluster Linking in Confluent Platform and Confluent Cloud. The declarative API enables users to express the state of their infrastructure and application state as code in YAML files, which can be checked into a Git repository for collaboration. This approach is referred to as GitOps. Confluent for Kubernetes provides a declarative ClusterLink API to configure and manage Cluster Linking, allowing users to define the desired state of their cluster links using a CustomResource based on the ClusterLink CustomResourceDefinition. The feature also includes authentication and TLS information, which can be configured directly via secrets or HashiCorp Vault. With this feature, users can create hybrid cloud environments with simplicity, flexibility, and efficiency without the complexities of Kafka-related infrastructure operations.

A Confluent keynote demonstration uses a merged-airline scenario to illustrate five principles for modern data systems: real-time streaming, decentralized data access, declarative development, developer-oriented workflows, and governance with observability. Infrastructure, Kafka clusters, topics, access controls, and connectors are provisioned through Terraform and GitOps practices, while flight updates and data from separate booking and customer systems are streamed into Kafka rather than connected through brittle point-to-point integrations. Schema Registry provides compatibility controls as data evolves, and ksqlDB transforms, joins, cleanses, standardizes, and optionally redacts data to create a trusted unified bookings data product. The same governed streams can support both analytical workloads, such as sending flight data to Snowflake, and operational services, such as identifying delayed flights and notifying affected passengers through serverless applications. The example argues that this architecture lets independent teams choose their own tools while sharing reliable real-time data, with lineage, access controls, monitoring, and infrastructure automation helping manage an expanding ecosystem of applications and data sources.

Autonomous networks are becoming increasingly critical for communication service providers as they aim to create rich customer experiences by integrating legacy operations support systems with modern 5G networks. This integration requires new approaches and methodologies to network operations, driven by technologies such as NFV, MEC, AI/ML, SDN, and cloud-native computing. Confluent offers a data platform that enables real-time, interconnected application services and provides a self-healing, self-service infrastructure for CSPs to deliver a unified, real-time experience for customers and clients. By leveraging Confluent's data platform, companies like BT Group have been able to create well-governed, real-time streams of data across hybrid cloud environments, enabling them to navigate the complexities of modern 5G networks and drive innovation in customer experiences.

Current 2022: The Next Generation of Kafka Summit opened its call for papers through June 26, with a newly announced program committee responsible for selecting the conference agenda. The committee includes returning Kafka Summit members and industry figures from organizations such as Red Hat, Snowflake, Confluent, Netflix, Honeycomb, AWS, Figma, and Treeverse, including contributors to projects and books involving Debezium, Apache Samza, Apache Spark, Apache Flink, and Kafka Connect. Although Apache Kafka will be a prominent topic, the event is designed to cover the wider data-streaming ecosystem through tracks on real-time application development, streaming technology internals, large-scale architectures, business case studies, data pipelines, and real-time analytics. Prospective speakers are encouraged to submit sessions and consult related guidance on preparing proposals.

No summary generated yet.

Confluent describes its focus on simplifying real-time data management for organizations whose information is spread across multiple systems, formats, and locations. It argues that conventional batch pipelines, which collect, clean, and load data into warehouses through several tools, can be costly, slow, and difficult to debug or reprocess, and proposes building data systems from the point where events originate. The post also emphasizes the importance of an engineering culture built on collaboration, trust, diverse perspectives, calculated risk-taking, and rapid onboarding, particularly because large software projects can be difficult for new engineers to understand. It presents Confluent as pursuing a next-generation data platform while seeking engineers motivated to address complex industry challenges with sustained organizational support and customer focus.

No summary generated yet.

Elasticity is a fundamental feature of mature cloud services, enabling users to adjust capacity based on demand fluctuations, and Confluent Cloud now supports this capability for dedicated Apache Kafka clusters. Users can expand or shrink Kafka clusters in terms of Confluent Units (CKUs) through a self-serve model available via Confluent Cloud's UI, CLI, and public APIs. This allows for alignment with business needs and cost optimization while maintaining performance. Monitoring cluster load is essential before resizing, as high loads can increase latency and throttling, while low loads may suggest a cost-saving opportunity. The process involves adding or removing CKUs and rebalancing data, which takes a few hours depending on the size adjustment. Safeguards prevent adverse impacts during resizing, and billing adjusts according to the actual capacity used. The blog post emphasizes user experience, covering key aspects such as the control plane's role in resizing operations and introducing features like cluster load metrics. Future plans include enabling autoscaling based on user-defined policies to further ease capacity management.

ksqlDB 0.26 has been released with several key improvements, including enhancements to aggregate functions, JOINs, and schemas. The new release extends the MIN and MAX aggregates to natively support DATE, TIME, and TIMESTAMP types, making it easier for users to perform aggregations without extra casting. Additionally, ksqlDB now supports Stream-Stream and Table-Table RIGHT OUTER JOINs, allowing for more flexible data joining capabilities. The release also improves schema management with enhanced Protobuf schema support, enabling the definition of multiple schemas within a single schema definition. Furthermore, the update includes bug fixes for null-handling improvements and dereferencing null structs. Overall, these enhancements aim to improve the user experience and expand ksqlDB's capabilities in data processing and analysis.