January 2022 Summaries

The text discusses the integration of Confluent and MongoDB to create an efficient Internet of Things (IoT) pipeline, highlighting its applications in various industries, such as healthcare, smart factories, and energy, among others. It distinguishes between "thick" and "thin" edge IoT use cases, where thick edge requires substantial computing resources for localized intelligence, while thin edge involves minimal processing at the device level, with data sent directly to centralized processing hubs. The text elaborates on reference architectures for both types, using examples like cash logistics management to demonstrate the implementation of IoT solutions using Confluent and MongoDB. The implementation details include configuring data streams, using ksqlDB for data transformation, and leveraging MongoDB Atlas for storage and analysis. Additionally, it notes the features of MongoDB and Confluent supporting these IoT architectures, emphasizing the potential for improved scalability and operational efficiency.

Enabling data in motion within an organization involves more than just setting up a Confluent cluster, requiring consideration of reliability, security, integration with delivery pipelines and governance policies. A successful strategy depends on factors such as the type of workloads being moved to production and the level of maturity of the organization. To help address these concerns, a comprehensive guide has been launched, written by Confluent's professional services team based on their experience working with hundreds of customers across various stages of platform adoption. The guide provides a set of topics or recipes grouped into major themes, addressing common questions faced during implementation programs and serving as a valuable resource for organizations at any stage of their journey.

Apache Kafka has released version 3.1.0, which includes numerous improvements and new features. Some key updates include support for topic IDs in FetchRequest, enhanced naming consistency for client metrics, a built-in implementation of interfaces defined in KIP-255 for authentication and token retrieval, two new metrics exposed by ZooKeeper and KRaft controller, and more. The release also includes various bug fixes and improvements beyond the listed KIPs. This was a community effort with contributions from 114 authors and reviewers.

In response to the surge in demand during the COVID-19 pandemic, Instacart rapidly expanded its operations, partnering with over 65,000 stores and serving more than 85% of U.S. households, necessitating a significant overhaul of its data infrastructure. The company transitioned from batch to stream processing using Kafka and Confluent Cloud to manage the vast data generated by around 10 million daily customers, ensuring data quality and discoverability while maintaining a reliable IT architecture. This shift facilitated the creation of a Customer 360 view and improved inventory management, allowing Instacart to integrate real-time data into its systems and enhance predictive models for better customer service. By adopting a data mesh strategy, Instacart clarified data ownership and standardized processes, empowering teams to access and utilize high-quality data efficiently, ultimately improving algorithms and personalizing customer experiences. Confluent Cloud's serverless capabilities and infinite storage have become central to Instacart's data strategy, streamlining operations and allowing the company to focus on expanding its offerings without worrying about infrastructure constraints.

The Confluent Q1 '22 Launch introduces a range of new features and enhancements designed to improve real-time data streaming capabilities on Confluent Cloud, a cloud-native data streaming platform. The launch highlights the addition of 50+ fully managed connectors, making it the largest portfolio of such connectors available, which simplifies integration across business systems and reduces operational overhead. New integrations with Datadog and Prometheus offer deeper visibility and unified monitoring capabilities, allowing businesses to manage their data streams alongside existing technology stacks. Confluent's Dedicated clusters now support self-service capacity expansion and shrinking, enhancing cost efficiency and operational simplicity for high-throughput workloads. The introduction of Schema Linking offers a streamlined approach to maintaining consistent data quality across cloud and hybrid environments by syncing shared schemas in real time. Additionally, the new Confluent v2 CLI provides a unified interface for managing data streams across diverse environments, further simplifying the operational management of Apache Kafka deployments. Overall, these updates aim to bolster the efficiency, scalability, and reliability of data streaming and processing for businesses leveraging Confluent Cloud.

ksqlDB 0.23.1 introduces pull queries on streams, allowing users to easily scan and filter over topics using SQL. It also provides access to topic partition and offset through new pseudocolumns, ROWPARTITION and ROWOFFSET. Additionally, the release includes the ability to specify a grace period when joining streams, improving accuracy of joined and non-joined records in a window. These features are now available in Confluent Cloud.

The webinar discusses common challenges and solutions for maintaining smooth operations in Apache Kafka pipelines, focusing on both client-side and broker-side configurations. It emphasizes the importance of setting appropriate client-side configurations like request.timeout.ms and understanding producer retries to prevent performance issues and message loss. On the broker side, it highlights the significance of monitoring key metrics and managing partition counts to avoid downtime and optimize throughput. Additionally, it advises careful configuration of segment.ms to prevent excessive small segment files that could negatively impact performance. The webinar suggests utilizing Confluent Cloud to manage Kafka services efficiently and introduces Apache Kafka 3.8.0, which brings new features and improvements.

Confluent Platform 7.1 and Confluent Cloud now offer Cluster Linking, a built-in capability that mirrors data, topic configuration, and consumer offsets from one cluster to another in real-time, providing a secure, scalable, and easy-to-use hybrid data bridge between on-premises and cloud-based clusters. To create a cluster link, you need two halves of the single cluster link - one on each cluster - which can be created using Confluent's CLI commands or REST API calls. The source-initiated cluster link requires creating separate configuration files for both clusters, containing security credentials and other settings. Once created, Cluster Linking allows geo-replication of data from on-premises to the cloud, enabling organizations to execute a hybrid cloud strategy or migrate from a datacenter to the cloud with minimal disruption.