May 2022 Summaries

During the COVID-19 pandemic in 2020, Instacart faced unprecedented demand, which tested its systems and processes, highlighting the importance of elasticity in cloud-native computing for businesses to scale efficiently and cost-effectively. Confluent has significantly enhanced its cloud platform's elasticity, making it 10 times more elastic than Apache Kafka by leveraging Confluent Cloud Intelligent Storage. This improvement allows for faster data movement and scaling operations, reducing the time required for cluster expansions and contractions, and minimizing operational burdens such as capacity planning and data rebalancing. The platform's elasticity not only enhances scalability but also improves resiliency against failures, lowers total cost of ownership, and enables seamless upgrades with minimal downtime, making it a robust solution for modern data infrastructure needs. Additionally, Confluent has announced the general availability of Platform 7.7, featuring enhanced security and new integrations, encouraging users to explore its capabilities through a free trial.

The blog post highlights the evolution and enhancement of Confluent Cloud, a cloud-native service based on Apache Kafka, which has been refined over five years with over 3 million engineering hours to deliver a service that is 10 times superior to self-managed or semi-managed Kafka setups. It emphasizes the complexity of building a cloud service that goes beyond merely deploying open-source software to Kubernetes, focusing instead on providing a true elastic utility that transforms fragile data capabilities into scalable, automated, and data-driven operations. The post draws parallels between Confluent Cloud and other managed services like Snowflake, illustrating the significant design and operational differences from traditional solutions. Confluent Cloud integrates not just Kafka, but also a full ecosystem to support real-time applications, offering a unified data fabric across AWS, GCP, and Azure, complemented by Cluster Linking for seamless data movement. The blog also announces the availability of Confluent Platform 7.7, featuring enhanced security, Apache Flink integration, and other tools, while inviting readers to explore the service's capabilities firsthand and stay tuned for deeper insights into its engineering advancements.

The State of Data in Motion report, conducted by Confluent and Lawless Research, highlights the growing importance of real-time data streams for businesses to stay competitive in today's digital world. The survey found that 97% of IT and engineering leaders have access to some degree of real-time data streams, which they use to improve customer engagement, ensure regulatory compliance, and mitigate cyber risks. However, challenges such as real-time data synchronization between multiple environments and data integration across multiple sources can pose barriers against providing personalized, on-demand experiences for customers. Confluent offers a fully managed, cloud-native data streaming platform to help businesses overcome these hurdles and unlock the power of real-time data in motion.

Stream processing is an essential technology that allows businesses to process and analyze real-time data, crucial for maintaining competitiveness in various industries such as finance, retail, and cloud services. Unlike traditional batch processing, which processes data at intervals leading to delays, stream processing provides immediate insights by continuously handling data as it arrives, featuring low latency and support for event-time processing. This real-time capability is beneficial for applications like fraud detection, predictive analytics, and real-time gaming, enabling organizations to respond swiftly to data-driven events. Apache Kafka is a prominent platform for stream processing, offering functionalities like duality stream tables and real-time data pipelines that facilitate the manipulation and analysis of continuous data streams. Kafka Streams and ksqlDB are tools within Kafka that allow developers to build streaming applications using Java, Scala, or SQL, making it easier to integrate stream processing into existing data architectures. Confluent, a company that builds on Kafka, provides solutions and recipes to help organizations adopt stream processing for common use cases like fraud detection and predictive maintenance, enhancing their operational efficiency and data-driven decision-making.

Apache Kafka 3.2.0 has been released, featuring numerous new features and improvements. The release includes several key enhancements such as KRaft mode, improved security with the replacement of log4j 1.x with reload4j, and a built-in authorizer that does not depend on Zookeeper. Additionally, there are several performance optimizations, including the ability to set the pool size of network threads individually per listener, and improvements to the Kafka Streams API for better fault tolerance and easier maintenance. The release also includes fixes for various issues and compatibility updates. With these features and improvements, Apache Kafka 3.2.0 aims to provide a more robust and scalable platform for developers to build their applications on.

The text outlines a process for deploying Apache Kafka using Confluent and AWS Snowball to manage data in remote areas lacking internet connectivity. Traditionally, transferring large volumes of data from such isolated sites has been challenging, but by using the Confluent Platform in conjunction with AWS Snowball Edge, data can now be uploaded to the cloud and utilized for informed decision-making. The setup involves creating a Confluent Platform image, deploying connectors like the S3 Source Connector on AWS Fargate for serverless operations, and using AWS Snowball devices to transport data to designated S3 buckets. This hybrid approach allows for data processing at the edge or in the cloud, offering flexibility and cost optimization. Additionally, the blog highlights the availability of Confluent Platform 7.7, which includes enhanced security features and new integrations, and discusses using OpenSearch Ingestion to integrate Confluent with Amazon OpenSearch.

Walmart's real-time inventory use case is uniquely challenging due to its scale and complexity, with the system processing over tens of billions of messages from close to 100 million SKUs in less than three hours. The company leverages Apache Kafka as a key part of its cutting-edge IT architecture to achieve accurate, reliable, and real-time replenishment. To address the challenges of this use case, Walmart has implemented various architectural decisions, including active-passive data replication, resiliency mechanisms, Kafka retries, and fallbacks. The company also optimizes its producer and consumer configurations, such as custom partitioning strategies, linger.ms, batch sizes, and acks, to ensure data consistency and accuracy. Additionally, Walmart has designed a fallback mechanism for when messages cannot be sent to the topic itself, using a rest service to write directly to the database. The system also includes alerts and notifications to monitor its performance and ensure that it meets the company's SLAs.

Commercial buildings significantly contribute to global energy consumption and CO2 emissions, yet many fail to provide adequate indoor air quality and comfort. Aedifion aims to improve this by optimizing building operations using digital Building Automation Systems (BASs), which can reduce energy consumption by around 35% while enhancing user comfort. This approach involves leveraging data-driven monitoring, analytics, and control products to help stakeholders meet Environmental, Social, and Corporate Governance (ESG) goals. Aedifion's use of Confluent Cloud for its data pipeline allows for scalable, secure, and reliable data processing, accommodating various building types and data points. Apache Kafka is used as the messaging system due to its delivery guarantees and performance, with Confluent Cloud providing necessary integrations, scalability, and schema management. This setup has been successfully implemented in projects like the Kaiser Hof Cologne, demonstrating significant energy and CO2 savings. As aedifion plans to expand across hundreds of buildings, it continues to focus on data efficiency and sustainability in building management, facilitated by Confluent's capabilities in managing dynamic and continuous data streams.

Confluent Cloud and Confluent Platform 7.0 have introduced a streamlined feature for removing Apache Kafka brokers, aimed at simplifying the process of shrinking Confluent Server clusters. While the removal of Kafka brokers appears straightforward, the feature addresses several complexities, including the need to pre-compute capacity to prevent under-replicated partitions and ensure sufficient in-sync replicas (ISRs). The enhanced broker removal operation includes intelligent partition reassignment using the AlterPartitionReassignments API and relies on Confluent's Self-Balancing Clusters to redistribute partitions and maintain cluster balance. It also introduces a Broker Replica Exclusion API to prevent new replica placements on brokers slated for removal, ensuring a robust and error-free process. The operation is designed to be idempotent and resilient to failures, with state persistence in a Kafka topic, allowing for seamless recovery and continuation of the removal process. This feature not only enhances flexibility in managing cluster resources but also aligns with Kubernetes operational patterns, maintaining backward compatibility through a new conditional flag for broker shutdown.