September 2023 Summaries

In Apache Kafka, consumer group strategy is crucial for efficient data processing and balancing workloads across partitions. To avoid hotspots and ensure parallelism, it's essential to consider the number of consumers in relation to the number of partitions, as each partition is assigned one thread per consumer. A single consumer group can lead to unavailability in case of unclean failure, whereas having multiple groups with separate configurations can improve scalability and fault tolerance. Proper configuration of consumer group IDs is vital for detecting new data, work sharing, and data recovery. The optimal approach depends on the specific use case, and careful consideration must be taken into account when designing a large piece of architecture.

The Current | The Next Generation of Kafka Summit showcased the latest trends, updates, and future developments in the worlds of Apache Kafka and Flink. Jay Kreps, original Kafka co-creator and Confluent CEO, emphasized the importance of data streaming becoming a fundamental part of data architecture for all types of data in every organization. Joe Foster, Cloud Computing Program Manager at NASA, discussed how NASA's GCN project uses data streaming to publish alerts in real-time from observatories worldwide. Danica Fine, Staff Developer Advocate at Confluent, highlighted the significance of Flink as a rising star in data processing, and Martijn Visser, Senior Product Manager at Confluent, cited Flink's broad set of APIs, low latency, high-throughput stream processing runtime, and robust community as reasons for its popularity. The summit also featured demos on data governance, stream processing with Flink, simplified protocol, Docker images & GraalVM support, queues for Kafka, and more. Industry leaders from major companies like BMW shared their experiences with Kafka, and the Confluent booth showcased Kora, a cloud-native 10x Kafka service. The event also included a Women in Tech panel discussion featuring powerful women executives who are committed to breaking down barriers for other women interested in working in tech.

The Q3 Confluent Cloud Launch, presented at the Current 2023 event, highlighted major advancements in data streaming with a focus on the integration of Apache Flink, Kora Engine improvements, and AI applications. Drawing an audience of over 3,500 attendees, the event featured sessions on data streaming innovations, with keynotes from industry leaders including Confluent CEO Jay Kreps and NASA's Joe Foster. The launch introduced Confluent Cloud's new features such as an open preview of its fully managed Flink service on AWS, designed to simplify stream processing by eliminating infrastructure management complexities, and the introduction of the Kora engine-powered Enterprise clusters for secure, serverless Kafka operations. Additionally, Confluent Cloud now offers policy-driven governance through its Terraform provider, and the upcoming Data Portal aims to enhance data stream management. The event underscored the importance of efficient data replication and disaster recovery with advancements in Cluster Linking, alongside a reduction in storage costs and new audit log enhancements for Kafka. Attendees were encouraged to explore these innovations through a free trial and upcoming webinar.

Confluent is announcing its Data Streaming for AI initiative, expanding its ecosystem to include critical AI technologies and committing to a roadmap of product enhancements. This includes plans for fully managed connectors for the modern AI stack, GenAI API calls built into Apache Flink, and an AI Assistant to help build faster on Confluent. The initiative aims to solve data problems that hinder the development of successful AI strategies, connecting AI models to enterprise data in real time. Confluent's platform enables organizations to consolidate their operational and analytical data from disparate sources, transform raw data into actionable insights, and decouple data science tools and production AI applications. The company is partnering with various tech firms to provide real-time contextual data for vector search capabilities and is building on its relationships with major cloud providers to develop integrations and reference architectures specifically around AI. Confluent's platform delivers unparalleled elasticity, resiliency, and performance, freeing up valuable engineering time and achieving significant cost savings. The company is also implementing GenAI capabilities within its own platform, introducing the Confluent AI Assistant to help teams get contextual answers quickly.

Confluent Cloud for Apache Flink is a cloud-native service that re-imagines Flink as a truly serverless experience, offering effortless data filtering, joining, and enriching, high-performance stream processing at any scale, and fully integrated monitoring, security, and governance. The service provides a unified platform for Kafka and Flink, enabling seamless scalability, simplicity, and cost efficiency. It offers a cloud-native approach to unifying Flink with Kafka, separating compute from storage, and providing a secure suite of capabilities to control, manage, and govern access to data. With Confluent Cloud, developers can focus on app development, not complex infrastructure-related tasks, and take advantage of a curated set of metrics to simplify the process of monitoring their workloads.

McAfee, a leading provider of online protection, has transitioned from managing multiple data streaming technologies like open-source Kafka to using Confluent Cloud to address the complexities of cybercrime and enhance real-time data capabilities. This shift supports McAfee's cloud-native modernization and microservices architecture, allowing the company to scale, improve performance, and focus on product development without being bogged down by infrastructure management. Confluent Cloud enables McAfee to streamline operations, decouple systems, and efficiently handle high-volume data ingestion across millions of devices, providing a centralized platform that supports various real-time use cases, including financial reporting and customer communications. The move has reduced operational burdens, reallocated resources towards innovation, and ensured high availability and security, positioning McAfee to continue growing in the digital age while maintaining flexibility and scalability in its data operations.

Confluent Cloud and AWS Lambda can be used to build scalable, fault-tolerant event-driven architectures. Confluent provides a streaming SaaS solution based on Apache Kafka, while AWS Lambda offers a serverless compute service that abstracts the need to provision, operate, and scale underlying infrastructure. To integrate Confluent with AWS Lambda, developers can use two patterns: the Fully managed AWS Lambda Sink Connector and the Native event source mapping (ESM). The connector supports high throughput and low latency, but may limit ordering guarantees, while ESM ensures ordering guarantees but may limit throughput and increase end-to-end latency. Best practices for running an event-driven Confluent and Lambda solution include using batching controls to save on lambda invocation costs, implementing idempotent consumption patterns, and establishing long-lived connections outside the function handler to reduce cold start issues. Additionally, schemas and Schema Registry are crucial for data integrity and ensuring seamless communication between microservices.

The text discusses how a grocery delivery company addressed technical challenges and improved its same-day delivery service by transitioning from Apache Kafka on Azure HDInsight to Confluent Cloud, a fully-managed, cloud-native Kafka service. Previously, the company faced significant issues with maintaining its data streaming infrastructure, which led to frequent outages and hindered innovation. The switch to Confluent Cloud, powered by the Kora engine, provided a more reliable and high-availability platform with features such as multi-AZ dedicated clusters, pre-built connectors, and a Schema Registry for data consistency. This transition reduced maintenance burdens and allowed the company's data team to focus on enhancing applications and delivering new features, ultimately improving customer satisfaction and operational efficiency.

Apache Kafka supports incredibly high throughput by partitioning topics and allowing producers to send multiple messages to different partitions simultaneously. Compression can also improve throughput and efficiency by reducing network traffic, but it's essential to consider the compression type and batch size to achieve optimal performance. The producer's compression type should be set to match the topic-level compression type whenever possible, as brokers will recompress data if they don't. Compressed data must be decompressed by consumers, which can impact CPU, network, and disk utilization. It's crucial to coordinate compression types across producers to achieve optimal end-to-end performance. Additionally, encrypted data should not be compressed, and default configuration settings should be double-checked for consistency between client and original distribution.

An individual with experience in streaming technologies shares their journey of learning Apache Flink, highlighting the challenges faced due to limited realistic learning resources and the lack of expert guidance. To address these issues, they developed a comprehensive course titled "Building Flink Applications in Java" on Confluent Developer, along with a blog post that details the process of dataflow programming with Flink. The blog post provides a practical example of building a Java pipeline that processes clickstream data from Apache Kafka, demonstrating key operations like grouping, windowing, and serialization. The implementation involves setting up a Maven project, configuring Kafka for message consumption and production, and utilizing Flink's capabilities to manage data streams and produce analytics. Although the author acknowledges areas for further exploration, such as state management and branching flows, they encourage others to delve deeper through the course and accompanying GitHub repository, providing video tutorials for a complete understanding of consuming and producing Kafka messages using Flink and Java.

As global demand for tradespeople rises alongside infrastructure needs, the challenges faced by tradespeople in job management, scheduling, and communication with customers have become increasingly complex, necessitating innovative solutions. The blog discusses how a major service marketplace addressed these issues by transitioning from a monolithic architecture to a microservices-based platform using Confluent Cloud, a managed event streaming platform built on Apache Kafka. This shift enabled real-time data processing, efficient communication between tradespeople and customers, and seamless integration of job postings, scheduling, quotes, payments, and feedback. The platform now provides a scalable, responsive ecosystem that enhances customer experience through features like GPS tracking and real-time updates, while streamlining operations for tradespeople. By leveraging Confluent Cloud, the marketplace has improved scalability, data integrity, and developer productivity, ultimately fostering better connectivity and driving revenue growth.

Confluent has launched the Data Streaming Startup Challenge, a global competition to recognize early-stage startups that are putting data streaming at the heart of their businesses. The challenge aims to inspire and fuel the growth of startups across the data technology landscape with a grand prize of up to $500,000 investment and two runner-up prizes of up to $250,000 each. Confluent for Startups provides cost-free opportunities for early-stage companies to build their data architecture without compromising on scalability or cost-effectiveness. The competition is open to startups from around the world that meet specific eligibility criteria, including being established within the last five years and using Confluent Cloud for their data streaming infrastructure.

[Flink SQL] is a powerful data processing engine that allows developers to process and analyze large volumes of data in real time using ANSI standard compliant SQL syntax. It provides users with a declarative way to express data transformations and analytics on streams of data, supporting various SQL operations such as filtering, aggregating, joining, and windowing. Flink SQL is highly interoperable with other Flink APIs, allowing developers to use one or many APIs depending on their requirements. Its unified API for batch and stream processing enables users to apply features to both bounded and unbounded data, making it well-suited for complex data processing tasks. With its advanced optimization techniques, Flink SQL ensures efficient query execution and minimal resource usage, even in complex scenarios. It offers various joins, aggregations, and windowing capabilities, including temporal joins, lateral joins, tumbling windows, hopping windows, and cumulative windows. Flink SQL's streaming mode has optimizations for temporal joins that take advantage of the time-based nature of the data, making them more efficient than regular joins. When choosing between streaming and batch processing modes in Flink SQL, consider the nature of your data and the type of processing needed to perform.

Kmart Australia Limited has implemented a digital loyalty program called OnePass, which uses data streaming to power its seamless in-store and online commerce experience. The company's Principal Architect - Enterprise Technology, Duane Gomes, discussed how the team used Confluent Cloud to stream processing capabilities, enabling real-time interaction with customers through digital receipts that contain offers and promotional deals. By combining purchasing data from various sources, including the point-of-sale system and a data lake, Kmart creates a rich dataset for analysis and personalization. The company aims to keep costs low while innovating with technology, leveraging Confluent Cloud's capabilities to unlock valuable insights into customer behavior and optimize its operations.

The insurance industry is undergoing rapid technological transformation to meet evolving consumer expectations, with innovations like personalized discounts via IoT, drone-assisted damage assessments, AI chatbots, and proactive health recommendations becoming prevalent. This shift is mirrored by a push for automation and real-time data integration in backend systems, addressing needs such as fraud detection, dynamic pricing, and ML-powered risk analysis. Apache Kafka plays a critical role in this evolution, particularly as Insurtech companies combine insurance expertise with modern technology to offer seamless digital experiences and personalized products, like cyber insurance. Confluent Cloud facilitates a data-driven approach by enabling real-time data streaming and processing through tools like CDC Connectors and ksqlDB, allowing Insurtechs to deliver instant insurance quotes and improve customer engagement. As both legacy and new companies strive to modernize, data streaming becomes essential for innovation and operational efficiency, with Confluent Cloud offering scalable and secure solutions for real-time analytics in the highly regulated insurance sector.

This summary provides an overview of deploying Apache Kafka in a multi-region architecture. Confluent Platform simplifies the process of connecting data sources to Apache Kafka, building streaming applications, securing, monitoring, and managing Kafka infrastructure. To deploy Apache Kafka on AWS EC2 machines, one can use either Elastic Block Storage (EBS) or instance storage. EBS provides consistent I/O performance and flexibility but adds cost, while instance storage is more cost-effective but takes longer to recover from a failed broker. The recommended practice when replacing a Kafka broker is to use the broker ID from the failed broker in the replacement broker. To ensure high availability, it's essential to deploy Kafka clusters with multi-availability zone configurations or stretch clusters across regions. Confluent offers its own implementation of the Kubernetes Operator API for automated provisioning and management of Kafka on Kubernetes, called Confluent for Kubernetes (CFK). CFK provides a standard interface to customize, deploy, and manage Confluent Platform through declarative APIs.