September 2023 Summaries

Women represent roughly 28% of technology professionals, while their share of leadership roles has recently declined, prompting a Current 2023: The Next Generation of Kafka Summit panel to discuss ways to advance gender equity in the industry. Executives from Cardinal Health, Denny’s, AWS, Confluent, and Penske Transportation Solutions emphasized authentic, empathetic leadership, kindness, vulnerability, collaboration, and cognitively diverse teams as foundations for stronger organizations. Panelists highlighted mentorship and sponsorship as important tools for building confidence, providing guidance, and opening career opportunities, alongside employee resource groups, training, and early outreach to girls in middle and high school. They also encouraged women to build trusted networks of “safe people” who can offer candid advice and support through difficult decisions. Addressing the gender gap, they argued, requires active allyship—particularly from men—and collective advocacy for inclusive, equitable workplaces rather than treating gender equity as an adversarial issue.

SMS spam creates customer-service, network-capacity, and operating-cost challenges for telecommunications providers, while traditional content scanning can be costly, privacy-sensitive, and unable to prevent traffic from reaching the network. Early metadata-based, rule-driven detection systems used batch metrics such as message volume and size, but spammers adapted their behavior, making rule maintenance increasingly complex. Machine-learning systems improved detection and reduced manual rule updates, yet batch processing still left time windows that attackers could exploit before being blocked. The proposed real-time approach uses Confluent to ingest SMSC events, synchronize and encrypt supporting customer and database data, calculate streaming metrics and enrich records, and send features to a machine-learning model that assigns anomaly scores. A microservice can then block suspected SIM cards within seconds, while data may also be retained in Snowflake for reporting, forensic review, and periodic model retraining. Centralizing real-time and historical data can also improve detection by linking related signals, such as prepaid SIM cards purchased under the same sales identifier, and supports additional data-driven use cases across the organization.

The Data Streaming Awards highlight organizations using real-time data technologies to improve operations, customer experiences, and innovation across industries. Uber was named Streaming Company of the Year for using Apache Kafka to process 10 trillion messages daily with high reliability and low latency, supporting rides, deliveries, dynamic pricing, analytics, and communication. Penske won for its predictive-maintenance system, which processes roughly 190 million IoT messages each day to anticipate truck failures and is expected to prevent more than 90,000 roadside breakdowns annually. Wise received recognition for a company-wide platform built with Kafka Streams and Apache Flink that enables real-time financial processing for tens of millions of customers and approximately £27 billion in quarterly transfers. The U.S. Postal Service was honored for applying event-driven streaming to track more than 421 million mail pieces daily, modernize legacy operations, and rapidly support programs such as nationwide COVID-19 test-kit distribution. Startup PAUL Tech AG was recognized for connecting building sensors and actuators to streaming and machine-learning systems, aiming to reduce building energy consumption by up to 40 percent with limited infrastructure changes.

In Apache Kafka, consumer group strategy is crucial for efficient data processing and balancing workloads across partitions. To avoid hotspots and ensure parallelism, it's essential to consider the number of consumers in relation to the number of partitions, as each partition is assigned one thread per consumer. A single consumer group can lead to unavailability in case of unclean failure, whereas having multiple groups with separate configurations can improve scalability and fault tolerance. Proper configuration of consumer group IDs is vital for detecting new data, work sharing, and data recovery. The optimal approach depends on the specific use case, and careful consideration must be taken into account when designing a large piece of architecture.

The Current | The Next Generation of Kafka Summit showcased the latest trends, updates, and future developments in the worlds of Apache Kafka and Flink. Jay Kreps, original Kafka co-creator and Confluent CEO, emphasized the importance of data streaming becoming a fundamental part of data architecture for all types of data in every organization. Joe Foster, Cloud Computing Program Manager at NASA, discussed how NASA's GCN project uses data streaming to publish alerts in real-time from observatories worldwide. Danica Fine, Staff Developer Advocate at Confluent, highlighted the significance of Flink as a rising star in data processing, and Martijn Visser, Senior Product Manager at Confluent, cited Flink's broad set of APIs, low latency, high-throughput stream processing runtime, and robust community as reasons for its popularity. The summit also featured demos on data governance, stream processing with Flink, simplified protocol, Docker images & GraalVM support, queues for Kafka, and more. Industry leaders from major companies like BMW shared their experiences with Kafka, and the Confluent booth showcased Kora, a cloud-native 10x Kafka service. The event also included a Women in Tech panel discussion featuring powerful women executives who are committed to breaking down barriers for other women interested in working in tech.

Penske Transportation Solutions uses real-time data streaming and AI to improve vehicle uptime, supply-chain visibility, customer service, and operational efficiency across its fleet of more than 400,000 vehicles. With hundreds of sensors per vehicle, Penske processes roughly 190 million IoT messages daily through Confluent, feeding a proactive diagnostics engine that can predict maintenance needs and is expected to prevent more than 90,000 roadside breakdowns in a year. Telematics data also supports location tracking, roadside assistance, fuel-tax compliance, inspection reporting, and repair workflows, while managed Confluent services were selected to avoid the operational complexity of running and scaling open-source Kafka. Penske has also used AI for customer-facing automation, including its Erica virtual assistant for rental reservations, and is exploring generative AI while emphasizing controls for accuracy, cybersecurity, traceability, explainability, anomaly detection, and differing levels of business risk tolerance. As electric, connected, and autonomous vehicles expand, the company expects growing volumes of edge-generated data and greater use of real-time analytics for vehicle health, routing, asset management, freight monitoring, and customer experiences.

The Q3 Confluent Cloud Launch, presented at the Current 2023 event, highlighted major advancements in data streaming with a focus on the integration of Apache Flink, Kora Engine improvements, and AI applications. Drawing an audience of over 3,500 attendees, the event featured sessions on data streaming innovations, with keynotes from industry leaders including Confluent CEO Jay Kreps and NASA's Joe Foster. The launch introduced Confluent Cloud's new features such as an open preview of its fully managed Flink service on AWS, designed to simplify stream processing by eliminating infrastructure management complexities, and the introduction of the Kora engine-powered Enterprise clusters for secure, serverless Kafka operations. Additionally, Confluent Cloud now offers policy-driven governance through its Terraform provider, and the upcoming Data Portal aims to enhance data stream management. The event underscored the importance of efficient data replication and disaster recovery with advancements in Cluster Linking, alongside a reduction in storage costs and new audit log enhancements for Kafka. Attendees were encouraged to explore these innovations through a free trial and upcoming webinar.

Confluent is announcing its Data Streaming for AI initiative, expanding its ecosystem to include critical AI technologies and committing to a roadmap of product enhancements. This includes plans for fully managed connectors for the modern AI stack, GenAI API calls built into Apache Flink, and an AI Assistant to help build faster on Confluent. The initiative aims to solve data problems that hinder the development of successful AI strategies, connecting AI models to enterprise data in real time. Confluent's platform enables organizations to consolidate their operational and analytical data from disparate sources, transform raw data into actionable insights, and decouple data science tools and production AI applications. The company is partnering with various tech firms to provide real-time contextual data for vector search capabilities and is building on its relationships with major cloud providers to develop integrations and reference architectures specifically around AI. Confluent's platform delivers unparalleled elasticity, resiliency, and performance, freeing up valuable engineering time and achieving significant cost savings. The company is also implementing GenAI capabilities within its own platform, introducing the Confluent AI Assistant to help teams get contextual answers quickly.

Confluent Cloud for Apache Flink is a cloud-native service that re-imagines Flink as a truly serverless experience, offering effortless data filtering, joining, and enriching, high-performance stream processing at any scale, and fully integrated monitoring, security, and governance. The service provides a unified platform for Kafka and Flink, enabling seamless scalability, simplicity, and cost efficiency. It offers a cloud-native approach to unifying Flink with Kafka, separating compute from storage, and providing a secure suite of capabilities to control, manage, and govern access to data. With Confluent Cloud, developers can focus on app development, not complex infrastructure-related tasks, and take advantage of a curated set of metrics to simplify the process of monitoring their workloads.

McAfee, a leading provider of online protection, has transitioned from managing multiple data streaming technologies like open-source Kafka to using Confluent Cloud to address the complexities of cybercrime and enhance real-time data capabilities. This shift supports McAfee's cloud-native modernization and microservices architecture, allowing the company to scale, improve performance, and focus on product development without being bogged down by infrastructure management. Confluent Cloud enables McAfee to streamline operations, decouple systems, and efficiently handle high-volume data ingestion across millions of devices, providing a centralized platform that supports various real-time use cases, including financial reporting and customer communications. The move has reduced operational burdens, reallocated resources towards innovation, and ensured high availability and security, positioning McAfee to continue growing in the digital age while maintaining flexibility and scalability in its data operations.

US Foods, a Chicago-area food distributor with both B2B operations and more than 70 retail locations, is using Apache Kafka and Confluent Cloud to support a digital transformation centered on real-time, event-driven services. Its Digital Platform Services team built a data mesh architecture to connect legacy mainframes, SQL and Oracle databases, Snowflake, MongoDB Atlas, and microservices as the company launches and migrates customers to its new MOXe e-commerce platform. After initially testing self-managed open-source Kafka and later Amazon MSK, US Foods adopted Confluent Cloud to reduce operational workload for its eight-person platform team while gaining managed connectors, scalability, reliability, stream governance, role-based access controls, and observability. The company says the platform helps it handle low-latency data flows, scale on demand, maintain service-level commitments, and integrate a diverse technology environment without dedicating extensive resources to Kafka administration.

AI and machine learning applications depend on reliable, high-quality real-time and historical data, and data streaming can support their deployment by enabling continuous model training, persistent synchronization between source systems and ML platforms, real-time model inference, and high-volume processing. The material highlights the growing production use of AI and presents a collection of learning resources on the convergence of machine learning and streaming, including discussions of technical and organizational barriers, real-time generative AI with GPT-4, retrieval-augmented generation, abstractions for real-time ML platforms, migrations from batch to streaming ML, and online learning with tools such as Kafka, River, and Bytewax. It also directs readers to recorded sessions from Confluent’s Current 2023 event and additional GenAI resources.

Confluent Cloud and AWS Lambda can be used to build scalable, fault-tolerant event-driven architectures. Confluent provides a streaming SaaS solution based on Apache Kafka, while AWS Lambda offers a serverless compute service that abstracts the need to provision, operate, and scale underlying infrastructure. To integrate Confluent with AWS Lambda, developers can use two patterns: the Fully managed AWS Lambda Sink Connector and the Native event source mapping (ESM). The connector supports high throughput and low latency, but may limit ordering guarantees, while ESM ensures ordering guarantees but may limit throughput and increase end-to-end latency. Best practices for running an event-driven Confluent and Lambda solution include using batching controls to save on lambda invocation costs, implementing idempotent consumption patterns, and establishing long-lived connections outside the function handler to reduce cold start issues. Additionally, schemas and Schema Registry are crucial for data integrity and ensuring seamless communication between microservices.

The text discusses how a grocery delivery company addressed technical challenges and improved its same-day delivery service by transitioning from Apache Kafka on Azure HDInsight to Confluent Cloud, a fully-managed, cloud-native Kafka service. Previously, the company faced significant issues with maintaining its data streaming infrastructure, which led to frequent outages and hindered innovation. The switch to Confluent Cloud, powered by the Kora engine, provided a more reliable and high-availability platform with features such as multi-AZ dedicated clusters, pre-built connectors, and a Schema Registry for data consistency. This transition reduced maintenance burdens and allowed the company's data team to focus on enhancing applications and delivering new features, ultimately improving customer satisfaction and operational efficiency.

Apache Kafka supports incredibly high throughput by partitioning topics and allowing producers to send multiple messages to different partitions simultaneously. Compression can also improve throughput and efficiency by reducing network traffic, but it's essential to consider the compression type and batch size to achieve optimal performance. The producer's compression type should be set to match the topic-level compression type whenever possible, as brokers will recompress data if they don't. Compressed data must be decompressed by consumers, which can impact CPU, network, and disk utilization. It's crucial to coordinate compression types across producers to achieve optimal end-to-end performance. Additionally, encrypted data should not be compressed, and default configuration settings should be double-checked for consistency between client and original distribution.

Confluent announces a Financial Services Data Streaming Resource Center designed to make information about data streaming in banking and finance more accessible. The hub includes on-demand virtual events, white papers, customer case studies, AI and machine-learning content, and information about upcoming events, with examples such as EVO Banco’s fraud-prevention work and 10X Banking’s digital core banking platform. Access requires completing a single form, after which users may receive occasional emails about related content and events. The announcement also promotes a September 28 webinar with Thrivent Financial and Improving on using data streaming to improve omnichannel customer experiences, while positioning the resource center as a source of guidance for real-time payments, risk management, fraud prevention, and other financial-services use cases.

An individual with experience in streaming technologies shares their journey of learning Apache Flink, highlighting the challenges faced due to limited realistic learning resources and the lack of expert guidance. To address these issues, they developed a comprehensive course titled "Building Flink Applications in Java" on Confluent Developer, along with a blog post that details the process of dataflow programming with Flink. The blog post provides a practical example of building a Java pipeline that processes clickstream data from Apache Kafka, demonstrating key operations like grouping, windowing, and serialization. The implementation involves setting up a Maven project, configuring Kafka for message consumption and production, and utilizing Flink's capabilities to manage data streams and produce analytics. Although the author acknowledges areas for further exploration, such as state management and branching flows, they encourage others to delve deeper through the course and accompanying GitHub repository, providing video tutorials for a complete understanding of consuming and producing Kafka messages using Flink and Java.

As global demand for tradespeople rises alongside infrastructure needs, the challenges faced by tradespeople in job management, scheduling, and communication with customers have become increasingly complex, necessitating innovative solutions. The blog discusses how a major service marketplace addressed these issues by transitioning from a monolithic architecture to a microservices-based platform using Confluent Cloud, a managed event streaming platform built on Apache Kafka. This shift enabled real-time data processing, efficient communication between tradespeople and customers, and seamless integration of job postings, scheduling, quotes, payments, and feedback. The platform now provides a scalable, responsive ecosystem that enhances customer experience through features like GPS tracking and real-time updates, while streamlining operations for tradespeople. By leveraging Confluent Cloud, the marketplace has improved scalability, data integrity, and developer productivity, ultimately fostering better connectivity and driving revenue growth.

Confluent has launched the Data Streaming Startup Challenge, a global competition to recognize early-stage startups that are putting data streaming at the heart of their businesses. The challenge aims to inspire and fuel the growth of startups across the data technology landscape with a grand prize of up to $500,000 investment and two runner-up prizes of up to $250,000 each. Confluent for Startups provides cost-free opportunities for early-stage companies to build their data architecture without compromising on scalability or cost-effectiveness. The competition is open to startups from around the world that meet specific eligibility criteria, including being established within the last five years and using Confluent Cloud for their data streaming infrastructure.

[Flink SQL] is a powerful data processing engine that allows developers to process and analyze large volumes of data in real time using ANSI standard compliant SQL syntax. It provides users with a declarative way to express data transformations and analytics on streams of data, supporting various SQL operations such as filtering, aggregating, joining, and windowing. Flink SQL is highly interoperable with other Flink APIs, allowing developers to use one or many APIs depending on their requirements. Its unified API for batch and stream processing enables users to apply features to both bounded and unbounded data, making it well-suited for complex data processing tasks. With its advanced optimization techniques, Flink SQL ensures efficient query execution and minimal resource usage, even in complex scenarios. It offers various joins, aggregations, and windowing capabilities, including temporal joins, lateral joins, tumbling windows, hopping windows, and cumulative windows. Flink SQL's streaming mode has optimizations for temporal joins that take advantage of the time-based nature of the data, making them more efficient than regular joins. When choosing between streaming and batch processing modes in Flink SQL, consider the nature of your data and the type of processing needed to perform.

A financial institution modernized its customer onboarding process with Confluent Cloud to address high customer expectations, regulatory requirements, manual workflows, fragmented data, limited scalability, security risks, and a lack of real-time insight. The organization selected Kafka-based event streaming over RabbitMQ because it required durable, scalable event processing, replayability, and integration across numerous systems. Its architecture connects microservices for registration, identity verification, credit checks, account creation, welcome communications, notifications, and customer support, with each service producing and consuming events through Confluent Cloud. Customer data is also synchronized in real time to Salesforce, PostgreSQL, and Snowflake for CRM, transaction processing, and analytics. The implementation uses managed Kafka, Schema Registry, connectors, and ksqlDB to support schema management, integrations, stream processing, and real-time analytics. According to the account, the resulting system reduced operational costs, accelerated onboarding and feature delivery, improved customer experiences, and provided a scalable foundation for further digital services.

Bans Sagoo, a Commercial Solutions Engineering Manager at Confluent, joined the company more than two years ago to move into the data streaming field and work with its technology, developers, architects, and experienced colleagues. After initially serving as a Solutions Engineer, he moved into management, supporting the UK and other EMEA regions by helping his team succeed autonomously and grow professionally. He is collaborating with regional leaders to create a global enablement program that develops practical soft skills and field learning for solutions engineers, with the aim of improving both employee development and customer outcomes. Sagoo describes his work as fast-paced and varied, including customer meetings, account strategy, team development, cross-regional initiatives, and platform learning. He characterizes Confluent’s culture as open, collaborative, and centered on knowledge sharing, while advocating for expanded community engagement and education efforts as part of the company’s ESG initiatives.

Kmart Australia Limited has implemented a digital loyalty program called OnePass, which uses data streaming to power its seamless in-store and online commerce experience. The company's Principal Architect - Enterprise Technology, Duane Gomes, discussed how the team used Confluent Cloud to stream processing capabilities, enabling real-time interaction with customers through digital receipts that contain offers and promotional deals. By combining purchasing data from various sources, including the point-of-sale system and a data lake, Kmart creates a rich dataset for analysis and personalization. The company aims to keep costs low while innovating with technology, leveraging Confluent Cloud's capabilities to unlock valuable insights into customer behavior and optimize its operations.

Apache Kafka can be run on Windows for learning and experimentation by using Windows Subsystem for Linux 2 (WSL 2), though Windows is not recommended for production workloads. The process involves installing and configuring WSL 2 with Ubuntu, updating the Linux environment, installing a supported Java version such as Java 11 or 17, downloading and extracting Kafka, generating a KRaft cluster ID, formatting storage, and starting a Kafka broker. Users can then validate the installation by creating a topic and exchanging messages through Kafka’s console producer and consumer tools before stopping the services and deleting temporary log directories. WSL 2 is preferred over direct Windows or WSL 1 installations because Kafka depends on POSIX features and a real Linux kernel for reliable operation, while Docker Desktop using its WSL 2 backend is another practical option. The broader Confluent Platform ecosystem, including tools such as ksqlDB, Schema Registry, and Control Center, is available for experimentation on Windows but is not supported there for production or development environments.

At Confluent’s Singapore Data in Motion Tour, co-founder Jun Rao discussed Kafka’s origins and the growing need for systems that can manage rapidly expanding volumes of real-time digital data without complex integrations or data silos, while noting the operational cost and complexity of self-managed Kafka. Kredivo Holdings, a Southeast Asian fintech provider whose Buy Now Pay Later product relies on automated credit scoring, described how its rapid growth made Kafka difficult to maintain internally. Senior Lead Data Engineering and Analytics Adityo August said that adopting Confluent initially addressed one issue but ultimately improved broader operational challenges, reportedly reducing monetary resource use by 40% and accelerating processes. The event positioned Confluent’s data-streaming platform as a way for financial-services organizations to support event-driven applications and use cases such as real-time payments, risk management, and fraud prevention.

The insurance industry is undergoing rapid technological transformation to meet evolving consumer expectations, with innovations like personalized discounts via IoT, drone-assisted damage assessments, AI chatbots, and proactive health recommendations becoming prevalent. This shift is mirrored by a push for automation and real-time data integration in backend systems, addressing needs such as fraud detection, dynamic pricing, and ML-powered risk analysis. Apache Kafka plays a critical role in this evolution, particularly as Insurtech companies combine insurance expertise with modern technology to offer seamless digital experiences and personalized products, like cyber insurance. Confluent Cloud facilitates a data-driven approach by enabling real-time data streaming and processing through tools like CDC Connectors and ksqlDB, allowing Insurtechs to deliver instant insurance quotes and improve customer engagement. As both legacy and new companies strive to modernize, data streaming becomes essential for innovation and operational efficiency, with Confluent Cloud offering scalable and secure solutions for real-time analytics in the highly regulated insurance sector.

This summary provides an overview of deploying Apache Kafka in a multi-region architecture. Confluent Platform simplifies the process of connecting data sources to Apache Kafka, building streaming applications, securing, monitoring, and managing Kafka infrastructure. To deploy Apache Kafka on AWS EC2 machines, one can use either Elastic Block Storage (EBS) or instance storage. EBS provides consistent I/O performance and flexibility but adds cost, while instance storage is more cost-effective but takes longer to recover from a failed broker. The recommended practice when replacing a Kafka broker is to use the broker ID from the failed broker in the replacement broker. To ensure high availability, it's essential to deploy Kafka clusters with multi-availability zone configurations or stretch clusters across regions. Confluent offers its own implementation of the Kubernetes Operator API for automated provisioning and management of Kafka on Kubernetes, called Confluent for Kubernetes (CFK). CFK provides a standard interface to customize, deploy, and manage Confluent Platform through declarative APIs.