April 2023 Summaries

In this blog series on understanding your Kafka costs, we discuss how to assess the development and operations costs associated with running Apache Kafka® clusters. We cover major activities that drive these costs, such as capacity planning, integrating additional features beyond the core Kafka broker, scaling the cluster, upgrading and patching regularly, and managing downtime and security risks. Additionally, we explore how moving to a fully managed Kafka service can help reduce these costs and improve efficiency.

EVO Banco, a 100% digital banking model in Spain's financial services industry, aims to create modern banking solutions that simplify customers' lives while using innovative technology. To achieve this goal, the bank chose Confluent Cloud for its data infrastructure and uses it to manage large volumes of data in real-time, delivering a better customer experience. EVO Banco's Chief Data Officer, Jose Enrique Perez, discussed how the bank uses machine learning to prevent fraud and improve the customer experience in real-time with Confluent's Duncan Ash. The bank is using Confluent Cloud to power its advanced fraud detection system, which combines behavior analysis, data analytics, and predictive modeling to protect against financial crime without compromising the customer experience. By adopting Confluent Cloud, EVO Banco has significantly increased the accuracy of its fraud algorithms and reduced reaction times to just seconds, blocking an average of 500 fraudulent transactions daily and reducing weekly fraud losses by 99%.

The Two Generals Problem highlights the challenge of achieving reliable communication in distributed systems, which is also relevant to science fiction concepts like teleportation. The problem arises when using unreliable channels, making it impossible to guarantee message delivery or success. This has consequences in various fields, including software development, where developers must make decisions about how to handle message duplication and failures. To address this issue, Confluent's Apache Kafka provides transactional guarantees, enabling developers to build robust event streaming applications with high-throughput and reliability.

Kafka Summit London 2023, held at ExCeL London, offered two days of Apache Kafka-focused programming including technical breakouts, panels, lightning talks, community meetups, and more than 60 sessions across five simultaneous tracks. Program Committee Chair Danica Fine advised attendees to choose sessions by technology interests, industries, and companies, highlighting topics such as Kafka Streams, Kafka Connect exactly-once semantics, Apache Flink pipelines, event-driven tracing, and real-world implementations. The event also expanded networking and learning opportunities through a 5K fun run, a community meetup hub, Kafka fundamentals training and certification, a day-one party featuring live-coded music, and Birds-of-a-Feather lunches for Women in Technology and Financial Services. Attendees could follow and share updates online using #KafkaSummit, while virtual registrants could access livestreamed keynotes and a session track; recordings would later be made freely available.

The text discusses the complexities and considerations involved in managing Apache Kafka infrastructure costs, particularly in the context of cloud services. It highlights that while compute and storage are commonly scrutinized expenses, networking emerges as the most significant and often overlooked cost driver, particularly due to cross-availability zone (AZ) traffic. The blog post is part of a series by Confluent, which aims to guide users in optimizing their Kafka-related expenditures, touching on aspects such as infrastructure, development, and operations resources. By using Confluent's managed services, many users reportedly find cost efficiencies, as the company has invested in architectural optimizations and partnerships to reduce costs related to networking, storage, and compute. The series promises further insights into cost management strategies for Kafka, with upcoming discussions focusing on development and operational costs.

Steven Zhang joined Confluent after graduating from Johns Hopkins University in 2019, seeking technically challenging work, and has since progressed to senior software engineer in the Stream Processing and Analytics organization. After spending three years developing ksqlDB features including SQL lambda syntax, metastore recovery, and faster provisioning, he moved to the Flink platform team to help build cloud infrastructure for Confluent’s managed Flink offering. His current work focuses on enabling upgrades with minimal or no query downtime, automatic rollback when upgrades fail, operational monitoring, autoscaling, and failure reporting to central fleet-management tools. Zhang describes Confluent as a remote-first, globally distributed workplace where teams coordinate across time zones through flexible scheduling, asynchronous code reviews, meeting-free periods, and virtual social activities. He values the company’s technical culture, cross-team collaboration, broad exposure to technologies, and annual team meetups, while suggesting that more cross-organizational events could strengthen belonging among remote employees.

Shaun Clowes, Confluent’s Chief Product Officer, argues that organizations need to treat data as a live, real-time asset rather than as historical information stored in warehouses, since stale or poorly synchronized data can undermine competitiveness and customer experiences. Drawing on prior roles at MuleSoft and Atlassian, he says Confluent’s streaming approach helps decouple applications, improve developer productivity, and enable teams to reuse and enrich data products instead of repeatedly extracting and reshaping data across fragmented systems. He identifies governance, discovery, secure access, connectors, and pipeline-building tools as key components of Confluent’s platform, built on Apache Kafka, and highlights the planned integration of Apache Flink following Confluent’s acquisition of Immerok to expand stream-processing capabilities across batch and real-time workloads. Confluent’s broader goal is to help customers create a real-time, distributed source of truth that connects their many systems into a more coherent data ecosystem.

At Current 2022, data streaming professionals highlighted a shift from broad digital transformation initiatives toward data transformation centered on real-time, shareable, governed, and self-service data. Organizations are increasingly encouraged to treat data as a product and first-class asset by ensuring its quality, consistency, lineage, discoverability, and secure accessibility across operational, analytical, and SaaS systems. However, survey findings cited in the discussion indicate that while trusted data is linked to improved customer satisfaction, few organizations have complete architectures for data management and control, and streaming data remains among the least trusted sources. Legacy batch-oriented, point-to-point pipelines are also struggling to meet growing real-time requirements because they are difficult to scale, reuse, govern, and maintain. The discussion emphasizes that modern streaming platforms and stronger governance can help teams spend less time managing infrastructure and more time developing applications, delivering use cases, and driving innovation.

This error occurs when the data formats of the publishers (Kafka producers) do not match the data formats of the subscribers (Kafka consumers), causing an "unknown magic byte" error. The Schema Registry solves this problem by validating messages before sending them, but reconciliation is necessary to ensure serialization methods are consistent on both production and consumption ends. This can be done in clients, ksqlDB, or Kafka Streams, depending on the project. A processing log can provide metadata about instances, including errors with "unknown magic byte" strings. Setting up an Avro schema in the client and registering it in Confluent Cloud can resolve this issue. Double-checking serialization setup and ensuring schemas are registered correctly is also crucial. If all else fails, deleting and re-producing messages may be necessary. The "unknown magic byte" error highlights the importance of data consistency and format validation in Kafka Streams.

The Biden-Harris administration’s March 2023 National Cybersecurity Strategy outlines five pillars for improving U.S. cyber defense: protecting critical infrastructure, disrupting threat actors, using market forces to improve security, investing in resilience, and strengthening international partnerships. A central theme is faster coordination and information sharing among government agencies, infrastructure operators, and other stakeholders whose cybersecurity data is often isolated in separate systems. The post argues that real-time data streaming, including Confluent’s platform, can help integrate telemetry, logs, network activity, application data, and user behavior across organizations, enabling quicker threat detection and response while potentially reducing retention and processing costs. It also highlights the security implications of connected infrastructure and IoT devices, the value of event-driven actions rather than retrospective analysis alone, and the use of Sigma’s shareable detection rules to distribute emerging threat intelligence across agencies and across both real-time and traditional security tools.

Spring 2023 data streaming resource roundup presents learning materials for understanding Apache Kafka, ksqlDB, streaming pipelines, decentralized data architectures, managed services, practical use cases, and community-based skill building. It recommends introductory Kafka content covering events, topics, partitions, producers, consumers, performance, and streaming pipeline design, with ksqlDB positioned as a processing layer that prepares data for downstream systems. The roundup also examines data mesh as a decentralized model based on domain ownership, data products, self-service platforms, and federated governance, while describing managed offerings such as Confluent Cloud as tools for simplifying mission-critical deployments through managed connectors and scalable storage. A retail example illustrates how event streaming can synchronize inventory and orders, and the 100 Days of Code for Apache Kafka challenge encourages hands-on practice and peer engagement through social media and broader exploration of related technologies.

The text discusses the integration of machine learning (ML) and data streaming, emphasizing the role of SQL as a common language across programming ecosystems and highlighting the benefits and limitations of using SQL for streaming solutions. It explores how SQL facilitates the creation of directed acyclic graphs (DAGs) for declarative programming but notes its lack of composability for complex analytics. The text also describes various approaches to bridging the language gap between Python and Java ecosystems, such as using libraries like Faust for Python and Kafka Streams for Java, and highlights the challenges of integrating ML experiment management platforms with Kafka. It showcases examples of real-time ML applications, such as Centrica Hive's ML-powered microservices and Babylon's health risk prediction system. The text concludes by asserting the potential and readiness of the community to integrate ML and streaming solutions, inviting developers to share their experiences and solutions on community forums.

At Mr. Cooper, a home mortgage company, they have built a powerful multicloud streaming platform using Apache Kafka and its ecosystem of integration and processing tools. The company has made data streaming the default platform for all real-time needs for its various business applications. They use Confluent's Kafka Streams and ksqlDB to perform stream processing, leveraging the platform's pub-sub and streaming processing capabilities. The goal is to provide real-time notifications when customers make payments or submit mortgage applications, reducing call volumes and increasing customer satisfaction. Mr. Cooper has also implemented a real-time streaming platform engineering team that employs stream processing extensively in its operations, with nearly 100 managed and self-managed connectors enabling seamless data integration across systems. They are looking forward to implementing more Confluent-managed connectors to increase the number of producer and consumer systems. The company's vision for the future is to create an event ingestion and discovery platform that allows internal users to access data and see how the market fluctuates and how customers respond to different products and services. Mr. Cooper recognizes the importance of data governance, especially when handling financial data, and has implemented Cluster Linking to migrate business events, consumer groups, and schemas without downtime. The company's leadership encourages a change in mindset towards data streaming, recommending identifying a data streaming use case that solves a challenging problem for the business and finding a business partner to implement it. They also emphasize the importance of building a minimum viable product for high-priority use cases and demonstrating the TCO and ROI compared to current processes. To govern data effectively without becoming a blocker, Mr. Cooper has built a centrally governed but federated partnership model with application engineering teams, establishing clear guidelines and standards, implementing an event catalog and governance process, fostering collaboration with different application teams, monitoring and optimizing performance, and enabling observability and monitoring.