April 2023 Summaries

In this blog series on understanding your Kafka costs, we discuss how to assess the development and operations costs associated with running Apache Kafka® clusters. We cover major activities that drive these costs, such as capacity planning, integrating additional features beyond the core Kafka broker, scaling the cluster, upgrading and patching regularly, and managing downtime and security risks. Additionally, we explore how moving to a fully managed Kafka service can help reduce these costs and improve efficiency.

EVO Banco, a 100% digital banking model in Spain's financial services industry, aims to create modern banking solutions that simplify customers' lives while using innovative technology. To achieve this goal, the bank chose Confluent Cloud for its data infrastructure and uses it to manage large volumes of data in real-time, delivering a better customer experience. EVO Banco's Chief Data Officer, Jose Enrique Perez, discussed how the bank uses machine learning to prevent fraud and improve the customer experience in real-time with Confluent's Duncan Ash. The bank is using Confluent Cloud to power its advanced fraud detection system, which combines behavior analysis, data analytics, and predictive modeling to protect against financial crime without compromising the customer experience. By adopting Confluent Cloud, EVO Banco has significantly increased the accuracy of its fraud algorithms and reduced reaction times to just seconds, blocking an average of 500 fraudulent transactions daily and reducing weekly fraud losses by 99%.

The Two Generals Problem highlights the challenge of achieving reliable communication in distributed systems, which is also relevant to science fiction concepts like teleportation. The problem arises when using unreliable channels, making it impossible to guarantee message delivery or success. This has consequences in various fields, including software development, where developers must make decisions about how to handle message duplication and failures. To address this issue, Confluent's Apache Kafka provides transactional guarantees, enabling developers to build robust event streaming applications with high-throughput and reliability.

The text discusses the complexities and considerations involved in managing Apache Kafka infrastructure costs, particularly in the context of cloud services. It highlights that while compute and storage are commonly scrutinized expenses, networking emerges as the most significant and often overlooked cost driver, particularly due to cross-availability zone (AZ) traffic. The blog post is part of a series by Confluent, which aims to guide users in optimizing their Kafka-related expenditures, touching on aspects such as infrastructure, development, and operations resources. By using Confluent's managed services, many users reportedly find cost efficiencies, as the company has invested in architectural optimizations and partnerships to reduce costs related to networking, storage, and compute. The series promises further insights into cost management strategies for Kafka, with upcoming discussions focusing on development and operational costs.

This error occurs when the data formats of the publishers (Kafka producers) do not match the data formats of the subscribers (Kafka consumers), causing an "unknown magic byte" error. The Schema Registry solves this problem by validating messages before sending them, but reconciliation is necessary to ensure serialization methods are consistent on both production and consumption ends. This can be done in clients, ksqlDB, or Kafka Streams, depending on the project. A processing log can provide metadata about instances, including errors with "unknown magic byte" strings. Setting up an Avro schema in the client and registering it in Confluent Cloud can resolve this issue. Double-checking serialization setup and ensuring schemas are registered correctly is also crucial. If all else fails, deleting and re-producing messages may be necessary. The "unknown magic byte" error highlights the importance of data consistency and format validation in Kafka Streams.

The text discusses the integration of machine learning (ML) and data streaming, emphasizing the role of SQL as a common language across programming ecosystems and highlighting the benefits and limitations of using SQL for streaming solutions. It explores how SQL facilitates the creation of directed acyclic graphs (DAGs) for declarative programming but notes its lack of composability for complex analytics. The text also describes various approaches to bridging the language gap between Python and Java ecosystems, such as using libraries like Faust for Python and Kafka Streams for Java, and highlights the challenges of integrating ML experiment management platforms with Kafka. It showcases examples of real-time ML applications, such as Centrica Hive's ML-powered microservices and Babylon's health risk prediction system. The text concludes by asserting the potential and readiness of the community to integrate ML and streaming solutions, inviting developers to share their experiences and solutions on community forums.

At Mr. Cooper, a home mortgage company, they have built a powerful multicloud streaming platform using Apache Kafka and its ecosystem of integration and processing tools. The company has made data streaming the default platform for all real-time needs for its various business applications. They use Confluent's Kafka Streams and ksqlDB to perform stream processing, leveraging the platform's pub-sub and streaming processing capabilities. The goal is to provide real-time notifications when customers make payments or submit mortgage applications, reducing call volumes and increasing customer satisfaction. Mr. Cooper has also implemented a real-time streaming platform engineering team that employs stream processing extensively in its operations, with nearly 100 managed and self-managed connectors enabling seamless data integration across systems. They are looking forward to implementing more Confluent-managed connectors to increase the number of producer and consumer systems. The company's vision for the future is to create an event ingestion and discovery platform that allows internal users to access data and see how the market fluctuates and how customers respond to different products and services. Mr. Cooper recognizes the importance of data governance, especially when handling financial data, and has implemented Cluster Linking to migrate business events, consumer groups, and schemas without downtime. The company's leadership encourages a change in mindset towards data streaming, recommending identifying a data streaming use case that solves a challenging problem for the business and finding a business partner to implement it. They also emphasize the importance of building a minimum viable product for high-priority use cases and demonstrating the TCO and ROI compared to current processes. To govern data effectively without becoming a blocker, Mr. Cooper has built a centrally governed but federated partnership model with application engineering teams, establishing clear guidelines and standards, implementing an event catalog and governance process, fostering collaboration with different application teams, monitoring and optimizing performance, and enabling observability and monitoring.