January 2020 Summaries

A scalable and reliable infrastructure for machine learning tasks can be built using the Apache Kafka ecosystem and Confluent Platform, simplifying the design of mission-critical real-time architectures. Streaming machine learning enables direct consumption of data streams from Confluent Platform into machine learning frameworks like TensorFlow, reducing the need for a traditional data lake. Tiered Storage in Confluent Platform combines local Kafka storage with remote storage layers, allowing data to be stored long-term without high costs or scalability issues. This simplifies model training and deployment, enabling rapid prototyping and data preprocessing, as well as robust and decoupled model management. A Kappa Architecture is a key pattern for building machine learning infrastructure, leveraging event streaming for processing both live and historical data. With this architecture, monitoring, testing, and analysis of the entire machine learning infrastructure can be critical but hard to realize in many architectures.

Confluent has introduced Tiered Storage in its Confluent Platform 5.4, allowing operators to add an additional storage tier for data. The feature aims to make the platform more scalable, cost-efficient, and easier to run operationally. Tiered Storage separates the concerns of data storage from data processing, enabling independent scaling of each. By offloading the majority of data to a remote store, it reduces the time and cost required for rebalancing, expanding or shrinking clusters, or replacing failed brokers. This also allows users to use different storage types at the broker level to improve performance and cost profile. Tiered Storage is currently in its preview phase as a feature of Confluent Platform and supports Amazon S3.

Confluent Platform 5.4 has been released, featuring enhancements in security, resilience, and compliance for production-ready deployments of event streaming platforms. New features include Role-Based Access Control (RBAC), Structured Audit Logs, Multi-Region Clusters, broker-level Schema Validation, and integration with Confluent Control Center for managing and monitoring clusters. The release also includes developer productivity tools like KSQL pull queries and the KSQL REST API, as well as features from Apache Kafka® 2.4. These updates aim to improve efficiency and scalability of event streaming infrastructure while maintaining a strong focus on security.

To build RAG-enabled GenAI models with Confluent, Flink & MongoDB, Netflix aims to create a robust and scalable financial application that can provide critical insights for planning, budgeting, and tracking content spending. The Content Finance Engineering Team uses a microservices-driven approach, modeling every financial application as a separate service. However, this leads to potential issues with data consistency and availability, particularly when dealing with complex event exchanges and synchronous request-based interactions. To address these challenges, Netflix adopts an event-driven architecture using Apache Kafka as the de-facto standard for messaging and stream processing. The platform enables asynchronous communication, promotes decoupling, and provides traceability as a first-class citizen. With Kafka's high durability and linear scalability, Netflix can process large amounts of data from various sources, including databases and external services. By leveraging Confluent Schema Registry and Apache Avro, the company defines schemas for its output streams to ensure backward compatibility and versioning. Additionally, Netflix uses Spring Boot and EVCache to implement idempotent behavior and guarantee exactly once delivery in its distributed system. The platform also provides a real-time view of service levels within the infrastructure, enabling the company to monitor key metrics and visualize dimensional time series data with Atlas. By adopting this event-driven architecture, Netflix can efficiently manage its financial application, provide critical insights for content spending, and drive business decisions.

The text explores the intricacies of Apache Kafka's processing layer, focusing on fault tolerance and elasticity through the stream-table duality. It details how streams and tables achieve fault tolerance by storing data in Kafka, with tables maintaining state information for operations like joins and aggregations. The changelog topic acts as a source of truth for tables, ensuring data can be restored after failures. Elasticity is achieved by dynamically migrating stream tasks across application instances, facilitated by Kafka's rebalancing process and storage features like compaction, which optimizes data storage and recovery times. The text emphasizes that elasticity and fault tolerance are interconnected, as both involve moving tasks and data across instances. Standby replicas and compaction are highlighted as methods to minimize recovery time during rebalancing and scaling. The text concludes by addressing challenges like data skew and the need for capacity planning, offering solutions to optimize parallel processing and storage. Overall, it provides a comprehensive understanding of how Kafka's architecture supports robust, scalable, and efficient real-time data processing.

The third installment in a series on Apache Kafka explores its processing fundamentals, focusing on streams and tables as tools for implementing distributed applications that enable parallel data processing. Building on the storage concepts discussed earlier, the article explains how events stored in Kafka topics are transformed into streams and tables for further processing using tools like ksqlDB and Kafka Streams. It delves into the concept of consumer groups, which facilitate scalable parallel processing by enabling distributed applications to form groups that read from the same input topics, thereby allowing dynamic workload distribution as application instances join or leave. The article also covers the state management necessary for tables, emphasizing the use of state stores for maintaining application state and ensuring efficient and fault-tolerant data processing. Additionally, it highlights the significance of data contracts and schema management in ensuring proper data serialization and deserialization across different clients, underscoring the advantages of using a schema registry for data governance. The discussion extends to more advanced topics like global tables and the partitioned design of Kafka's processing layer, which enhance scalability and performance. The article concludes by hinting at further discussions on elastic scaling and fault tolerance in the subsequent part of the series.

The second part of a series on Apache Kafka delves into its storage fundamentals, focusing on the concepts of topics, partitions, and brokers, which are essential for Kafka's scalability, elasticity, and fault tolerance. Topics in Kafka serve as the storage layer where events are durably stored and can be configured with settings like data retention policies. Partitions, a key element, allow for distributed data placement, enabling better scalability and fault tolerance through replication. Kafka's architecture decouples event producers from consumers, allowing for efficient event partitioning and processing in parallel, which is vital for stream processing applications. The article highlights that proper event partitioning is crucial for maintaining event order and load distribution across partitions, recommending strategies such as over-partitioning to ensure scalable performance. The understanding of these storage concepts sets the stage for the next part of the series, which will explore Kafka's processing capabilities through streams, tables, and data contracts.

Apache Kafka is an event streaming platform that provides scalable, fault-tolerant, and reliable data processing capabilities. It captures events into event streams, which record the history of what has happened in the world as a sequence of events. These streams can represent hundreds of years of history and are ordered, allowing for causality to be inferred between events. In contrast, tables represent the state of the world at a particular point in time, typically "now," and are views of event streams that are continuously updated. The stream-table duality is a fundamental concept in event streaming, where a table is fully defined by its underlying change stream, and both streams and tables can be easily converted into each other. This duality enables continuous, streaming updates to both the stream and the table, allowing for real-time processing and analysis of data.

This article discusses how to deliver enterprise-grade Apache Kafka to customers by offloading data from on-premises transactional databases to cloud-based datastores, such as Snowflake and Amazon S3 with Athena. The process involves ingesting data into Kafka, enriching it using ksqlDB, and then streaming it to the target data store. The benefits of this approach include improved performance, scalability, and flexibility, as well as enabling multiple teams within an organization to use the same data without introducing unnecessary dependencies or complexities. By using Kafka, organizations can address issues related to data security, governance, and enterprise service buses, while also providing a flexible foundation for making data available across applications and teams. The article concludes by highlighting the availability of Confluent Platform 7.8 and its latest features, including Confluent Platform for Apache Flink, mTLS Identity for RBAC Authorization, and more.

Google Cloud Platform (GCP) has integrated Confluent Cloud into its marketplace, making it easier for developers to access and manage Apache Kafka services. This integration allows users to leverage their existing GCP billing service when using Confluent Cloud, a fully managed event streaming platform built around Apache Kafka. The streamlined process reduces the need for separate bills and credit card management, enhancing user experience. Additionally, this integration makes Kafka a first-class citizen on GCP, enabling developers to be more productive in building event streaming applications. Confluent Cloud offers serverless scaling and pricing based on usage, further simplifying the process of managing Apache Kafka services.