September 2017 Summaries

The text explores the transformative impact of machine learning, particularly when integrated with Apache Kafka, in building mission-critical real-time applications. It highlights how machine learning enables computers to derive insights from unstructured data, enhancing traditional business processes like fraud detection and predictive maintenance. The text delves into the architecture of using Apache Kafka as a central streaming platform to facilitate the development, deployment, and monitoring of analytic models, emphasizing its role in enabling scalable, proactive decision-making. It describes the lifecycle of model development, from training with big data environments such as Apache Spark or Hadoop to deployment using Kafka Streams, which ensures real-time inference and scalable operations. The integration of machine learning with Kafka's ecosystem components is presented as a means to improve business outcomes through timely, data-driven insights, while also addressing the technical challenges of productionizing analytic models. The text concludes by pointing to resources and future demonstrations that showcase Kafka's capabilities in real-time machine learning implementations.

To minimize downtime and data loss from disasters, enterprises create business continuity plans and disaster recovery strategies that often require multi-datacenter Apache Kafka deployments. A disaster recovery plan requires a centralized configuration of cross-datacenter replication to ensure seamless failover and failback in case of a catastrophic event. Confluent Replicator is the key to any multi-datacenter design, managing multiple Kafka deployments and replicating data across clusters. The Confluent Platform 7.7 offers enhanced security features and new connectors, while OpenSearch Ingestion enables integration with Amazon OpenSearch for improved scalability and reliability.

Apache Kafka's Streams API provides various join types for stateful and stateless streams, including inner joins, left joins, outer joins, KTable-KTable joins, stream-table joins, and KStream-GlobalKTable joins. These joins can be used to combine data from multiple sources, such as combining view events with click events in an online advertising domain. The join semantics differ from SQL semantics and have runtime dependencies on processing order or timestamp synchronization. The Streams API also supports changelog streams, tombstone messages for deletes, and global tables that hold a complete copy of all data over all partitions of their input topic. The choice of join type depends on the specific use case and requirements, such as stateful vs. stateless processing, windowing, and partitioning constraints.

Kafka can be used as a primary store for data, but it's not necessarily crazy to do so, especially since it was designed with storage in mind. However, treating Kafka as a primary store does require running the system correctly and knowing its limitations. Unlike traditional messaging systems, Kafka is more akin to a distributed file system or database, with features like replicated logs, high-throughput reads, and concurrent writes. While Kafka can be used for stream processing, it's not intended to replace databases entirely, but rather serves as a commit log or storage layer for derived indexes or views. Its true mission is to make streams of data and stream processing a mainstream development paradigm, acting as a central nervous system for modern digital businesses.

Apache Kafka is used by thousands of companies, including banks, financial exchanges, and tech companies, for critical market data systems and distributed databases. Ensuring correctness and performance in these demanding environments requires a set of practices across the software development lifecycle, from design to production. The goal of this blog post is to give insight into how Confluent and the Apache Kafka community handles testing and other practices aimed at ensuring quality. The trend in software is away from up-front design processes towards an agile approach, but for distributed systems, a good design is essential. Kafka requires any major new feature or subsystem to come with a design document, called a Kafka Improvement Proposal (KIP), which allows changes to go through a broad and open debate. The community has a culture of deep and extensive code review that tries to proactively find correctness and performance issues. A hierarchy of testing approaches is needed, including unit tests, integration tests, and system tests, with the latter providing a good check of correctness in realistic environments. Confluent uses a framework called ducktape to aid in creating distributed environments, setting up clusters, and introducing failures, making it easier to debug and test the system. The tight feedback loop from people running Kafka at scale and the engineers writing code has long been an essential part of development, ensuring that the tools, configs, metrics, and practices for at-scale operation really work.

The text provides an in-depth look at Single Message Transforms (SMT), a feature in Kafka Connect that allows for the transformation of individual messages as they flow through data pipelines. This functionality is highlighted as a powerful yet simple tool for modifying messages without needing complex code, with examples showcasing how SMTs can be used to apply keys to messages, mask or drop fields, and route messages to different topics based on conditions like timestamps or regex patterns. The article emphasizes the flexibility and scalability of using SMTs within Kafka Connect, illustrating their utility in maintaining static views in data stores like Elasticsearch. The text concludes by noting the extensibility of SMTs through an open API and mentions the recent release of Apache Kafka 3.8.0, which includes various new features and improvements.

The New York Times has developed a new log-based architecture for publishing content, leveraging Apache Kafka as the central feature. This approach replaces their traditional API-based system, which had several issues including disparate schemas and normalized data formats. The log-based architecture simplifies accessing streams of content, provides immutable deployments of stateful systems, and enables consumers to easily build materialized views of the log. The Monolog is the new source of truth for published content, where all assets are appended in chronological order, while a denormalized log supports many-to-many relationships between assets. The architecture has simplified software development processes, reduced deployment complexity, and improved monitoring capabilities, but requires a mental shift for developers familiar with traditional pub/sub models.

The Kafka Summit brought together around 800 attendees, including committers, users, and enthusiasts, for a day of networking, food, and education. Allen Wang from Netflix shared battle-tested recommendations for running Kafka at scale, while Amy Boyle from New Relic discussed the importance of real-time stream processing at scale. Matthias Sax from Confluent introduced interactive queries in the Streams API for Kafka, providing an additional line of architectural defense against deploying a database instance as part of a microservice. The event also highlighted the growth of the Kafka community and announced upcoming Summits in London and San Francisco. Additionally, KSQL is now in developer preview, Apache Kafka will reach version 1.0 this year, and attendees can download a free copy of "Kafka: The Definitive Guide" online.