October 2017 Summaries

Kafka Summit London, the first of its kind in Europe, is set to take place on April 23-24, 2018, reflecting the growing demand for Kafka-related events in the region. Following successful summits in the United States, the event opens a call for papers until December 1st, inviting proposals on real-time data integration, stream processing, and in-depth technology discussions. With only 30 sessions available, prospective presenters are encouraged to submit engaging and innovative ideas that fit into one of three tracks: Streams, Pipelines, and Internals. The summit aims to provide attendees with fresh insights and practical knowledge, curated by a program committee consisting of experts from LinkedIn, O'Reilly Media, and Confluent. Additionally, the upcoming Current 2024 event in Austin, Texas, will further explore the data streaming landscape, featuring discussions on Apache Kafka and Apache Flink.

Zalando, a major European online fashion retailer, has been utilizing Apache Kafka's Streams API since its initial release in 2016 to build real-time applications on AWS, despite initial challenges with stability and resource management. The Kafka Streams API, a Java library within Apache Kafka, facilitates real-time data processing with features like stateless and stateful operations, elastic scalability, and fault tolerance, which are crucial for mission-critical applications. Zalando faced issues such as stream consumer group rebalancing, AWS I/O credit depletion, and memory management challenges, which were gradually resolved through software updates and infrastructure adjustments, including the allocation of larger disk spaces and optimized monitoring solutions. The company developed tools like Remora for consumer lag monitoring and leveraged Dropwizard and Prometheus for comprehensive system metrics, ultimately achieving enhanced productivity and ease of scaling with the Kafka Streams API's user-friendly programming model. Future installments of Zalando's series will explore disaster recovery strategies involving Kafka and Zookeeper backups to Amazon S3.

Zalando, Europe's largest online fashion retailer, has developed an automated methodology to discover the next "best" fashion-related websites by using a modified version of Jon Kleinberg's HITS algorithm. The core concept in HITS is that of Hubs and Authorities, where a Hub points to lots of other good entities and an Authority is pointed to by lots of good entities. Zalando uses Apache Kafka's Streams API for real-time ranking of domains and building applications on top of it. This approach allows them to build scalable microservices that fit well into their existing deployment setup, stay close to the data, and solve important problems in their business domain.

The text illustrates the use of Kafka Streams API, specifically its KTable and GlobalKTable abstractions, to enable real-time machine learning applications. The example focuses on predicting flight arrival times, utilizing a streaming approach that continuously updates models with new data, rather than the traditional batch processing method. By leveraging Kafka Streams API, the process involves consuming live flight data, joining it with pre-trained models stored in GlobalKTables, and generating predictions using logistic regression. The system updates models dynamically as more data becomes available, improving prediction accuracy from an initial 50% to between 80-90%. This approach demonstrates how Kafka's stream processing capabilities allow seamless model updates, eliminating the need for separate batch training and enabling integration into existing workflows. The implementation shows that elastic, scalable, and distributed applications can be developed without additional big data processing clusters, simplifying architecture and enhancing predictive capabilities in real-time environments.

KSQL, a streaming SQL engine for Apache Kafka, enables sophisticated stream processing on Kafka topics through an interactive SQL interface. This article demonstrates how to set up a sandbox environment using Confluent Platform to explore KSQL with Twitter as a demo data source. The process involves ingesting tweets via a Kafka Connect Twitter Connector, defining schemas, and executing SQL queries to filter, transform, and aggregate data. The article highlights the power of KSQL to handle unbounded streaming data by creating continuous queries and aggregates, converting streams into tables, and managing late-arriving data. It showcases KSQL's capabilities in transforming raw streaming data into structured insights, emphasizing its utility in building robust stream processing applications.

Here's a neutral and interesting summary of the text in one paragraph: Pinterest uses Apache Kafka Streams API to provide inflight spend data to thousands of ads servers, reducing overdelivery by building a predictive system that analyzes price, impression rate, and action rate. Overdelivery occurs when free ads are shown to out-of-budget advertisers, reducing opportunities for discovery. The company's system was initially slow to react, leading to extra impressions being shown at no cost, but after implementing the predictive spend pipeline, they significantly reduced overdelivery. By using Apache Kafka Streams API, Pinterest achieved a fast, stable, fault-tolerant, and scalable solution that provides near-instant information about inflight spend, allowing them to respect the budgets of their advertising partners and provide the best experience for advertisers.