September 2018 Summaries

The text provides an in-depth guide on troubleshooting common issues encountered when using KSQL, a SQL streaming engine for Apache Kafka. It highlights the problem of KSQL queries not returning data and explains various potential reasons, such as incorrect topic names, data not being present in the source Kafka topic, or KSQL reading from the latest offset by default. The article suggests using tools like kafkacat to verify the presence of data in Kafka topics and advises changing the offset setting to 'earliest' to read all existing data. It also touches on serialization issues, emphasizing the need for correct serialization and deserialization formats, and illustrates how malformed messages can lead to incomplete data retrieval. Additionally, it covers the importance of setting the correct key when declaring tables in KSQL to ensure data is returned, and provides troubleshooting tips for resolving these and other related issues.

Zenreach's journey to implement a real-time customer presence detection platform evolved from initial Python scripts to a monolithic Spark Streaming application, and eventually to a robust, cost-efficient system using Kafka Streams, deployed in containers on AWS ECS. This transition was driven by the need to handle increased data throughput as the company scaled, necessitating a move away from infrastructure-heavy solutions like EMR. Kafka Streams allowed Zenreach to simplify its streaming infrastructure, enhance developer productivity, and improve fault tolerance and observability while maintaining significant cost savings. The platform, designed to detect customer walk-ins through Wi-Fi data, enriches this information with CRM data and keeps track of customer visits, enabling features like marketing email distribution and real-time customer tracking on mobile applications. The implementation also involved overcoming challenges related to state management, data backfilling, and service stability, with strategies like using EBS volumes for persistent storage and tuning Kafka consumer parameters to minimize downtime and rebalancing during deployments. The company plans to continue leveraging Kafka technology to make more components of its product event-driven while acknowledging the support from the Kafka community in navigating these technical challenges.

I am honored and excited to deliver the opening keynotes in both cities, sharing my take on streaming platforms, their evolution, and future industry trends. I'll also highlight real-world examples of use cases such as real-time analytics, event-driven microservices, IoT, and fraud detection. Confluent Streaming Events will bring together developers, architects, and business stakeholders to learn about building faster and more responsive digital businesses using the Confluent Platform, Flink, and MongoDB. These events are ideal for those taking their first steps in streaming technology or experts looking to take their knowledge to the next level.

The article explores the shift from traditional data handling methods to event-driven architectures and event streaming platforms. It discusses the blurring lines between analytic systems and transactional systems, emphasizing the importance of real-time data processing for modern applications. The adoption of ETL (Extract, Transform, Load) techniques in broader contexts is highlighted, with a focus on its application to microservices and event-driven architectures. The use of Apache Kafka® and KSQL project as an example of implementing this design pattern is also discussed. The benefits of streaming raw data back into Kafka are outlined, including improved flexibility, real-time accessibility, and reduced complexity in system architecture.

The paper "Streams and Tables: Two Sides of the Same Coin" published by Confluent and Humboldt-Universität zu Berlin presents the Dual Streaming Model, which is the foundation of Kafka Streams' and KSQL's stream processing semantics. This model provides a natural way to cope with inconsistencies between physical and logical order of streaming data in a continuous manner, without explicit buffering and reordering. The model decouples handling of out-of-order data from latency concerns, offering a design space between processing cost, accepted latency, and result completeness for users. The adoption of Kafka Streams and KSQL among enterprises demonstrates the Dual Streaming Model's ability to solve real-world problems across industries.

The text discusses the use of KSQL and Kafka in transforming and managing data pipelines, highlighting the benefits of compartmentalizing functionality through independent processes like Kafka Connect for data ingestion and KSQL for transformation. It explains how data is wrangled by performing operations such as flattening nested structures, reserializing data formats, unifying multiple streams, and creating derived columns, with the results being continuously updated in Kafka topics. The text emphasizes the flexibility and scalability of Kafka systems, allowing for easy modification and extension of data pipelines without impacting existing processes. It describes streaming transformed data to Google BigQuery for analytics using a Kafka Connect community connector and mentions the potential for archival and batch access via Google Cloud Storage (GCS). Additionally, it illustrates how transformed data can be visualized through tools like Google Data Studio, enhancing the utility of the data for driving analytics and applications.

Stitch Fix has built a robust data integration platform using Kafka, Flink, and MongoDB to support its machine learning and AI algorithms. The platform is designed to be self-service for Data Scientists, allowing them to easily configure their event data pipelines without requiring direct involvement from the engineering team. The solution leverages Kafka Connect's REST API to build a unified admin API that exposes a simpler API for validation and default parameters. The platform also includes a stream processing engine in Python, which provides a runtime environment with automated monitoring and log collection. The system has been running in production for almost eight months without major outages, freeing up the engineering team to focus on other impactful projects while giving Data Scientists autonomy and freedom.