August 2017 Summaries

We're excited for today's event as a huge number of people have been working hard to bring it to life, including speakers preparing presentations on new Apache Kafka features and production use cases. A pre-Summit gathering took place last night with the community, and we'll be live streaming keynotes starting at 9:00 am PDT featuring Confluent co-founder Neha Narkhede, Stitch Fix's VP of Engineering Randy Shoup, and Salesforce's Pat Helland. Video recordings of all 32 talks will be available in a few weeks, and attendees can download the mobile app for real-time updates throughout the day. The event is part of Current 2024, a gathering of the data streaming world in Austin, Texas on September 17-18, where tech leaders and industry giants will share knowledge on Apache Kafka and Apache Flink.

KSQL is a streaming SQL engine that enables real-time data processing on Apache Kafka streams. It provides a simple and interactive SQL interface for stream processing, eliminating the need to write code in programming languages like Java or Python. KSQL supports various stream processing operations, including aggregations, joins, windowing, sessionization, and more. This technology simplifies streaming applications by integrating tables and streams, allowing users to join current state data with continuous event streams. KSQL enables real-time monitoring, anomaly detection, security, online data integration, and application development. It uses Kafka's Streams API internally and provides a distributed SQL engine for queries and control. The technology is designed to turn the database inside out by using Kafka as the core storage abstraction for streaming data, allowing for real-time processing and materialized views over the log. KSQL is currently in developer preview and will receive additional capabilities as it evolves into a production-ready system.

This article explains how to build a scalable data pipeline by streaming data from a relational database into Elasticsearch using Kafka Connect and Confluent Platform. The author assumes prior knowledge of setting up Apache Kafka and provides instructions on loading the Elasticsearch connector, configuring it for mapping and converting data types, and querying the data using the Elasticsearch REST API. The article highlights the benefits of using Kafka Connect, including scalability, fault-tolerance, and customizable pipelines, with examples that demonstrate how to extend this pipeline to land data in other destinations such as HDFS or BigQuery.

A database 'unbundled' is an approach to rethinking systems by breaking out various concepts found inside a traditional database into separate layers, allowing for more control and flexibility in building independently deployable services that can share datasets without shared mutable state. This approach uses Kafka's distributed log as the core element, which provides a low-coupling mechanism for sharing data. By unbundling a materialised view into a continuously updating cache, services can have functional properties similar to traditional databases but with better scalability and predictability. The unbundled database concept is beneficial in reducing coupling between services, making it easier to release individual components without affecting others, and enabling services to take only the data they need at a point in time, resulting in smaller and more efficient views.

The article provides a step-by-step guide on building a simple data streaming pipeline using Apache Kafka, Kafka Connect, and other tools like MySQL and Elasticsearch, demonstrating how to stream data from MySQL into Kafka and then into a text file or Elasticsearch, using the Kafka Connect API without the need for coding. It showcases the ease of integrating data from sources into targets and allows for scalable streaming data pipelines, emphasizing Kafka's role as a central integration point for data feeds and real-time applications. The guide includes detailed instructions for setting up the environment on a Mac using tools like Confluent Platform, jq, and brew, and provides insights into configuring connectors with Kafka Connect Distributed mode for scalable operations. The article also highlights the automatic management of timestamps by MySQL and the offset-tracking capability of Kafka Connect, making it easier to handle new or updated rows in the database, and offers a glimpse into further extending the pipeline in upcoming parts of the series.

The next Kafka Summit will take place on August 28, 2017 in San Francisco at the Hilton Union Square. This event will feature keynotes from industry leaders discussing the latest developments and use cases for Apache Kafka®. Attendees can expect to learn about running event-integrated microservices at scale, why event-driven architectures are advantageous, and innovative ways companies like HomeAway, Shopify, Netflix, and IBM are using Kafka. The Summit will also include an "ask me anything" booth with various Kafka experts available for conversation and answering questions. A party celebrating Kafka's maturity is scheduled for the evening of August 28th. Registration is open for those interested in attending.

This post discusses Event Sourcing in the context of Apache Kafka, examining the need for a single source of truth that spans entire service estates. The concept of notification via a broker breaks down complex call graphs, allowing processes to decouple themselves from one another. Events are facts that evolve and tell a story, forming a single source of truth. Implementing event sourcing in Kafka is a no-op if you've taken the event-driven approach, as events are simply written as they occur. The advantage of this 'database-fronted' approach is it provides a consistency point: you write through it into Kafka, meaning you can always read your own writes. Event sourcing is often used with Memory Images, which provide an optimal combination of fast, in-process reads and write performance far faster and more scalable than a typical database. The log breeds extensibility, empowering bootstrapping new services and migrating away from legacy systems.

Confluent Platform 3.3 and Apache Kafka 0.11 introduce significant updates that enhance the platform's capabilities, including the introduction of exactly-once messaging semantics which improve transactional guarantees for real-time streaming applications. These updates focus on reducing data loss and duplication, thereby facilitating reliable data processing for critical tasks such as financial transactions and ad tracking. Additionally, performance improvements have been made to mitigate the impact of exactly-once processing, while the new plugin system for Kafka Connect simplifies connector management, allowing for the integration of various systems without compatibility issues. The Confluent CLI further streamlines user experience by providing easy management of services and connectors. Enhanced monitoring capabilities now support multiple languages, expanding beyond Java, and improvements to the REST Proxy enable per-user authentication for better security. The announcement highlights the continuous evolution of the Confluent Platform to meet the needs of diverse data-driven organizations.