March 2018 Summaries

The text explores the integration and enrichment of data using Apache Kafka and KSQL, explaining how these technologies can streamline the process of managing data from multiple sources in real-time. By using tools like Debezium for streaming data from MySQL to Kafka and the kafka-connect-spooldir plugin for ingesting CSV data, the article illustrates how to create a seamless, continuous data flow that enables real-time analytics without the need for complex coding. It highlights the use of KSQL for enriching data by joining streamed orders with customer information, demonstrating the simplicity and efficiency of SQL-based stream processing. Furthermore, the article details how enriched data can be streamed to Amazon S3 for further analysis, showcasing the capabilities of Kafka Connect in maintaining data integrity and facilitating analytics with tools like AWS Quicksight. The narrative underscores the power of Apache Kafka and KSQL in transforming data management, emphasizing the ease of configuration and the potential for real-time data processing and analysis.

Integrating databases with Apache Kafka for streaming data is a frequent topic in community discussions, and the process often involves Oracle, SQL Server, or DB2 databases. The primary methods for achieving this integration include using the JDBC connector for Kafka Connect or employing log-based Change Data Capture (CDC) tools. The JDBC connector is ideal for prototyping and low-volume tasks, allowing data to be streamed to and from any relational database that supports JDBC. In contrast, CDC tools provide a more comprehensive solution by capturing every event from a database's transaction log, making them suitable for high-fidelity, low-latency data streaming, particularly in event-driven applications. While JDBC is good for simple, low-volume tasks, CDC is recommended for applications requiring full event data capture. Both methods leverage Kafka Connect, a core component of Apache Kafka, to facilitate scalable and resilient integration between Kafka and other systems, offering flexibility in switching data sources and targets within the architecture.

The text discusses how Apache Kafka can be used for event sourcing, which involves storing a stream of events related to entities in an application instead of their current state. It explains that Kafka provides flexibility and power through its data streaming engine design, allowing users to work with high-level streaming APIs or low-level consumers. The text also covers querying the "current" state of each entity using Kafka Streams and state stores, handling side-effects, and implementing aggregations or projections. It concludes by stating that while Kafka wasn't originally designed for event sourcing, its design makes it well-suited for this purpose.

The February 2018 release of KSQL version 0.5 focuses on enhancing performance and stability through comprehensive bug fixes and rigorous testing strategies aimed at improving the operational experience. This release marks a significant step towards the general availability of KSQL, supported by extensive soak, bounds, and performance testing, with a soak testing cluster accumulating over 1,000 hours of continuous operation. The testing process involves over 42 different tests that gather more than 700 metrics to evaluate the performance of both stateless and stateful KSQL queries. As KSQL approaches its general availability release, it is anticipated to offer a reliable, production-ready product with a potential for a regular cadence of new features. The KSQL community is encouraged to participate in its development by providing feedback, voting on issues, and contributing via the Confluent Slack community and issue tracker. Additionally, recent enhancements to Kafka Streams with versioned key-value state stores are highlighted, which improve stateful processing capabilities by allowing multiple record versions per key.