June 2017 Summaries

The Apache Kafka community has introduced exactly-once semantics in Kafka, which enables correct exactly-once stream processing using the Kafka Streams API. This feature provides end-to-end guarantees for deterministic operations and non-deterministic operations with a subset of legal outputs. The feature was designed to provide minimal performance overhead and to make Kafka faster for everyone. It involves idempotence and transactions features that work together to ensure correct exactly-once stream processing. The feature is available in Apache Kafka 3.8.0 and Confluent Platform v3.3 and later, including Confluent Cloud.

The traditional ETL (Extract, Transform, Load) paradigm is being replaced by distributed systems and event-driven applications in modern enterprises. Businesses now process data in real time and at scale, treating data as a first-class citizen. Apache Kafka® has emerged as the core of these modern architectures, providing connectors for extracting data from different sources, a rich API for complex transformations and analysis, and more connectors for loading transformed data to another system. The end-to-end reference architecture includes Confluent Schema Registry for managing schemas, validating compatibility, and ensuring data conformity. This blog post demonstrates how easily streaming ETL pipelines can be implemented in Apache Kafka using the JDBC connector, Single Message Transform (SMT) functions, and the Kafka Streams API. The workflow includes extracting data from a SQLite3 database, transforming it into key/value pairs, and loading it to a Kafka topic for real-time stream processing. Finally, the transformed data can be written to another system using Kafka sink connectors.

We are excited for the general availability of Kafka release 0.11.0.0, which brings new features such as Exactly Once Semantics, providing more stringent idempotent producer semantics with exactly once delivery per partition and stronger transactional guarantees. The recent Apache Kafka Report showed that 37% of adopters are using the Kafka Connect API to load data from other systems into Kafka and extract Kafka data into other systems. We also released Apache Kafka 3.8.0, which contains many new features and improvements. To engage with the Kafka community, we invite you to join Google Group, Slack, Reddit, LinkedIn, Twitter, or attend a Kafka Meetup group in your area. Additionally, we'll be building an app using the confluent-kafka-javascript client to demonstrate the debate on tabs versus spaces.

Apache Kafka is being integrated with Python applications, leveraging the Confluent Platform for ease of installation and configuration. The Confluent Python client, confluent-kafka-python, uses the high-performance C client librdkafka. A simple program demonstrates how to write a message to a Kafka topic using a Producer instance, while another example shows how to create a consumer that joins a consumer group and reads messages from assigned partitions. Advanced features of kafka python integration include AvroProducer and AvroConsumer classes for serializing data in Avro format, as well as support for schema registry. Expert advice on deploying or operating Kafka is available through Confluent's training and technical consulting services, including enterprise-grade support and a hosted and fully managed version of Apache Kafka with Confluent Cloud.

Apache Kafka, commonly perceived as an ETL or data integration tool, is positioned as a more disruptive solution by being a robust streaming data platform that integrates event streams from both applications and databases, making them a critical element in modern digital businesses. Traditional ETL tools, which focus on moving data between databases, fail to address the need for a comprehensive infrastructure that treats data streams as first-class citizens, whereas Kafka enables real-time processing, scalability, and reliability across an enterprise. By facilitating the direct interaction of applications with event streams, Kafka supports microservices-based ETL, offering developers the flexibility to independently create and manage ETL processes within their applications, enhancing agility and efficiency. The platform's capacity to handle complex data workloads with features like built-in connectors, transformations, and APIs makes it suitable for modern business needs that require seamless integration and real-time processing. Consequently, Apache Kafka transcends the traditional ETL paradigm, akin to comparing a car's utility beyond that of a heavy umbrella, and continues to evolve through ongoing improvements to provide a comprehensive solution for stream processing and event-driven applications.