July 2016 Summaries

Apache Kafka 3.8.0 has been released, bringing numerous new features and improvements, including several prominent ones that are highlighted in this blog post. The release marks an exciting milestone, especially considering it's already August and the monthly "Log Compaction" series is nearing its one-year anniversary. A new app will be built using the confluent-kafka-javascript client to demonstrate a key debate - whether to use tabs or spaces - in a Kafka-style approach, producing and consuming data from a topic.

The blog post is part of a series exploring Apache Kafka's Streams API, focusing on its security features crucial for applications with stringent internal and external security requirements, particularly in industries like finance, healthcare, and governmental services. It outlines the essential Kafka security options available to secure data-in-transit and enforce client authentication and authorization, allowing users to tailor security setups to their specific needs, balancing security, convenience, and agility. The post emphasizes the flexibility of configuring secured and non-secured Kafka clusters, detailing how to implement these features in stream processing applications using the Streams API. Additionally, it warns of potential runtime failures due to misconfiguration of security settings and advises monitoring log files for errors. The post concludes by indicating further resources for deeper exploration into Kafka's Streams API, including advanced features like versioned key-value state stores, which enhance stateful processing capabilities.

The blog post serves as an introduction to the elasticity and scalability features of the Streams API in Apache Kafka, focusing on its ability to dynamically scale stream processing applications. It explains that a Kafka Streams application is a regular Java application where each JVM process represents an instance of the application; running additional instances increases processing capacity based on the environment's available resources. The Streams API utilizes Kafka's group management functionality to dynamically adjust workload distribution among instances, ensuring fault tolerance and eliminating downtime during scaling. The number of topic partitions determines the upper limit for parallelism, and while Kafka Streams doesn't include a resource manager, it allows seamless integration with existing infrastructure by enabling users to deploy instances using tools like Kubernetes or manually. This flexibility allows for efficient scaling without disrupting operations, making Kafka Streams suitable for resilient and scalable stream processing.

The July 2016 edition of Log Compaction, a monthly digest of highlights in the Apache Kafka and stream processing community, has been released. It includes the announcement of Apache Kafka 3.8.0 with many new features and improvements. A blog post discusses some prominent features, while the full list of changes can be found in the release notes. Additionally, a debate on using tabs or spaces is addressed using the new confluent-kafka-javascript client to build an app that interacts with Kafka topics.