July 2017 Summaries

Apache Kafka's transactions feature is designed to simplify event-driven systems by providing exactly-once guarantees, eliminating the need for manual idempotence handling. Transactions allow services interconnected with Kafka to focus on their core logic without worrying about deduplicating inputs or constructing keys for outputs. This feature enables developers to build fine-grained "micro" services that behave predictably in complex chains as they would standing alone. By leveraging transactions, developers can create reliable event-driven systems that are easier to test and maintain.

Apache Kafka is a distributed, scalable messaging technology designed as a backbone for service-based systems to exchange and buffer events. It scales well, runs always-on, and retains datasets long-term. Kafka provides durability through replication, allowing messages to be written to multiple machines so that data is not lost in case of machine failure. The system also features a log-structured approach, making reads and writes sequential operations, which helps with overall performance and makes it suitable for storing messages longer term. Kafka's throughput control feature allows services to operate within strictly enforced SLAs by allocating defined bandwidth to specific services. Strong ordering guarantees can be achieved through partitioning messages that require relative or global ordering. The system also includes a compacted topic feature, which reduces the growth of datasets and makes them easier to move from machine to machine. Kafka supports event sourcing and stream processing, making it an ideal choice for building event-driven systems. It is designed to be scalable, with production clusters starting at three machines and growing to hundreds. The system also includes features such as transactions, schema validation, and batching, which provide a high degree of control over data movement and storage.

The new bidirectional client compatibility feature in Apache Kafka allows for easier upgrades, enabling users to upgrade their clients first, rather than waiting for the brokers to be upgraded. This feature builds on backwards compatibility and removes limitations that previously required manual reconfiguration of clients when upgrading brokers. With improved client compatibility, administrators can continue running older brokers with newer clients for a longer period, reducing the need for frequent upgrades. The new feature is supported by brokers at version 0.10.0 or later and is enabled automatically in Java clients.