Company
Date Published
Author
Olivia Greene, Antony Stubbs, Ahmed Saef Zamzam, Prabha Manepalli, Weifan Liang
Word count
1516
Language
English
Hacker News points
None

Summary

The Confluent Parallel Consumer is an experimental JVM-based client library for Apache Kafka that provides an alternate approach to parallelism, subdividing the unit of work from a partition down to a key or even a message. It offers three new features: increased parallelism beyond the number of partitions, key-level ordering guarantees, and non-blocking I/O to efficiently call out to other services. The library addresses issues such as slow consumer scenarios, where messages are held up by slow-to-process messages, and provides high-performance handling for these situations. By processing at three different levels - partition, key, or unordered - the Confluent Parallel Consumer increases performance while addressing common problems in Kafka-based applications.