ScyllaDB Student Projects, Part III: Kafka Client for Seastar and ScyllaDB

In a collaborative effort sponsored by ScyllaDB and conducted at the University of Warsaw, a group of computer science students, including the author Piotr Grabowski, developed a Kafka client for Seastar and ScyllaDB as their final year project. This initiative aimed to enhance the performance, correctness, and feature set of Kafka clients by leveraging ScyllaDB's Seastar framework, which emphasizes asynchronous, non-blocking operations to optimize latency and resource utilization. The project successfully resulted in a high-performance Kafka producer that supports various features such as future-based APIs, batching, and error handling. The students specifically designed the client to address limitations in existing libraries, such as librdkafka, by avoiding cross-core data transfers and improving throughput using Seastar's capabilities. Benchmarks demonstrated the client’s superior speed compared to established Java and C++ Kafka clients, particularly in multi-broker environments and scenarios involving small message sizes. Additionally, the project integrated the Kafka client with ScyllaDB’s Change Data Capture (CDC) feature, showcasing its potential for facilitating asynchronous data replication. The project served as the foundation for the students' Bachelor’s thesis and highlighted the benefits of industry-academia collaboration in advancing innovative data engineering solutions.