Chasing Messages and Offsets in the Land of librdkafka

Kostas Kyrimis from the Memgraph Core team shares his experience implementing offset management for Kafka consumers using librdkafka during the Memgraph 2.1 release. The goal was to create a query procedure that sets the consumer's offset for streams, a task that appeared straightforward but revealed complexities due to librdkafka's handling of assignments and offsets. Kyrimis encountered issues with librdkafka's assignment() function, which did not guarantee proper synchronization, leading to a race condition. After several attempts and debugging, the team discovered the position() function, which accurately retrieved the current offsets, resolving the problem. Alongside this, Kyrimis experimented with Redpanda, a Kafka-compatible streaming engine, appreciating its ease of use and quick startup. The experience highlighted areas for potential improvement in librdkafka, such as better synchronization mechanisms and more intuitive offset management. Despite the challenges, the endeavor was ultimately successful, culminating in a new Memgraph feature and a personal sense of achievement.