How Numberly Replaced Kafka with a Rust-Based ScyllaDB Shard-Aware Application

Numberly's Omnichannel Delivery team replaced their messaging infrastructure, previously reliant on Kafka, with a Rust-based ScyllaDB shard-aware application to streamline message processing and improve reliability. The new system consolidates message handling into a single platform, allowing for deterministic workload distribution and reduced dependency on multiple technologies, enhancing reliability and idempotence. The change required significant architectural adjustments, including the use of ScyllaDB's shard-per-core architecture to align data processing with CPU cores, effectively turning ScyllaDB into a distributed cache and ordered-data bus. The team chose Rust for its efficient, low-level capabilities, contributing to the ScyllaDB Rust driver to facilitate this transition. The implementation faced challenges, such as ensuring idempotence and optimizing memory usage, but ultimately led to a highly efficient and scalable system that earned the ScyllaDB Innovation Award for Top Technical Achievement.