Numberly: Learning Rust the Hard Way for Kafka + ScyllaDB in Production

Alexys Jacob, the CTO of Numberly, a French digital data marketing company, embarked on a significant technical challenge by transitioning key components of their data processing pipeline from Python to Rust to enhance performance with ScyllaDB and Apache Kafka. Despite initial resistance due to the lack of Rust expertise at Numberly, Alexys was driven by Rust's promise of creating reliable, efficient software and its advantageous features such as strong type safety, comprehensive error management, and improved dependency management. Although Rust was not faster to develop or prototype than Python, it was chosen for its potential to provide faster data processing, driven by the need to innovate and ensure reliability in their event streaming applications. Alexys highlighted that adopting Rust required overcoming a learning curve, but its long-term benefits in software reliability and performance justified the transition. The project involved integrating the Rust data processing application into their existing infrastructure, including Kubernetes, Prometheus, and Grafana, marking a high-stakes move straight into production rather than a simple exploratory project.