Making a Shard-Aware Python Driver for ScyllaDB, Part 2

In a presentation at the Europython 2020 Conference, Alexys Jacob, CTO of Numberly, discussed the development of a shard-aware Python driver for ScyllaDB, highlighting its structural differences from the Cassandra driver. The ScyllaDB driver opens a connection per core per node to efficiently account for cluster topology and shard ID calculations, unlike Cassandra which opens one connection per node. Despite protocol limitations that prevent the client from determining specific shard connections, the implementation uses an optimistic and asynchronous method to improve application startup times and performance. Alexys praised Israel Fruchter's contributions to the shard-aware driver code, especially in optimizing performance with Cython, which significantly reduced latency. The shard-aware driver demonstrated a 15% to 25% performance boost in production, with progressive improvements as the application runs longer, leading to reduced cluster load and resource savings. Alexys invited the audience to try the ScyllaDB driver, sharing its success in production and ongoing improvements in documentation and functionality.