Migrating from Postgres to ScyllaDB, with 349X Faster Query Processing

Coralogix, an observability platform, dramatically improved its query processing speed by migrating from PostgreSQL to ScyllaDB, reducing processing times from 30 seconds to 86 milliseconds. The move was motivated by the need to handle large-scale, semi-structured data efficiently, as the original PostgreSQL-based metastore implementation could not meet the low-latency and scalability requirements. By switching to ScyllaDB, Coralogix was able to improve performance significantly, handling 10,000 writes per second with sub-millisecond latency and processing 50,000 Parquet files in less than 500 milliseconds. The migration involved reworking data modeling to accommodate ScyllaDB's NoSQL architecture, focusing on partitioning and read/write patterns. Despite challenges such as high tail latency in column metadata processing and managing large partition sizes, the transition was completed in just two months, aided by tools like the ScyllaDB Operator for Kubernetes. Future plans include leveraging WebAssembly User Defined Functions to further optimize data processing.