Large Partitions Support in ScyllaDB 2.3 and Beyond

ScyllaDB 2.3 introduces enhanced support for managing large partitions, which are known to cause performance issues, by utilizing the system.large_partitions table to identify and analyze them. The update aims to improve data modeling and performance by allowing users to track large partitions within a cluster, using a schema that records key attributes like keyspace name, table name, partition size, and compaction time. The table functions as a node-local repository and updates its entries during memtable flushes and compaction, providing insights into the distribution of large partitions. Users can configure the threshold for what constitutes a large partition through the compaction_large_partition_warning_threshold_mb parameter in the scylla.yaml file, with the default set to 100MB. Although the current release does not support the ALLOW FILTERING keyword in queries, this feature is planned for future updates. The introduction of system.large_partitions reflects ScyllaDB's ongoing commitment to improving large partition handling, encouraging users to explore and provide feedback on their experiences with managing data in this context.