Company
Date Published
Author
Premkumar Thangamani
Word count
1349
Language
English
Hacker News points
None

Summary

In distributed databases, hotspots can occur when data grows in one node before moving to another node, causing a single node to become a hotspot for reads and writes. To avoid this, creating an ASC index on the create timestamp can distribute the data across multiple tablets, but this method loses the ordering needed for range queries. A better solution is to implement lightweight application-level sharding, which distributes the index across multiple tablets while retaining the ordering by create_ts. This involves creating an index of the form (shard_id HASH, create_ts ASC) and specifying the shard_ids in the IN clause or as a UNION of multiple selects for range queries. By doing so, the hotspot problem is removed, and the query executor can efficiently retrieve rows from the sorted lists.