Company
Date Published
Author
Sam Kleinman
Word count
714
Language
English
Hacker News points
None

Summary

GlareDB's v0.8.0 introduces "workload sharding" features to facilitate the efficient processing of data by multiple stateless application servers, ensuring equal data distribution without redundancy. The method involves hashing a field with high cardinality, such as a user ID, into a 64-bit integer and using modulo operations to distribute workloads evenly among workers, minimizing the need for coordination and infrastructure. However, this approach presents challenges, including difficulties in adjusting the worker pool size and potential issues with workload reallocation if a worker fails. To support this sharding method, GlareDB has added fnv() and siphash() functions, which generate uint64 values, and a partition_results function, allowing users to easily implement this strategy in queries. This function works by returning a boolean that can be used in a WHERE clause, facilitating the assignment of tasks to specific workers based on partition IDs.