Company
Date Published
Author
Evan Cordell
Word count
2460
Language
English
Hacker News points
10

Summary

At AuthZed, they are building SpiceDB, a distributed authorization system inspired by Google's Zanzibar. They recently scale-tested SpiceDB on CockroachDB to 1 million authorization events per second with 1% writes and 100 billion stored relationships. To achieve this, they implemented an advanced connection pooler that allows for better performance from CockroachDB at scale. The crux of the problem is not just about modeling permissions as a graph, but also about carefully unlocking scale and performance. They identified several scaling limits in SpiceDB, including uneven load across CockroachDB nodes, which artificially caps the capacity of the cluster. To address this, they developed a solution that uses a sentinel bit to identify the gateway node for a connection, allowing them to track connections properly and balance connections evenly across all CockroachDB nodes. They also implemented a pruning algorithm to deal with cases where the number of connections does not divide evenly between the nodes. Additionally, they increased the lifetime of connections to reduce tail latency, and developed a solution to detect cluster changes with long-lived connections. The work has been wrapped up in a new library called crdbpool, which makes it available to the larger community.