Building a distributed rate limiter that scales horizontally

The article discusses the development of a horizontally scalable rate limiter for Ably's realtime messaging platform, addressing challenges such as enforcing instantaneous rate limits and maintaining service quality for other customers during sudden traffic surges. The solution involved introducing an account-wide, global, near-instantaneous message rate limit with specific design constraints including guaranteed message delivery, no blind fanout limit, scalability, throughput stability, predictability, and fast reaction time. The final implementation used a probabilistic suppression factor based on the degree by which the account was beyond its limit, combined with a moving average of the rate measured over 24 seconds. Further improvements included tracking the total number of messages that would have been generated had they succeeded, allowing for more accurate calculation of the suppression rate needed to ensure aggregate message rate exactly at the limit and avoid oscillation.