Fair Delivery Groups: Solving noisy neighbors in multi-tenant webhook delivery

Webhook traffic can be highly variable and can cause significant delays for other tenants in multi-tenant platforms, as illustrated by a case where a single Shopify store sent 30 million events in five minutes, clogging the queue for others. Traditionally, solutions to this "noisy neighbor" problem have been inadequate, either requiring substantial infrastructure overhead or being incompatible with webhook delivery paths. Fair Delivery Groups, however, offer a managed solution by allowing users to set rate limits for each tenant using a key extracted from the webhook payload, thereby distributing events more evenly and preventing one tenant's traffic surge from affecting others. By creating individual delivery groups for each tenant, the system ensures that each tenant's events are delivered at their own rate, allowing for smooth operation even during traffic spikes. Additional features include the ability to handle per-group rate overrides for VIP or noisy tenants, comprehensive observability, and backpressure notifications, which collectively enhance the management and reliability of webhook deliveries in multi-tenant environments. This solution is now in early access, inviting user feedback to refine the approach further.