Delivery Groups: Solving noisy neighbors in multi-tenant webhook delivery

Webhook traffic can be unpredictable and uneven, often leading to issues in systems handling multiple tenants, such as eCommerce apps or SaaS connectors, where one tenant's burst of activity can cause delays for others. A notable example involved a Shopify store sending 30 million events in a short period, resulting in significant backlogs for other stores. Existing solutions like waiting it out, terminating the noisy tenant, or building custom per-tenant queuing systems are inadequate. Delivery Groups, a new feature, addresses this by assigning a unique delivery rate to each tenant based on a key extracted from webhook payloads, ensuring fair distribution of resources and preventing one tenant from monopolizing the system. The feature allows setting per-group rate overrides and integrates with Event Gateway for dynamic rate management, reducing delays and enhancing system observability with alerts for queue degradation. Currently in early access, Delivery Groups invite user feedback to refine its approach to solving the multi-tenant fairness challenge.