September 2024 Summaries

Churnkey, a platform that helps businesses predict and prevent customer churn, faced challenges with managing large volumes of data from various sources. Their homegrown webhook solution struggled to scale as their customer base grew, leading them to seek a more reliable and scalable solution. They discovered Hookdeck, which provided the necessary scalability, reliability, and observability without requiring extensive integration or maintenance. By adopting Hookdeck, Churnkey was able to reduce unnecessary inbound events by 50%, scale their operations to handle larger volumes of data, ensure event delivery reliability, and simplify development and testing processes.

The article discusses how Harvust, a farm labor compliance software tool, faced challenges handling thousands of simultaneous Twilio webhooks due to the high volume of requests causing server timeouts. To address this issue, they moved their business logic into an asynchronous Celery task and utilized Hookdeck's Connection Rules Filter functionality to reduce the number of webhooks by about a third. Additionally, they discovered that immediate processing wasn't necessary for delivery status updates, so they used Hookdeck's Destination Max delivery rate feature to set a maximum number of requests per second, which helped prevent spikes and allowed them to utilize their excess server capacity more efficiently.

Hookdeck experienced a significant event delivery delay on August 28th and 29th due to their database vendor's storage auto-scaling feature. This incident led to severe consequences for many customers, resulting in the need for significant changes to maintain trust. The root cause was identified as the provisioning of slower disks instead of high-performance ones. No data was lost during the incident. Hookdeck is revamping its infrastructure and has decided to credit all customers based on their SLA policy. They will also make several changes, including updating their SLA to offer higher credit compensation, making real-time p99 latency public, working with the database vendor for improvements, and improving communication during incidents.