Company
Date Published
Author
Raymond Tukpe
Word count
3253
Language
English
Hacker News points
None

Summary

Designing a webhook delivery system faces challenges with bad or zombie endpoints, which can lead to system inefficiencies. To tackle this, the article explores the implementation of a stateful distributed circuit breaker in Convoy, specifically designed to manage these issues by temporarily halting requests to failing endpoints. The article contrasts synchronous and asynchronous circuit breakers, highlighting their respective advantages and disadvantages. Synchronous breakers are simpler and offer clear visibility but limit concurrency and lack synchronization across replicas. Asynchronous breakers, although more complex, enhance availability and concurrency, and synchronize state across replicas. The article delves into various solutions using Redis and PostgreSQL, ultimately favoring an approach that leverages database polling and distributed locks to optimize the system's performance by reducing failed delivery attempts significantly. The design aims to work efficiently even in scenarios where the leader agent or Redis faces downtime, ensuring resilience and reliability in production environments.