Centrifuge: a reliable system for delivering billions of events per day

Centrifuge, developed by Twilio Segment, is a robust system designed to deliver billions of messages daily to numerous public APIs, addressing challenges such as endpoint failures, latency spikes, and rate limiting. Traditional queueing systems proved inadequate due to their limited operations and inability to handle high-failure environments efficiently, prompting the creation of Centrifuge as a fully distributed job scheduler that offers message delivery, retry capabilities, and message archiving. The system employs a unique architecture that isolates message queues per source-destination pair, allowing for dynamic throughput adjustments and efficient error handling without excessive data copying. Centrifuge utilizes Amazon RDS with MySQL for its data model, which supports immutable rows and avoids complex database operations, enabling efficient message reordering and delivery management. The Director service manages job execution, caching, and database interactions, ensuring fast, reliable message processing even under high-load scenarios. Centrifuge's architecture allows it to scale dynamically with traffic, supported by a JobDB Manager that adjusts the number of databases to match the compute layer, ensuring efficient data handling and minimal delays. The system's effectiveness in handling high-failure scenarios was demonstrated during a significant third-party API outage, where it successfully managed to absorb and deliver a massive volume of events without data loss or significant customer impact.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.