Home / Companies / Incident.io / Blog / Post Details
Content Deep Dive

We turned off Pub/Sub and nobody noticed

Blog post from Incident.io

Post Details
Company
Date Published
Author
Patrick Hamann and Mike Fisher
Word Count
3,059
Company Posts That Month
13
Language
English
Hacker News Points
4
Post removed?
No
Summary

Incident.io redesigned its event-driven messaging infrastructure to remove Google Cloud Pub/Sub as a single point of failure and support its 99.99% availability commitments, despite processing roughly 240 million messages daily across more than 800 topics and 1,000 subscriptions. Building on an existing code abstraction called eventadapter, the team added NATS as a second broker and created a load-balancing implementation that distributes publishes between both systems, automatically reroutes failed requests through circuit breakers, and can be reconfigured without deployment. On the consumer side, it preserved existing concurrency limits through a scheduler that selects messages based on the oldest available publish timestamp, allowing capacity to shift fairly toward the broker with more delayed work without increasing resource consumption. After gradual production rollout, the company used chaos testing by deleting its NATS cluster and injecting simulated Pub/Sub failures; during a full Pub/Sub shutdown, traffic failed over to NATS without dropped messages, customer impact, or engineer alerts.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
Kubernetes 3 3,490 385 112 +26%
Real-time 2 4,432 1,050 222 -31%
Observability 1 3,175 737 186 -24%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.