We turned off Pub/Sub and nobody noticed
Blog post from Incident.io
Incident.io redesigned its event-driven messaging infrastructure to remove Google Cloud Pub/Sub as a single point of failure and support its 99.99% availability commitments, despite processing roughly 240 million messages daily across more than 800 topics and 1,000 subscriptions. Building on an existing code abstraction called eventadapter, the team added NATS as a second broker and created a load-balancing implementation that distributes publishes between both systems, automatically reroutes failed requests through circuit breakers, and can be reconfigured without deployment. On the consumer side, it preserved existing concurrency limits through a scheduler that selects messages based on the oldest available publish timestamp, allowing capacity to shift fairly toward the broker with more delayed work without increasing resource consumption. After gradual production rollout, the company used chaos testing by deleting its NATS cluster and injecting simulated Pub/Sub failures; during a full Pub/Sub shutdown, traffic failed over to NATS without dropped messages, customer impact, or engineer alerts.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| Kubernetes | 3 | 3,490 | 385 | 112 | +26% |
| Real-time | 2 | 4,432 | 1,050 | 222 | -31% |
| Observability | 1 | 3,175 | 737 | 186 | -24% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.