How We Serve 268,000+ Chat Events a Second

During an NFL broadcast, two rapidly growing prediction-market customers reached more than 35,000 and 45,000 concurrent WebSocket connections, illustrating that real-time chat capacity is driven less by message writes than by fanout: 16.7 messages per second generated nearly 39,000 message deliveries and more than 268,000 total events per second. Stream isolates customers on independent shards containing database, cache, HTTP, event-bus, and stateful WebSocket components, with events partitioned by app, channel, and users to preserve per-channel ordering. Its WebSocket nodes use approximate Cuckoo filters to cheaply discard irrelevant events, in-memory subscription indexes for relevant recipients, event classifications that preserve critical and ordered data while merging replaceable updates, and per-client queues sized by allowable lag time rather than event count; when queues overflow, lower-priority events such as typing or presence updates are dropped first. The company found that database sharding and API write capacity were not the central fanout constraints, but operational testing exposed other weaknesses, including machine-level storage I/O limits, an unmeasured latency increase from an LLM moderation service, and autoscaling that could take 90 minutes to meet traffic arriving in minutes. After adjusting infrastructure, moderation, and instance limits through active monitoring, both customers experienced no tracked endpoint failures during the peak, while the system maintained message latency near 493 milliseconds despite tripled connections.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.