Interning at incident.io: rate limiting, resiliently

Anthony Oparaocha describes an internship project at incident.io focused on making the company’s rate limiting more resilient if its Valkey backing store becomes unavailable. After learning the existing system and writing product and technical scopes with support from teammates, he evaluated competing top-k traffic-tracking algorithms through simulations and selected HeavyKeeper because it performed more accurately under high-cardinality traffic. He implemented per-pod in-memory buffers that continue to enforce local rate limits during Valkey outages rather than allowing all requests through, then verified the behavior with local failure tests, staging load tests, and a gradual production rollout. A later enhancement addressed the fact that independently enforced pod limits could multiply the intended system-wide limit by using a Kubernetes informer to adjust each pod’s budget according to the live pod count. The account emphasizes the internship’s combination of meaningful ownership, evidence-based engineering decisions, careful testing, and mentorship while explaining relevant distributed-systems terminology in an appendix.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.