Company
Date Published
Author
Nicolas Chariglione
Word count
1574
Language
English
Hacker News points
None

Summary

Koyeb, a serverless platform provider, faced a latency issue with their global load balancing system, which caused delays in deploying applications. The issue was traced back to Consul's caching behavior, which unexpectedly increased CPU load due to the frequent calls to the Catalog.ListServices() endpoint. Despite not having made recent changes to their code, Koyeb discovered that enabling the UseCache option in their CoreDNS plugin inadvertently overloaded the Consul servers with background queries. This problem was exacerbated by the frequent changes in the app list, which invalidated the cache and caused the Consul Agent to repeatedly query the servers. To resolve the issue, Koyeb disabled the built-in cache parameter and implemented a minimal in-process cache to stabilize their system. This experience highlighted the importance of understanding the internal workings of tools and the potential pitfalls of hidden caching mechanisms.