Company
Date Published
Author
Kiruthika Meena Ravichandran
Word count
1035
Language
English
Hacker News points
None

Summary

Harness engineers identified and resolved a critical memory leak issue in their CI/CD delegate service, caused by an improper use of Go's context.Context within worker loops. This leak was revealed through a tight correlation between CPU and memory usage, indicating that garbage collection struggled to manage memory efficiently. The problem arose because reassigning context variables in a loop created a chain of contexts that the garbage collector could not free, leading to exponential memory retention across thousands of goroutines. By reducing the test environment to a single worker, the engineers pinpointed that each task's context was improperly linked to the previous one, preventing cleanup. The solution involved altering the variable scoping to prevent these chains from forming, highlighting the importance of careful variable management within loops. The resolution of this issue also paved the way for further optimizations, as a profiler later uncovered an additional goroutine leak to be addressed in the future.