Build queue issues

Semaphore experienced significant issues following a planned platform update, leading to build and deployment delays due to unexpected build failures and decreased performance. The problems were primarily caused by the migration to a new OS version, Ubuntu 14.04, and changes in system libraries, affecting dependencies like the mysql2 gem. This necessitated a tool to clear dependency caches, which was promptly implemented. Additionally, the infrastructure faced a memory leak issue related to Java services, initially thought to be due to a switch to Oracle JVM, prompting a reversion back to OpenJDK. However, the true cause was a memory limiter on LXC containers, which was eventually reverted, resolving the build queue delays. Despite thorough pre-release testing, Semaphore acknowledged the challenges of replicating real-world scenarios and plans to enhance infrastructure testing by running parallel jobs with new platform versions to mitigate future disruptions.