Dec 12-17 Network Incident Report

Between December 12 and December 17, Semaphore experienced network instabilities due to a faulty device at a Tier 1 network provider, affecting communication with platforms like GitHub and Docker. The issue began with high network latency and sporadic TLS handshake timeouts, prompting Semaphore to collaborate with its hosting provider to trace the root cause, which was initially elusive. Despite efforts such as altering routing paths, deploying proxy servers, and running comprehensive network tests, the problem persisted until collaboration with GitHub led to the detection and replacement of a faulty line card by Telia, a network operator. Semaphore's response included setting up a new build cluster and planning for future on-demand multi-region provisioning to mitigate similar issues swiftly. The company acknowledged the importance of reliable service and committed to enhancing monitoring and rapid response capabilities to prevent future disruptions.