How Blacksmith used Tailscale Services to work around ISP degradation

Aditya Maru, co-founder of Blacksmith, details how the company utilized Tailscale Services to address ISP degradation affecting their continuous integration (CI) infrastructure, specifically with GitHub traffic. Faced with an outage on Thanksgiving that caused GitHub Actions jobs to fail due to upstream ISP issues, Blacksmith implemented a transparent proxy solution without requiring changes from their customers. This proxy reroutes GitHub-bound traffic through a network path with direct GitHub peering using Squid for proxying and caching, while Tailscale Services provide secure load balancing and encryption. By leveraging a Linux kernel data structure called ipset for efficient packet matching, Blacksmith was able to circumvent problematic network routes and maintain seamless service. The implementation also benefits from Tailscale’s intelligent routing, which replaces traditional load balancing methods, ensuring that only authorized devices can access the proxy service. This solution not only resolved the immediate issue but also provides a sustainable disaster recovery mechanism for future connectivity disruptions, with potential expansions to other critical services.