How Blacksmith survives ISP degradation with Tailscale Services
Blog post from Blacksmith
A Thanksgiving-day routing degradation at an upstream ISP caused 7–10% of GitHub-bound requests from a datacenter to stall for five to 20 seconds, disrupting GitHub Actions checkouts and other services such as APIs, packages, and ghcr.io for hundreds of customers. Because the affected ISP could only be identified by disabling providers individually and external routing problems can take days to resolve, the team developed a disaster-recovery mechanism that transparently redirects only GitHub traffic through a proxy network with direct GitHub peering, requiring no customer changes. Linux iptables and an ipset populated from GitHub’s published CIDR ranges efficiently identify and redirect relevant traffic, while ProxyManager recovers each packet’s original destination and sends it through HTTP CONNECT tunnels to Squid proxies. Tailscale Services provide the proxy pool with stable addressing, load balancing, health checks, WireGuard encryption, and tailnet-only access without a separate load balancer or publicly exposed proxy IPs. The system is being load tested and could later be adapted for other critical services with known IP ranges, including container registries and package repositories.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.