September 2022 Summaries

Tailscale's September newsletter highlights the company's recent developments and community contributions, underscoring its growth and commitment to open-source values. The newsletter introduces a new reward system that increases device limits for users sharing nodes, and it announces various job openings for motivated individuals comfortable with asynchronous collaboration. Community contributions include discussions on Tailscale's NAT traversal technology, a significant funding boost, and user experiences with Tailscale in managing infrastructure and VPN connectivity. The newsletter also covers technical insights, such as using Tailscale with Raspberry Pi, GitOps for ACLs, and enhancements to their DERP relay network. Additionally, it delves into passing a SOC 2 Type II audit, privileged access management, and securing Remote Desktop Protocol, while promoting upcoming events like GopherCon Chicago and Monktoberfest.

Tailscale is presented as a modern evolution of Hamachi, building upon its foundational concepts to create a more advanced and secure networking tool. While Hamachi allowed users to connect devices for gaming and file sharing, Tailscale enhances this by integrating identity verification through identity providers, allowing for more precise access control and security. Unlike Hamachi's reliance on network numbers and limited control over user permissions, Tailscale uses access control lists and identity-based memberships to manage access more effectively, addressing potential security risks and conflicts. Tailscale also offers broader platform support, including open-source components and the ability to self-host its infrastructure. By utilizing a private address range and supporting numerous operating systems, Tailscale ensures stable IP addresses and connectivity across diverse devices. Its versatility extends beyond basic networking, facilitating collaborative environments for sharing applications and services among colleagues, friends, and family.

Tailscale, a company built on open source principles, heavily relies on and contributes to open source development, including the core use of the WireGuard protocol. Initially developing their project in a closed repository, Tailscale eventually open-sourced their client software to enhance transparency and trust, especially for Linux and Android, while keeping GUIs for proprietary systems closed due to the complexity involved. Although their coordination server remains closed source due to maintenance challenges, an independent open-source alternative called Headscale has emerged, which Tailscale supports as a complementary project. Tailscale actively engages with and contributes to the open source community through various initiatives, such as maintaining compatibility with Headscale, upstreaming changes, adopting community projects, and offering their service for free to open source projects. They emphasize transparency and community involvement, encouraging contributions and collaboration to enhance their products and the broader open source ecosystem.

Tailscale has enhanced its network by adding nine new DERP relay server locations globally, including cities like Amsterdam, Denver, and Hong Kong, to improve the reliability and speed of direct connections between clients. These servers are part of a global infrastructure that helps devices establish connections even in challenging network environments by using DISCO packets for communication and providing alternative paths when direct connections are not feasible. The addition of more DERP locations and the expansion of existing servers aim to reduce latency and improve throughput by ensuring devices are closer to a relay, which in turn lightens the load on current servers and enhances user experience. Users can utilize tools like "tailscale ping" and "tailscale netcheck --verbose" to identify the nearest DERP server and assess their connection path.

In an exploration of a peculiar issue with file descriptors at Tailscale, the team discovered a sawtooth pattern in the file descriptor count after deploying stateless reverse proxies to reduce HTTP connections to the coordination server. Despite the pattern being benign, it coincided with the release of a new server build, prompting an investigation. The root cause was traced to the autocert package's handling of Let's Encrypt certificates, which was attempting to renew certificates for a domain now served by reverse proxies. This resulted from users hardcoding DNS entries, causing certificate requests to reach the coordination server instead of the proxies. The problem was exacerbated when previous certificates expired and a server deployment reset caches, necessitating certificate re-requests. By removing the domain from autocert's allowed list, the team resolved the issue, highlighting the complexities of handling diverse client configurations and reinforcing the importance of detailed investigation to ensure seamless service operations.

Tailscale has achieved SOC 2 Type II compliance, a significant milestone that validates the effectiveness of its security controls over a specified period, unlike the Type I audit, which only assesses the presence of policies. The audit process revealed challenges such as the time-consuming nature of defining internal processes and the limitations of using automated tools due to Tailscale's non-standard infrastructure. Tailscale developed its own tools and processes, including the ToBeReviewed Bot for managing production changes and open-sourcing security policies, ensuring these controls are tailored to its specific business needs. Despite completing the audit, Tailscale acknowledges areas for improvement, such as streamlining vendor security reviews and enhancing endpoint security, and continues to work on making its SOC 2 report more accessible while maintaining transparency about its security practices.

The blog post discusses implementing a GitOps workflow to manage Tailscale Access Control Lists (ACLs) using GitHub Actions, which allows for the centralized management of ACLs in a Git repository with "config as code" practices. This setup ensures that changes to the tailnet policy file undergo review and testing similar to code changes, leveraging the Sync Tailscale ACLs GitHub Action to automate updates and enforce version control. Users are warned about the GitOps setup in the admin console to prevent unauthorized changes, although emergency edits remain possible. While this approach offers a structured method for maintaining ACLs, it is noted that Tailscale API keys require regular renewal every 90 days, which currently must be done manually.