October 2025 Summaries

Fly.io transforms Docker containers into micro-VMs called Fly Machines, utilizing a unique orchestration model where individual servers act as the source of truth, rather than a centralized database, allowing for global scaling without bottlenecks. The platform encountered its worst outage when a new Fly Machine configuration led to a deadlock across its network due to a concurrency bug in Rust, emphasizing the challenges of distributed systems. To improve global routing and state synchronization, Fly.io developed Corrosion, a service discovery system that uses a gossip protocol inspired by routing protocols like OSPF, eschewing traditional distributed consensus methods. Corrosion propagates a SQLite database across a global WireGuard mesh without central servers, enabling rapid synchronization while reducing state conflicts. Despite setbacks, such as costly schema changes and dual system dependencies, Fly.io iterated on Corrosion by implementing watchdogs, simplifying updates, and regionalizing the system to limit the impact of state bugs, ultimately achieving a robust and efficient distributed system.

Fly.io experienced a phishing attack that compromised their Twitter account, highlighting vulnerabilities due to outdated security practices and a lack of phishing-resistant authentication. The attack stemmed from a well-crafted phishing email targeting CEO Kurt Mackey's insecurities, leading him to inadvertently share credentials. Although the breach caused temporary loss of account control and brand damage, no sensitive user data was at risk. Fly.io emphasizes the importance of adopting modern security measures like phishing-resistant multi-factor authentication (MFA) and single sign-on (SSO) to prevent such incidents in the future. The company's relaxed tone indicates the incident's limited impact, and they use it as a learning opportunity to improve their security practices while humorously reflecting on their social media strategy.

Ben Johnson discusses the latest updates and developments for Litestream, an open-source tool designed to enhance SQLite's resilience to server failures by streaming WAL checkpoints to object storage for real-time backup and recovery. Recent updates include faster performance and support for point-in-time recovery, achieved through the introduction of the LTX file format, which allows for efficient transaction-based data reconstruction and compaction. The focus has shifted back to Litestream from LiteFS due to user preference for its simplicity and ease of use, leading to the integration of transaction-aware features inspired by LiteFS. The new version, Litestream v0.5.0, simplifies backup management by eliminating the concept of generations and introducing transaction IDs for tracking database states, though it limits each database to a single replica destination to maintain consistency. Additional upgrades include enhanced file compression, support for NATS JetStream replication, and compatibility improvements with modern SQLite libraries and cloud storage providers. Future enhancements will include a virtual file system (VFS) for read replicas, allowing for immediate database access while data is still being retrieved.