February 2025 Summaries

The authors of the service experienced an incident where their proxy servers were stuck in a busy-loop, causing them to receive and bounce multiple pings. The issue was eventually traced to a bug in the Rust `tracing` library's `Subscriber` module, which is used for fine-grained span records. The problem arose because the `poll` function of a `Future` returns immediately when it reaches a `Ready` state, but the underlying async executor doesn't actually progress its state machine. This caused an infinite loop in the proxy servers. The issue was found to be related to the `TlsStream` type from Rust's `tls` library, which is used for TLS connections. A bug in this type caused it to spin out and trigger a busy-loop when a certain condition was met. The authors were able to identify the issue by analyzing the stacktrace and using instrumentation to monitor their proxy servers. They were able to fix the issue by updating the `tls` library and implementing additional testing infrastructure to detect such bugs in the future.

We're building a public cloud on our own hardware, and one of our bets was to GPU-enable our customers. We created Fly GPU Machines, which are Docker/OCI containers running inside virtual machines with Nvidia GPUs. However, we underestimated the importance of AI/ML and overestimated the demand for GPU-enabled workloads. The main problem is that developers don't want GPUs; they prefer using LLMs like OpenAI's GPT. Our hardware product/market fit is rough, and we're scaling back our GPU ambitions without sacrificing our isolation story or developer experience. We learned that our customers didn't want a new Javascript runtime but just wanted their native code to work, and similarly, we were wrong about GPUs. This experience has taught us the importance of being open to learning from our mistakes and adjusting our bets accordingly.

JP Phillips, the architect of Fly Machines, has left his position after 4 years to join another company. He cited a mismatch between his desired focus and the direction of Fly Machines as the reason for his departure. Despite this, he expressed pride in several key projects, including the development of deterministic and durable execution capabilities using Cadence technology. Phillips also highlighted the importance of platform components like corrosion2, which enables SQLite queries to access near-real-time information about any Fly Machine around the world. He praised the Fly Machines API, citing its ability to run workloads from an OCI image and API call across the globe, and noted that his most impressive achievement was building the Fly proxy team's codebase. In a lighthearted tone, Phillips expressed frustration with certain aspects of the platform, including GraphQL and Elixir, while also acknowledging the value of tools like OpenTelemetry for troubleshooting and debugging. He offered words of caution to future candidates regarding the company's management structure and communication style, but concluded by expressing gratitude for his time at Fly.io.

The concept of "closed-loop" LLM agent code generation, where the generated code runs and observes its own interactions with users, is gaining traction in software development. This approach allows for more efficient bug fixing, testing, and error handling, as the LLM can learn from its mistakes and improve over time. The idea is to use system prompting to enable an LLM to generate Semgrep rules, test them, and fix errors, potentially automating tedious tasks like predicting potential issues and manually testing codebases. This could revolutionize the way software development teams work, making it more efficient and effective.

We're shifting away from carefully choreographed EffortPosts, which were our primary outreach tool for writing content that resonates with Hacker News. This change comes after realizing that the team's focus on producing high-quality content was becoming an impediment to publishing more frequently and exploring other topics. The writer acknowledges their past mistakes in overthinking the approach and being too focused on avoiding repetition. Instead, we're embracing a more relaxed style, allowing for longer-form blog posts that prioritize quality over optimization for Hacker News, and looking forward to sharing more content with our audience.

VSCode has a feature called Tramp that allows users to extend its functionality by running Bourne shell commands in an interactive environment, such as an SSH session. In contrast, VSCode's remote editing feature over SSH mounts a full-scale invasion of tools and agents, including a binary installation of Node, which can run arbitrary files, launch shell processes, and persist itself, raising security concerns. The author is concerned about the potential risks of using this feature on development servers or during incidents in production, but notes that a custom connection to a Fly Machine can be established without these issues. As a result, the blog will return to its previous format, with the author sharing knowledge and insights through writing rather than relying on code demonstrations.