February 2024 Summaries

In January 2024, a decision was made to transition the production database from PlanetScale to AWS Aurora RDS due to several challenges and limitations experienced with PlanetScale, including the lack of a true serverless offering, opaque pricing, and insufficient control over server resources. Initially drawn to PlanetScale for its marketed serverless capabilities and simplicity of billing based on rows read or written, the database's shift in focus towards high-performance offerings, alongside unexpected increases in CPU usage and costs following an upgrade, prompted the move. Despite PlanetScale's initial benefits, such as ease of scaling and performance, its limitations—including no private network options, lack of foreign key support, and manual server monitoring—became increasingly problematic. The migration to AWS Aurora RDS, though complex and costly, was seen as a necessary step to accommodate the company's growth, offering advantages like auto-scaling, private networking, and multi-AZ failover, despite requiring additional planning and optimization efforts.

A team successfully reduced their GitHub Actions usage from 46,144 minutes to 19,155 minutes, cutting their bill by 63% and speeding up pull requests by eliminating unnecessary CI checks. Initially, all checks ran for each pull request regardless of whether changes were relevant, which led to inefficiencies and higher costs. By utilizing an open-source Action called paths-filter, they were able to detect file changes and conditionally run relevant checks, significantly reducing server usage and wait times. Looking ahead, the team plans to further optimize costs by using custom runners, as the default GitHub runners are comparatively expensive. They are considering using AWS EC2 instances, which offer more resources at a lower cost, or a service called Warpbuild for easier setup. Additionally, they are developing an alternative to AWS CodeBuild for faster builds and aim to use their build system as GitHub Action runners.

Flightcontrol, a layer on top of AWS, faces a multitude of errors stemming from various sources, including user code, AWS edge cases, and third-party outages. To manage this complexity, a robust error system was developed by senior engineer Camila Rondinini, which involves categorizing errors based on their origin, providing unique codes for easy identification, and offering detailed user-facing information to facilitate resolution. The error handling framework includes a centralized error dictionary, full type safety in the code, and a structured approach to storing errors in a database, allowing for retroactive updates to user-facing content. The system aims to display comprehensive error information to users for expected issues while minimizing details for unexpected errors, which are silently logged for engineering review. Despite acknowledging room for improvement, the implementation supports ongoing refinements and is openly shared for community feedback and collaboration.