Company
Date Published
Author
Natalie Nagele
Word count
507
Language
English
Hacker News points
None

Summary

The application experienced a maintenance process on Saturday morning that was expected to be low-impact but ultimately resulted in significant outages and delays, with email sending being slowed down. The team realized that the transfer of data from the old database to the new one was incomplete, causing inconsistent data in test accounts and delayed sending. To fix the issue, the application was put in emergency offline mode and the old database dump was reimported, but some records were still missing. A decision was made to revert back to the previous state, which resulted in lost data between 10am EST and 2:30pm EST, including new account sign-ups and credit deductions. The team is now investigating what went wrong and plans to migrate again next weekend, with a plan to notify customers of any upcoming downtime.