Company
Date Published
Author
Chris Nagele
Word count
650
Language
English
Hacker News points
None

Summary

The Postmark email service experienced a major four-hour outage from 4:49am to 8:49am EST, which was the longest outage since its launch in 2010. The outage occurred due to an update to a database table that caused a background services failure and resulted in messages being queued but not sent during the downtime. Despite initial attempts to recover the table, it took manual intervention and recreating the table from scratch to resolve the issue. Fortunately, there were no reported lost emails, and sending and inbound functionality was restored after four hours. To avoid similar issues in the future, Postmark plans to have a plan B in place for unexpected problems and is working with Percona to identify potential solutions.