August 2013 Summaries

AWS experienced a network and storage outage on Sunday, affecting some of its best-known services like Instagram, Vine, and IFTTT. The incident impacted 15-30 percent of one of the busiest data regions in the Amazon cloud, but the affected company's systems were able to withstand the interruption with minimal latency impact. The company uses Redis Cloud and Memcached Cloud services that rely on resources from the affected zone, but its in-memory databases are designed to operate without accessible storage, allowing them to work around the failure and recover themselves. The incident highlighted the importance of dynamic stability in system design and provided a test case for how such systems can cope with challenges when built for it.

The text discusses a challenge faced by users migrating their sharded Redis backend with minimal application downtime. It explains that traditional backup and restore mechanisms require the database to be stopped, which is unsuitable for some use cases. An alternative approach using database replication is introduced as a solution, reducing downtime to a minimum. The text then describes a tool called "redis-migrate" developed by the author's team to simplify this process. This Python script provides real-time information on replication status and guides users through each step of migration.

Maintaining high availability (HA) for in-memory datastores in Cloud environments is crucial for most applications, as data persistence and accessibility are essential requirements. While technologies like Redis and Memcached offer some built-in HA tools, they may not be sufficient or compatible with all cloud environments. To ensure HA, additional mechanisms such as replication, automatic failover detection, and backup storage should be implemented. In-memory replication offers near-instantaneous recovery but can be costly, while disk-based replication is more affordable but slower. Multi-AZ plans provide protection against entire availability zone failures by maintaining replicas in different zones. Data persistence mechanisms like snapshots and Append Only Files help recover from severe failures affecting multiple datacenters. Automated daily and on-demand backups to remote storage ensure data recovery even during major outages.