Storage challenges in the evolution of database architecture
Blog post from Postman
Postman's Sync service, an integral part of its backend ecosystem since 2014, recently faced storage issues due to its use of AWS RDS Aurora, which has a 128TiB storage limit. With a rapidly increasing data ingestion rate, the team anticipated hitting this limit within 6-8 months, potentially affecting the service's reliability. The Postman engineering team explored several strategies, such as pruning old data and creating dual tables, but these initial methods faced challenges due to MySQL's storage management and the risk of downtime. Eventually, they implemented a three-step approach: removing unused tables, adjusting strategies for append-only tables, and performing cross-cluster replication to safely free up space. This complex operation involved spinning up a secondary cluster (C2) from the main one (C1), executing TRUNCATE operations, and ensuring seamless replication between the clusters. Their efforts resulted in freeing up 60TB of critical database space and reducing the data ingestion rate, all while maintaining platform availability during a designated maintenance window. The team is now focused on designing a next-generation database infrastructure, considering sharding, partitioning, and data isolation to enhance scalability and reliability.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.