Supertoast tables

For the Hatchet team, optimizing their web application’s data storage involved initially leveraging Postgres due to its general-purpose capabilities and familiarity among engineers. However, the challenge arose when storing large jsonb data, which led to inefficient storage usage and prolonged autovacuum operations due to TOAST tables. To address this, the team developed the "supertoast" table approach, which involved partitioning data by time and offloading older, infrequently accessed payloads to Amazon S3 while maintaining references in Postgres. This method, which replaced a less efficient write-ahead log (WAL) strategy, allowed them to batch compress and store payloads in S3, significantly reducing costs and operational overhead. The innovative "write-and-swap" process, involving creating partition copies and safely swapping them post-processing, enabled efficient data management without increasing CPU usage or S3 expenses, proving highly effective in managing hundreds of millions of payloads daily.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.