March 2026 Summaries

For the Hatchet team, optimizing their web application’s data storage involved initially leveraging Postgres due to its general-purpose capabilities and familiarity among engineers. However, the challenge arose when storing large jsonb data, which led to inefficient storage usage and prolonged autovacuum operations due to TOAST tables. To address this, the team developed the "supertoast" table approach, which involved partitioning data by time and offloading older, infrequently accessed payloads to Amazon S3 while maintaining references in Postgres. This method, which replaced a less efficient write-ahead log (WAL) strategy, allowed them to batch compress and store payloads in S3, significantly reducing costs and operational overhead. The innovative "write-and-swap" process, involving creating partition copies and safely swapping them post-processing, enabled efficient data management without increasing CPU usage or S3 expenses, proving highly effective in managing hundreds of millions of payloads daily.