December 2025 Summaries

Hatchet, a durable queue built on Postgres, faced significant challenges related to data partitioning and query performance. Initially, Hatchet stored hundreds of millions of tasks daily in a single Postgres table, which led to index bloat and inefficient storage management. To address this, Hatchet implemented a custom time-based partitioning system to manage large volumes of data, focusing particularly on tables storing task data and event histories. Despite initial success, performance issues arose when queries on partitioned tables began to slow down significantly, primarily due to incorrect query statistics caused by the autovacuum process not analyzing partitioned parent tables. Expert intervention revealed that manually running ANALYZE on these tables resolved the issue, highlighting the importance of maintaining accurate query statistics for partitioned tables. This experience underscored the need for strategic partition management practices, such as using DETACH PARTITION...CONCURRENTLY to minimize lock contention and manually analyzing parent tables to ensure optimal query planning.

In a detailed exploration of durable execution and its relevance to task queues and orchestration engines, the author reflects on their experiences at Porter, where they initially struggled to understand the benefits of Temporal for managing Kubernetes deployments. Through years of trial and error, and eventually founding Hatchet, they developed a deeper understanding of how durable execution platforms like Temporal work by persisting intermediate workflow states, enabling tasks to resume from checkpoints after failures. The text highlights the complexity of managing tasks that are not easily idempotent, especially those involving multiple microservices and distributed systems. Durable execution platforms offer solutions for handling unexpected failures, implementing complex workflows, and reducing unnecessary resource-intensive operations by leveraging a deterministic approach to maintain consistent execution history. However, the author cautions against over-reliance on durable execution, emphasizing the importance of understanding its limitations and the potential overhead involved.