May 2025 Summaries

Hatchet, a platform for managing background tasks and data pipelines at scale, often encounters issues with blocked event loops, which can cause warnings and performance bottlenecks. This problem frequently arises when using Hatchet's Python SDK due to blocking I/O operations within the event loop, typically caused by CPU-bound tasks or synchronous operations, such as API calls or database queries, that should not be run directly in async code. To address these issues, developers are advised to replace synchronous operations with asynchronous alternatives, offload CPU-bound work to separate threads using asyncio.to_thread, and use debugging tools like asyncio's DEBUG mode and OpenTelemetry Instrumentor to identify and resolve bottlenecks. By following these best practices, the performance of Hatchet workers can be optimized, reducing unnecessary task delays and mitigating the occurrence of alarming warning messages.

Over a six-month period, Hatchet explored the scalability of Postgres by running benchmarks to address performance issues observed in their database, which is critical to their task queue and monitoring systems. They discovered that with simple optimizations like reducing network latency and using connection pooling, as well as more complex strategies such as batched inserts and leveraging the COPY...FROM command, significant performance improvements could be achieved. While increasing throughput, these tweaks also highlighted the trade-offs between throughput and latency, suggesting that the optimal setup depends on balancing batch sizes and connection management to avoid overhead and bottlenecks. The team's findings underscore the potential to enhance Postgres performance without switching to specialized data stores, despite the challenges of managing shared resources like CPU and locks. Future explorations will delve into more nuanced scenarios, such as handling foreign keys and employing alternative insert strategies, illustrating the depth of optimization possible in Postgres environments.