Migrating from Temporal to a Postgres-based task orchestrator

Nango, an open-source developer tool, transitioned from using Temporal to a custom-built queuing and scheduling system on Postgres to handle interactions with third-party APIs due to the challenges of running untrusted user code and enterprise adoption barriers. The team evaluated several alternatives, considering scalability, minimal dependencies, and flexibility, ultimately opting to create their own system using Postgres due to its simple yet effective locking and concurrency capabilities. This decision allowed them to remove a complex external dependency and gain more control over their infrastructure, while managing millions of tasks efficiently without downtime during the migration process. The implementation involved creating a tasks table in Postgres, using built-in locking mechanisms to handle task scheduling, and overcoming performance bottlenecks by linking tasks in a series. The migration strategy involved running the new system in parallel with the old one, using feature flags to switch accounts gradually, and ensuring seamless performance, which proved successful and introduced minimal overhead. Nango now utilizes Postgres for multiple workloads, including relational databases, object storage, and task orchestration, highlighting its versatility and reliability.