Running a Billion Workflows a month with Orkes Conductor

Conductor, a platform for service orchestration, enables developers to build stateful applications in the cloud without concerns about resiliency, fault tolerance, or scalability. A benchmark experiment demonstrated Conductor's scalability, testing its capacity to handle significant workloads using a single-node setup on a MacBook Pro. The experiment measured throughput and latency, achieving consistent task execution at over 1450 tasks per second and processing data at approximately 80 MB/sec. The setup involved a simulation of a sustained workload of 210 workflows per second, which translates to about 540 million workflows monthly. Utilizing tools like wrk2, Prometheus, and Grafana, the experiment confirmed Conductor's ability to scale, emphasizing the importance of optimizing server configurations and using Redis and Postgres efficiently. Conductor's stateless server design allows horizontal scaling, and with adequate hardware and configuration, it can handle billions of workflows monthly. Founded by the original Netflix Conductor engineers, Orkes offers a managed service version of Conductor, available for developers interested in exploring its capabilities.