Building a modern Durable Execution Engine from First Principles

Restate is a robust Durable Execution Engine designed from foundational principles, aiming to offer a complete and self-contained stack that rivals the best log systems in durability and operations without relying on traditional databases or logs. Built as a single Rust binary, it combines a command log and event-processor to enhance distributed coordination and simplify operations, contrasting with traditional wisdom that suggests using existing stateful systems like Postgres. Restate operates through a server handling invocations and application services that execute durable functions, maintaining a bidirectional communication for durable actions and scaling efficiently on serverless infrastructure. It employs a partitioned scale-out model, where each partition operates independently, allowing efficient scaling and seamless event handling. The system features a unique architecture that integrates a fast replicated log, Bifrost, and utilizes object storage for data durability, balancing between latency, cost, and disk usage. The control plane coordinates the data plane, ensuring efficient failover and leader election, while maintaining a simple setup adaptable to various cloud and on-prem environments. Restate's design, focusing on durability, resilience, and operational tooling, provides impressive performance metrics, with plans to further optimize and extend its capabilities in upcoming releases.