Re-architecting Trino: The development of fault-tolerant execution

Trino, originally developed as a fast, interactive query engine to replace Hive, faced limitations when handling batch and ETL workloads, notably encountering a "memory wall" that required either costly scaling or query fragmentation to manage large datasets. To address these challenges, Trino's architecture, which initially relied on massively parallel processing, was re-engineered to incorporate fault-tolerant execution, enabling queries to continue processing despite individual task failures, thus reducing wasted computation and resource use. This new architecture, introduced at Datanova 2023, allows for more efficient resource management, flexible query execution, and the ability to dynamically adjust execution strategies midstream, offering users the ability to run large queries with fewer resources. This advancement is accessible through Starburst Galaxy, where users can explore the fault-tolerant execution mode by creating a free cluster and querying without restarting from scratch in case of task failure.