Company
Date Published
Author
Shalabh Chaturvedi
Word count
2101
Language
English
Hacker News points
155

Summary

Dagster Cloud, a serverless development platform, aims to improve the deploy speed of Dagster code from over 3 minutes to around 40 seconds. To achieve this, they implemented a system that uses Python Executable (PEX) files, which bundle Python packages and some bootstrap code into executable files. This allows for faster builds and deploys by avoiding the need to build and launch Docker images. The PEX tool provides features such as isolation, determinism, composition, and cross-platform builds. By using PEX in combination with S3 storage, Dagster Cloud can significantly improve deploy speed without sacrificing repeatability or consistency. However, there are trade-offs, including potential memory issues if multiple environments are run on the same container, and limitations when building for non-Linux platforms. The implementation also replaces traditional Docker-based workflows with PEX-based ones, eliminating the need to download and launch Docker images.