Company
Date Published
Author
Sandy Ryza
Word count
341
Language
English
Hacker News points
None

Summary

Sandy Ryza, a lead engineer on the Dagster project, joins Tobias Macey on The Machine Learning Podcast to discuss how his background in machine learning has informed his work on the Dagster project and its foundational principles that allow for collaboration across data engineering and machine learning concerns. Building complex machine learning models requires an orchestration engine like Dagster that understands the data it is manipulating, enabling the creation of repeatable workflows and managing experiment iterations. Ryza shares insights into how he balances flexibility with repeatability in ML projects and highlights innovative ways Dagster has been applied to machine learning. He also discusses lessons learned from working on orchestrating ML workflows and what's planned for future improvements in ML support within Dagster.