September 2020 Summaries

Apache Spark is a powerful tool for data processing, but its development can be challenging due to the need for different setups for various stages of the development cycle. These setups demand drastically different configurations, such as local setups with small data for quick error detection, representative sample datasets for data edge cases, and production-sized datasets on clusters for performance issues. Dagster, a data orchestrator, addresses this complexity by organizing Spark code and deployment setups, providing pre-built utilities for deploying Spark code to environments like EMR and Databricks. It cleanly separates business logic from setup configurations, defining DAGs of Python functions called solids that can be run in different modes, such as local development, EMR, or production. The use of Dagster's integration with PySpark allows developers to easily switch between these setups and deploy code on each job run, automating packaging and S3 upload for a tighter development loop.

The Dagster community has had their first regular monthly update meeting, where members can share what they're working on and ask questions to the core dev team. The next meeting is scheduled for October, with recorded meetings available for those who cannot attend live. The 0.9.0 release featured a focus on bugfixing, internal refactor, and hardening, including user code isolation, Kubernetes hardening, scheduling improvements, and the introduction of an experimental hooks API. The team is working towards a more fault-tolerant scheduler in 0.10.0, with features like dynamic orchestration, cross-DAG dependencies, multi-container orchestration, and event-driven scheduling. Prezi has successfully migrated about 10% of their jobs to Dagster, with plans to migrate the remaining jobs, and is excited about improvements to backfill capabilities on the roadmap for 0.10.0. The community is encouraged to reach out with feedback, questions, or suggestions for future features.