February 2023 Summaries

The Dagster Labs team has been working on building Modern Data Stack (MDS) integrations available to Dagster users, saving teams time and allowing them to tap into more value of their chosen solutions. These integrations span various applications including Alerting, Data Quality, Deployment, Ingestion, ML Ops, Monitoring, Data Science, Secret Management, Storage, and Transformation. Recently, updates have been made to IO Managers for Snowflake, DuckDB, and BigQuery integrations, adding partition support and ergonomic improvements. Additionally, enhancements have been made to the Airflow migration integration and a new ML Ops integration with Weights & Biases has been added, allowing users to orchestrate their MLOps pipelines and maintain ML assets with Dagster. The team encourages users to explore the Dagster integrations and provides resources for building out new integrations.

With the release of `dagster-airflow` 1.1.17, data engineering teams can now easily transition Airflow DAGs to Dagster, providing a radically better developer experience. This tooling makes it possible for organizations to migrate away from legacy imperative approaches and adopt a modern declarative framework that offers excellent developer ergonomics. The `dagster-airflow` library enables users to lift-and-shift their existing Airflow DAGs into Dagster, or orchestrate Dagster job runs from Airflow, providing a seamless transition for data teams looking for an Airflow alternative. By using the `dagster-airflow` library, organizations can boost their team's ability to rapidly develop, deploy, and maintain high-quality pipelines, delivering more effectively against stakeholder expectations.