March 2025 Summaries

Dagster University has announced a new course on testing with Dagster, emphasizing the importance of testing in data engineering to build with confidence. The six-lesson course covers various aspects of testing, including unit testing, asset tests, integration testing, and best practices for handling external dependencies. It also provides examples, such as mocking API calls, to demonstrate how to test Dagster code effectively. The course is free and available to everyone, allowing users to track their progress and learn at their own pace.

Dagster+, a tool for data platforms, has revamped its alerting tooling to improve observability and monitoring. The new features aim to provide meaningful alerts that stakeholders can trust, including customizable alert configurations, clear and actionable notifications, and reliable testing tools. With the updated tooling, users can manage their alerts through either the user interface or a config file, ensuring seamless integration with CI/CD pipelines. As part of Dagster's continued investment in observability, future improvements will focus on areas like asset freshness SLAs, understanding trends in platform health, and managing pipeline costs.

Gu Xie, Head of Data Engineering at Group 1001, transformed the company's data platform using Dagster, achieving remarkable speed and efficiency. By adopting a platform approach, Gu's team built a core component around Dagster, empowering the organization to accelerate its strategy execution. The team automated processes such as handling PDF files, processing 1035 exchanges, and migrating legacy systems, resulting in significant growth and increased velocity. With a focus on maintainability, automation, and collaboration, Group 1001 achieved high development velocity by removing friction for development teams and prioritizing speed without sacrificing quality.

The text compares building data pipelines with Dagster and Airflow. Dagster takes a different approach by viewing everything as an asset, rather than focusing on DAGs and tasks. This allows for more flexibility and scalability in managing dependencies between assets. Dagster also encourages explicit code writing over templating or Jinja, which is used in Airflow. The documentation and UI experience are also highlighted, with Dagster's supporting Python docstrings making it easier to document assets, while Airflow's UI can be cryptic. Finally, the text discusses how Dagster makes it possible to launch a UI quickly and easily using the `dagster dev` command, which allows for more manageable and scalable graph management compared to Airflow.