Company
Date Published
Author
Christian Hollinger
Word count
1869
Language
English
Hacker News points
None

Summary

Dagster is an open source data orchestrator and workflow management framework that helps orchestrate ETL and ELT processes by viewing result of your data as valuable data assets and emphasizing proper software engineering standards for data pipelines. It has a Python SDK with mostly decorator-based ergonomics, helpful features, and pleasant developer ergonomics. ngrok uses Dagster to run its entire data platform, managing an influx of about 20TB of net new data each month in over 400 Dagster assets and 170 distinct dbt models. The integration between Dagster and dbt allows for a fully modeled Dagster dependency graph with manageable Python code, while the ngrok Ingress Controller for Kubernetes provides protection from unauthorized access to Dagster instances.