Company
Date Published
Author
Fraser Marlow
Word count
1984
Language
English
Hacker News points
None

Summary

In this article, the author shows how to set up a Singer ELT pipeline using Dagster. The setup involves creating reusable resources for the Tap and Target, defining assets that invoke these resources, configuring code location definitions, and setting up a simple job. The author highlights several advantages of running Singer Taps within Dagster, including making Singer an integral part of scheduled pipelines, adding metadata to taps and targets, retrying on failure, sending tap payload to multiple targets, and debugging without hitting the upstream API. However, it also notes that running Singer can increase latency and storage requirements. The author provides a step-by-step guide for setting up this pipeline, including code examples in Python. By leveraging Dagster's orchestration system, developers can create more integrated, observable, and trackable data pipelines using Singer.