Stop Reinventing Orchestration: Embedded ELT in the Orchestrator

The text discusses the challenges of data ingestion, a crucial part of the data engineering lifecycle. It highlights the limitations of traditional approaches, such as rolling one's own solution or relying on heavyweight third-party frameworks, and notes that many teams struggle with the complexity of building reliable ingestion pipelines. The author introduces Dagster Embedded ELT, a new library that aims to simplify data ingestion by providing pre-built assets and resources around lightweight frameworks. This allows users to build ELT pipelines with Dagster without duplicating the complexity of heavier ingestion frameworks. The library is designed to address key pain points in data ingestion, including observability, error handling, state management, data quality, type conversions, schema drift, and more. By providing a simpler and more cost-effective solution, Dagster Embedded ELT aims to make data engineering easier and more accessible.