How Kestra's Assets Give You Complete Pipeline Lineage

Kestra 1.2 introduces a feature called Assets, which provides a comprehensive approach to tracking both data artifacts and infrastructure resources within workflows, offering a complete pipeline lineage. This feature allows users to declare what each task in a workflow reads and writes, encompassing data assets like tables and datasets, as well as infrastructure elements such as cloud buckets and virtual machines. By automatically generating a dependency graph, Kestra enables users to trace the full chain of dependencies, thus enhancing visibility into the entire data and infrastructure ecosystem. This capability addresses the limitations of traditional orchestration tools that primarily focus on data assets, offering a more integrated view that includes infrastructure components. Kestra's Assets are particularly beneficial for data teams managing multi-layer data pipelines, infrastructure provisioning, and operations, as they facilitate asset-based targeting and integration with external lineage systems like OpenLineage. Available in Kestra Enterprise Edition 1.2 and above, the Assets feature supports efficient collaboration and change management by maintaining an up-to-date inventory of all pipeline components, thereby simplifying the tracing of impacts and dependencies across systems.