Company
Date Published
Author
Adrian Brudaru
Word count
646
Language
English
Hacker News points
None

Summary

A single pane of glass for pipelines running on various orchestrators can greatly improve operational coherence, but managing multiple serverless run environments with different reporting mechanisms is a challenge. To address this, leveraging runtime metrics collected by the pipeline and using alerts instead of monitoring can be more cost-effective and efficient. Additionally, implementing governance measures such as annotating raw data schema and using views for transformed data can help stay compliant with regulations. Standardisation through tools like dltHub enables cross-orchestrator observability and monitoring, making it essential to introduce standardised data collection practices across different locations.