Data quality testing

Data observability is a crucial aspect of ensuring accurate, consistent, complete, and reliable data for business operations and analytics. It involves validating data against predefined rules and criteria. At incident.io, dbt's native testing features are used to ensure the reliability of their data. They faced two common challenges: relationship tests failing due to timing issues in the ingestion pipeline, and a cumbersome debugging process for failed tests. To address these issues, they implemented customized logic within their data pipeline and utilized dbt's store_failures feature.