Company
Date Published
Author
Kyle McNair
Word count
920
Language
English
Hacker News points
None

Summary

Monitoring source data freshness in dbt can be challenging due to the manual configuration of expected cadences, which often leads to alert fatigue from false alarms. To address this, the integration of Datafold alerts, which utilize machine learning-based forecasting models, allows for the dynamic configuration of thresholds that adapt to data seasonality and trends, reducing noise and improving response efficiency. By creating a dbt snapshot of the information schema and writing SQL queries to track table changes, users can leverage Datafold to detect anomalies in data freshness. This approach simplifies the alerting process, enabling alerts to be configured easily and delivered via Slack or email, although it is specifically applicable to materialized tables, as views lack row counts. The method is demonstrated using the Snowflake database but can be adapted for other databases.