Minimizing Data Science Model Drift by Leveraging PagerDuty

PagerDuty's Early Warning System (EWS) model is a crucial tool used by the Customer Success and Sales departments to evaluate customer wellness and predict potential account churn based on product usage and external business factors. In January 2021, an upstream change led to an inaccurate customer risk score, highlighting the need for robust monitoring to prevent such issues in the future. The DataDuty team implemented automated tests and PagerDuty alerts to detect model drift, which can manifest as concept, data, or upstream drift. The team's strategy involves using various statistical tests to ensure the model's accuracy, such as Cohen’s d, kurtosis, the Kolmogorov–Smirnov test, and T-Tests, while maintaining data integrity through tools like Snowflake, Monte Carlo, Apache Airflow, and Databricks. An incident involving a data anomaly was swiftly resolved using these measures, demonstrating the effectiveness of PagerDuty's incident triage capabilities. This proactive approach ensures model reliability and maintains stakeholder trust, emphasizing the importance of monitoring and alerting systems in managing critical data science infrastructure.