New Relic's Dynamic Baseline Alerts use predictive analytics to help customers define dynamic, rather than static, alert thresholds. To determine what works best, they defined "better" in the context of helping customers and used both human and machine scoring. They developed a Mean Absolute Scaled Error method with two tweaks: evaluating predictions against anomaly-dampened observations and putting a guardrail around predictable series to prevent overly large errors. This method allows for analysis of regression and classification error, including precision, recall, and F1 scores. The system also includes human evaluation to ensure the alerts are useful on real-world systems. Dynamic Baseline Alerts are currently in limited release and will be generally available later this year, offering a great option for alerting on metrics with cyclical patterns.