Company
Date Published
Author
Dimitrios Athanasiou • Pete Harverson • Pete Harverson
Word count
2181
Language
English
Hacker News points
None

Summary

Elastic machine learning anomaly detection is a fully unsupervised method that builds a dynamic model of the data in real-time to automatically identify statistically anomalous events. However, to refine this detection process, domain knowledge is crucial, prompting the introduction of custom rules in version 6.4 of the Elastic Stack. These custom rules allow users to incorporate domain-specific insights to adjust the behavior of anomaly detectors, enhancing the relevance of detected anomalies. For instance, a custom rule can be set to prevent anomaly alerts for low CPU utilization that falls below a certain threshold, ensuring only significant deviations are flagged. Similarly, in security analytics, rules can exclude known safe domains from triggering alerts, thereby focusing on potentially suspicious activities. The rule actions, including 'skip_result' and 'skip_model_update,' provide flexibility in handling anomalies by either preventing their creation or influencing model updates. Custom rules can be applied to both numerical and categorical data, and while they immediately impact new results in real-time jobs, applying them to historical data requires cloning and re-running jobs. This functionality enhances anomaly detection by aligning it with the user's domain expertise, allowing for more meaningful scoring and ranking of anomalies.