How to Capture Domain Knowledge in Elastic Machine Learning Jobs with Custom Rules

Elastic machine learning anomaly detection is a fully unsupervised method that builds a dynamic model of the data in real-time to automatically identify statistically anomalous events. However, to refine this detection process, domain knowledge is crucial, prompting the introduction of custom rules in version 6.4 of the Elastic Stack. These custom rules allow users to incorporate domain-specific insights to adjust the behavior of anomaly detectors, enhancing the relevance of detected anomalies. For instance, a custom rule can be set to prevent anomaly alerts for low CPU utilization that falls below a certain threshold, ensuring only significant deviations are flagged. Similarly, in security analytics, rules can exclude known safe domains from triggering alerts, thereby focusing on potentially suspicious activities. The rule actions, including 'skip_result' and 'skip_model_update,' provide flexibility in handling anomalies by either preventing their creation or influencing model updates. Custom rules can be applied to both numerical and categorical data, and while they immediately impact new results in real-time jobs, applying them to historical data requires cloning and re-running jobs. This functionality enhances anomaly detection by aligning it with the user's domain expertise, allowing for more meaningful scoring and ranking of anomalies.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.