Company
Date Published
Author
Rich Collier
Word count
841
Language
-
Hacker News points
None

Summary

Rich Collier's article explores the process of using custom Elasticsearch aggregations to manage derivative calculations in machine learning jobs, particularly focusing on detecting sharp rates of change in data, such as a "brown-out" recovery. The article emphasizes setting up a job configuration using the ML API in Kibana and creating a datafeed that defines the data source, which includes aggregating data through a date histogram and performing a sum aggregation followed by a derivative calculation. This process results in the creation of a new field called "orders_deriv" that is used by the machine learning job to detect anomalies, such as a significant recovery in order volume. Although the results can be viewed in Elasticsearch's UI, the article notes a limitation in visualizing derivatives over time due to constraints in the current Single Metric Viewer, which cannot dynamically reverse-engineer complex queries for raw data plotting. The author suggests that future updates may address this limitation while also recommending existing resources for machine learning users to enhance their workflows.