Company
Date Published
Author
Adrien Grand
Word count
876
Language
-
Hacker News points
None

Summary

Elasticsearch 8.9 has introduced an optimization for cardinality aggregations through dynamic pruning, significantly improving query performance under certain conditions. This optimization can dramatically speed up cardinality computations, sometimes up to 1,000 times faster, by using index structures to dynamically reduce the set of matches needing evaluation. For example, when determining the number of unique Kubernetes deployments monitored by Elastic Kubernetes integration, the optimization efficiently filters out previously seen values, reducing unnecessary evaluations. The process involves introducing a filter on a disjunctive query to match only new values, which dynamically updates as new values are detected, thus skipping redundant document evaluations. The most significant speedups occur when all unique values of a field are identified early in the query evaluation, enabling Elasticsearch to bypass much of the index. However, this optimization is only applicable to fields with a relatively small cardinality, specifically on segments with no more than 1,024 unique values, and it is limited to keyword fields where the cardinality aggregation is the sole and top-level aggregation. This improvement has been notably beneficial in enhancing dashboard loading times within the Elastic Kubernetes integration, achieving a 90% reduction in latency for some queries.