How We're Using Prometheus Subqueries at Grafana Labs.

Prometheus 2.7 introduced subqueries, a feature that allows for more flexible ad hoc querying by permitting range vectors of function outputs, which was previously not possible due to concerns about performance and incorrect usage practices. Grafana Labs utilizes subqueries to streamline processes in billing and capacity planning. For billing, subqueries enable the calculation of P95 data points per minute without relying on recording rules, thus allowing more dynamic and experimental queries. Similarly, in capacity planning, subqueries facilitate the assessment of per-container CPU usage by combining multiple queries into one, enhancing efficiency in monitoring and resource allocation across Kubernetes clusters. Despite their advantages, subqueries require careful handling, particularly avoiding summation before applying the rate function to ensure accurate calculations.