Prometheus Backfilling: Recording Rules and Alerts

Prometheus users often face challenges with recording rules and alerts, as they are generated in real-time and do not apply to historical data, creating difficulties in testing their impact. To address this, a utility called promutil.py has been developed to generate OpenMetrics data, which can be used to backfill Prometheus starting from release 2.25. This allows users to run experiments on separate servers with historical data, helping them assess the effect of new alert rules without affecting production systems. The process involves using promutil.py to produce OpenMetrics files, which are then used to create Prometheus blocks via promtool, enabling the inclusion of historical data in recording rules and alerts. Although promutil.py does not generate historical alerts, it can create corresponding metrics for alert testing and is also useful for ad hoc alert management, offering a valuable tool for system diagnostics and monitoring enhancements.