How we use metamonitoring Prometheus servers to monitor all other Prometheus servers at Grafana Labs

Grafana Labs employs a metamonitoring system using Prometheus servers to monitor other Prometheus servers, ensuring that monitoring failures are quickly identified and addressed. This system involves a network of geographically distributed metamonitoring Prometheus servers that monitor each other across clusters, with a security mechanism similar to a dead-man’s-switch. The setup includes high-availability (HA) Prometheus pairs within Kubernetes clusters, a global Alertmanager cluster, and the use of Vault for managing authentication and secrets across clusters. Alerts are routed through Prometheus to Alertmanager and finally to PagerDuty, with a heartbeat system in place using Dead Man’s Snitch to ensure notification even if the alerting chain fails. This approach ensures redundancy and reliability, allowing Grafana Labs to maintain observability and sound alerts during outages of any part of the monitoring infrastructure.