How to Do Automatic Annotations with Grafana and Loki

Grafana and Loki enable automatic annotations on Grafana dashboards by using Loki queries to correlate events with metrics, facilitating better event tracking and analysis for operators and developers. Since version 6.4.0, Grafana has allowed the use of Loki queries to automatically generate annotations, where each log line from a query is displayed as an annotation at the correct time. By setting up an annotation query in Grafana's dashboard settings and selecting a Loki data source, users can specify log lines to appear as annotations, such as changes in a Kubernetes deployment. The kubernetes-diff-logger application logs changes to Kubernetes objects, and Promtail can be used to extract JSON fields as Loki labels, making them available for direct querying. This setup helps users correlate changes in their deployments with metrics such as memory usage, latency, or success rates, providing clear visual markers on Grafana dashboards. The technique is especially useful for tracking infrastructure changes in Kubernetes environments and is open to contributions from those interested in expanding the capabilities of the kubernetes-diff-logger project.