Practicing observability as code: Defining New Relic alert policies with Terraform

Observability is crucial for managing modern complex architectures, but creating an effective system with a high signal-to-noise ratio can be challenging. The concept of "observability as code," where configurations are automated and managed like code, is recommended to minimize human error, supported by the integration of tools like Terraform for Infrastructure as Code (IaC). This approach is exemplified using the New Relic Terraform provider to establish monitoring and alerting configurations based on the four golden signals of monitoring: latency, traffic, errors, and saturation. The process involves setting up alert policies and conditions for a sample application, linking them to notification channels such as PagerDuty, and storing configurations in a version-controlled repository, facilitating collaboration and continuous improvement. The blog also emphasizes the importance of maintaining these configurations in a repository and suggests automating them through CI/CD pipelines to enhance efficiency and reliability.