How to survive a horror movie (if that movie is your prod environment)
Blog post from New Relic
The blog post uses horror movie analogies to depict challenges in managing production environments and offers solutions for common issues like orphaned spans, unidentified latency, and telemetry bloat. It describes how missing spans can lead to fragmented traces, with New Relic helping to identify and fix instrumentation gaps. Another scenario involves a service experiencing unexplained latency due to infrastructure issues, where New Relic's integration of infrastructure and application monitoring helps pinpoint the source. The final horror story addresses telemetry bloat, with New Relic's tools providing methods to manage excessive data collection by using data drop rules and tail-based sampling. Overall, the post emphasizes the importance of observability and offers practical strategies to overcome these technical challenges, while promoting New Relic as a comprehensive solution.