Saving Three Months of Latency with a Single Trace: Coralogix and OpenTelemetry on Checkly

We monitored our Node service with OpenTelemetry and sent the data to a Coralogix dashboard, and with a little digging found an issue that was causing over a month of latency on all our tasks. A mysterious re-authentication request for security credentials was happening more than once, which we were able to identify by analyzing the traces in detail. We made a small change to save the results of our authentication call locally, obviating repeated requests, and this resulted in a measured drop in the time of housekeeping tasks. The 40ms average reduction in execution time may seem like little, but it translates into significant savings on infrastructure costs over time. Observability tools like OpenTelemetry provide visibility to understand and improve systems in ways we hadn't anticipated, reducing the unknown unknowns and enabling us to catch issues before they cause problems.