Cracking Production Performance Issues with Open Source Observability

Avi Froelich shares a case study addressing a significant performance issue at Logz.io, where a minor code change dramatically improved HTTP endpoint efficiency. The investigation involved a deep dive into the pillars of observability—logs, metrics, and distributed tracing—using tools like Jaeger to identify and visualize bottlenecks, particularly during peak usage times. The team discovered that high latency was linked to a cache issue affecting user session tokens, which was resolved by adjusting the session token cache eviction policy. This fix, implemented with just two lines of code, resulted in a noticeable reduction in database traffic and latency, significantly improving performance and earning positive customer feedback. The study highlights the importance of correlating logs, metrics, and traces using unique identifiers to trace requests efficiently, emphasizing the benefits of using Logz.io’s integrated platform for observability.