Distributed tracing with Envoy, Kuma, Grafana Agent, and Jaeger

Observability is crucial for cloud service providers like Koyeb, as it ensures the platform's service availability and performance. A year ago, Koyeb faced challenges with random 5xx errors due to insufficient cross-service visibility, prompting the implementation of end-to-end distributed tracing to improve error diagnostics and performance issue resolution. This involved selecting Jaeger as the distributed tracing backend for its scalability and Grafana Agent as the telemetry collector due to its capability to handle various telemetry data. The tracing system assigns unique traceIDs to requests and generates spans to track request paths through services, aiding in rapid identification and resolution of issues. To manage the high data volume and associated costs, probabilistic sampling retains only 1% of spans while ensuring complete traces are exported to Jaeger. Future plans include refining sampling methods, enhancing backend resilience, and extending tracing features to Koyeb users, aiming to improve overall platform efficiency and user experience.