Observability at the Edge: OpenTelemetry in Ingress Controllers

OpenTelemetry has emerged as a crucial component in observability, with nearly half of organizations now utilizing it in production, as noted in the CNCF Annual Survey. It is evolving beyond a mere integration point, becoming the standard mechanism for producing and collecting correlated telemetry across modern platforms. The discussion highlights the importance of OpenTelemetry's predictable behavior at system boundaries, especially at ingress, where requests first enter the cluster. The role of ingress controllers and gateways is crucial as they are the first observable hop for external requests and significantly shape trace data. While tracing has stabilized with most ingress controllers participating cleanly in distributed traces, metrics are still predominantly exposed in Prometheus format, leading to a hybrid model where traces and logs may use OpenTelemetry Protocol (OTLP) while metrics are scraped. The OpenTelemetry Collector plays a significant role in bridging these differences, enriching telemetry, normalizing fields, and ensuring coherence. Semantic precision and resource identity vary across implementations, affecting operational complexity and system trust. As OpenTelemetry becomes foundational infrastructure, the focus shifts from mere support to how well it integrates into a shared telemetry model, with further explorations into specific ingress and gateway implementations planned.