September 2025 Summaries

Ingress controllers are crucial components in Kubernetes clusters, managing external traffic flow to services by handling TLS termination, routing, and policy enforcement. The post highlights the observability challenges and solutions for Kubernetes ingress controllers using OpenTelemetry, focusing on Emissary Ingress, formerly Ambassador API Gateway. Emissary, built on Envoy, extends to a comprehensive API gateway, emphasizing the importance of observability due to its complexity. The demo showcases how to implement tracing, metrics, and logs with OpenTelemetry, using the Collector to integrate these signals into a cohesive observability solution. Emissary's tracing setup lies between Contour and Traefik, requiring a single TracingService configuration, while its metrics are extensive, akin to Contour's, exposing Envoy and ambassador_* series. Logs are transformed to include trace context for correlation, which is handled by the OpenTelemetry Collector. This setup enables a unified view of ingress behavior, crucial for platform transparency and reliability. The post encourages hands-on experience with Emissary and OpenTelemetry through the dash0-examples repository, demonstrating how it transforms the ingress layer into a transparent, manageable part of the platform.

Traefik stands out among ingress controllers by fully embracing OpenTelemetry, providing native support for tracing, metrics, and experimental log exports, positioning it as one of the most OpenTelemetry-friendly options available. Unlike Ingress-NGINX and Contour, which require additional setup for comprehensive observability, Traefik supports OpenTelemetry traces and metrics natively and can export logs over OTLP, albeit still in an experimental phase. Designed for cloud-native environments, Traefik features dynamic service discovery, middleware chains, and CRD-driven configuration, with observability deeply integrated into its model, allowing for seamless trace, metric, and log correlation. The OpenTelemetry Collector plays a crucial role in enriching signals with Kubernetes metadata and exporting them to platforms like Dash0, facilitating a unified view across all observability data. Traefik's approach reduces the need for scraping or parsing and demonstrates a cleaner setup compared to its counterparts, making it easier for platform engineers to achieve comprehensive observability from the edge of the cluster to the services behind it.

Ingress controllers, which serve as the critical entry point for Kubernetes clusters, play a vital role in handling TLS termination and routing requests. This text explores the observability of Contour, an ingress controller that uses Envoy and is gaining popularity due to its CRD-driven architecture and separation of control and data planes. Like ingress-NGINX, Contour supports OpenTelemetry for distributed tracing but requires the OpenTelemetry Collector to integrate logs and metrics, scraping Prometheus metrics and parsing logs to enrich them with trace context. The Collector operates in two forms: a DaemonSet for node-local log collection and a Deployment for centralized trace and metric aggregation, ensuring all signals reach the observability platform Dash0. This approach highlights the current state of OpenTelemetry adoption in Kubernetes, demonstrating progress in tracing and the necessary role of the Collector in unifying logs and metrics for effective monitoring.

Ingress-nginx serves as a critical ingress controller in Kubernetes, acting as the entry point to a cluster by managing external requests, terminating TLS, and directing traffic to appropriate services, thus playing a vital role in maintaining system reliability and performance. Despite its importance, many teams view it as a "black box," lacking insights into its operations; however, OpenTelemetry can enhance observability by enabling distributed tracing, which helps correlate ingress spans with backend services, giving a detailed view of request flows. While ingress-nginx natively supports tracing via the OpenTelemetry Protocol, metrics and logs are exposed in Prometheus format and classic NGINX logs, respectively, requiring integration with the OpenTelemetry Collector for unified telemetry data. The Collector, deployed as both a DaemonSet and a Deployment, centralizes the collection of traces, metrics, and logs, allowing tools like Dash0 to provide a comprehensive view of ingress behavior, including performance metrics and error rates. Although the project is effectively in maintenance mode with no major new features planned, the current setup allows platform engineers to gain valuable insights into ingress-nginx's operations, helping ensure the cluster's edge is fully observable.

KEDA (Kubernetes Event-Driven Autoscaling) enhances Kubernetes by allowing workloads to scale based on events and external signals, beyond the traditional metrics like CPU and memory used by the Horizontal Pod Autoscaler (HPA). By integrating with OpenTelemetry and Dash0, users can gain insights into KEDA's internal metrics, improving observability and enabling more efficient workload autoscaling. KEDA operates by monitoring defined triggers and adjusting workload replicas accordingly, even scaling down to zero when no work is pending—a feature the HPA alone cannot achieve. The integration with OpenTelemetry offers visibility into KEDA's decision-making processes, allowing users to diagnose scaling issues effectively. Additionally, KEDA's capability to scale based on various event sources, such as message queues and scheduled events, makes it a versatile tool for managing dynamic workloads. With Dash0, these metrics become accessible and actionable, providing a comprehensive solution for event-driven scaling in Kubernetes environments.

Dapr (Distributed Application Runtime) enhances microservice development by abstracting common functionalities like service invocation, state management, pub/sub, and workflows into a sidecar, which necessitates robust observability for runtime operations. This abstraction can make system behavior less visible, requiring a unified approach to observability that integrates both application and runtime telemetry using OpenTelemetry (OTel). Dapr's integration with OpenTelemetry allows for comprehensive tracing and metrics collection, facilitating a vendor-neutral pipeline where signals from both application code and Dapr sidecars converge. Tracing in Dapr is essential for visualizing request flows, identifying latency sources, and monitoring retries and failures, while metrics offer insights into system performance and health over time. The OpenTelemetry Collector and Operator further enhance this setup by ingesting, enriching, and exporting telemetry data, while also enabling no-touch instrumentation for applications, bridging the gap between application and runtime telemetry. A demo application illustrates these observability concepts in action, highlighting how Dapr and OpenTelemetry work together in a distributed system to provide end-to-end visibility and performance monitoring.