Deep Diving into Dapr Workflows and OpenTelemetry – Tracing the Invisible Parts of Asynchronous Communication

The blog post details a collaborative effort between the author and Mauricio Salatino to achieve end-to-end tracing of Dapr Workflows using OpenTelemetry, focusing on asynchronous workflows. They discovered challenges in tracing due to the unique architecture of Dapr Workflows, where orchestration runs inside the Dapr sidecar and communicates with applications via a single long-lived gRPC stream. This architecture caused trace context to be lost, breaking the trace chain and creating separate traces for workflow activities and their outbound calls. To solve this, they embedded trace context into the activity messages, repaired workflow span relationships within Dapr, and restored the trace context inside the Java workflow SDK, ensuring seamless trace continuity without requiring developers to write custom tracing code. The improvements allow for a single, continuous trace view of an entire workflow execution, demonstrated through a pizza-ordering application, and are based on standard W3C TraceContext and OpenTelemetry APIs, ensuring compatibility across languages and future-proofing the solution. The authors aim to integrate these enhancements into the Dapr community, further explore multi-language workflows, and extend the approach to other Dapr components.