September 2026 Summaries

No summary generated yet.

Datadog developed dd-trace-rs, an open-source, opinionated Rust distributed tracing library built on the OpenTelemetry SDK, to address fragmented traces, inconsistent context propagation, and uneven sampling across its growing fleet of performance-sensitive Rust services. Rather than relying solely on flexible but complex OpenTelemetry configurations or extending Rust’s tracing crate beyond its intended scope, the library applies Datadog defaults for metadata, propagation, and sampling while retaining compatibility with existing OpenTelemetry instrumentation. Datadog also contributed upstream improvements that synchronize tracing and OpenTelemetry contexts, including a stack-based context model that fixes broken parent-child relationships and improves performance. Its custom pipeline supports deferred sampling decisions and buffers spans into trace-level chunks, allowing complete traces to be retained when valuable while exporting aggregate metrics for dropped spans. Internal adoption reportedly reduced trace-ingestion volume by 20 times and tripled indexed spans per service, while Datadog plans to continue combining upstream OpenTelemetry contributions with targeted extensions for production-scale Rust observability.

Datadog Experiments introduces a CUPED adjustments visualization that explains why CUPED-adjusted experiment lift can differ from raw lift by breaking the change into individual covariate contributions. CUPED uses pre-exposure metrics and assignment properties to reduce variance and improve confidence intervals, but its adjustments can also alter the estimated treatment effect in ways that are difficult to interpret without additional context. Available from a CUPED-enabled metric’s analysis menu, the visualization presents raw and adjusted relative lift, estimated experiment run-time reduction, and a waterfall showing how each metric lookback or subject property raises or lowers the result. This helps users identify whether a difference is driven mainly by one covariate or several smaller ones, while clarifying that covariate adjustments account for observed imbalances rather than demonstrating causal effects.

Agentic AI systems require more than extensive telemetry to improve reliably; teams need to connect aggregate trace analysis, offline evaluations, and production experiments in a repeatable optimization loop. Trace signals such as latency bottlenecks, cost anomalies, quality scores, tool-selection accuracy, user feedback, and downstream outcomes can identify specific underperforming segments and support testable hypotheses. Candidate changes should first be evaluated on production-representative regression datasets and edge-case coverage datasets, using calibrated evaluators and segment-level analysis, before being tested through controlled, feature-flagged production experiments with predefined success metrics and guardrails. After rollout, continued monitoring and incorporation of new failure modes into evaluation datasets help detect quality drift and strengthen future tests. The post argues that integrating observability, datasets, evaluations, experimentation, and sensitive-data redaction within a shared platform, such as Datadog’s tools, can reduce manual work and help teams verify that agent changes produce durable improvements.