September 2026 Summaries

Datadog redesigned its Python profiler to better analyze asyncio applications, where traditional flame graphs treat concurrent tasks as separate stacks and obscure the application paths responsible for work. The new async-aware model reconstructs task dependency relationships, creating “stacked stacks” that attribute activity such as database queries to the tasks awaiting them while accommodating tasks that outlive their creators. Because tracking every asyncio task substantially increased sampling costs, the team also optimized the profiler by removing exceptions from the sampling path, interning repeated strings, and replacing costly process_vm_readv memory reads with protected memcpy operations that safely recover from rare memory-access faults. These changes reduced profiler overhead by more than 60% in deployed services, enabled higher sampling fidelity, and helped identify production issues including an infinite decompression loop, wasteful logging string construction, and inefficient numerical code. The async-aware profiling and lower-overhead improvements are available in recent ddtrace versions.

Datadog has introduced supported Real User Monitoring and Product Analytics integrations for Shopify and Salesforce, helping teams gain visibility into revenue-critical journeys despite limited control over those platforms’ frontend runtimes. For Shopify, teams can add the Datadog Browser SDK to storefront themes and use a custom pixel for checkout events, preserving a shopper’s session from browsing through purchase confirmation and enabling funnel analysis, segmentation, error tracking, and correlation with feature flags. For Salesforce Experience Cloud, Datadog provides a Lightning Web Security-compatible bundle that is uploaded as a static resource and loaded through Lightning Web Components, capturing navigation-based views, supported clicks and custom actions, frustration signals, and certain console or custom errors. These integrations use each platform’s supported extension mechanisms to help organizations monitor user behavior, identify friction and errors, and avoid maintaining custom monitoring workarounds as Shopify and Salesforce evolve.

TypeSafe AI’s Jev is a decision-focused model that accepts structured state and typed questions to return probabilistic yes/no, categorical, and scored judgments without written explanations, positioning it as a lower-cost alternative to using text-generating models for evaluation tasks. The post demonstrates a shared five-question rubric for evaluating a fictional airline support agent’s replies against retrieved policy context, measuring grounding, whether the question was answered, handoff behavior, failure mode, and customer impact in one parallel request. It emphasizes retaining raw probabilities and confidence values rather than reducing results to binary labels, using uncertainty or near-tied categories to identify cases for human review, while keeping thresholds, arithmetic, dates, and composite business rules in application code. For Datadog Agent Observability, Jev can score live spans asynchronously through external evaluations joined by a domain tag and can also power offline experiments, where caching enables multiple evaluator metrics to reuse one Jev request per dataset row. The approach recommends pinning and recording model versions, filtering state to relevant evidence, calibrating thresholds against labeled data, and validating agreement and consistency with human reviewers before operational use.

Datadog’s Tap to Parse is an AI-assisted feature that converts values embedded in unstructured log messages into searchable structured attributes without requiring users to manually write regular expressions or Grok parsing rules. It analyzes sample logs, identifies stable patterns and variable fields, and generates a parser for review before use. In Log Explorer, it creates session-only calculated fields to help investigators quickly filter, group, and analyze events, such as identifying a payment provider responsible for checkout failures. In Log Pipelines, users can save generated Grok parsers to enrich future ingested logs with reusable attributes for searches, dashboards, and monitors. In Observability Pipelines, the feature can structure logs before they are sent to multiple destinations, helping teams maintain consistent fields across Datadog, archives, data lakes, and SIEMs. Tap to Parse is generally available in Log Explorer and Log Pipelines, while its Observability Pipelines support is available upon request.

Datadog describes fine-tuning the smaller Qwen3.5-9B model to perform agentic change attribution for production alerts, aiming to identify deployments, feature flags, or configuration changes that may have contributed to incidents at lower cost than frontier models. Using a data flywheel, the company generated successful investigation traces from GLM-5.3, retained traces matching proxy labels derived from Bits Investigation conclusions, and trained the student model with LoRA; the dataset grew from 100 to 186 internal-incident examples over two cycles. In an evaluation of 326 later production incidents, the fine-tuned model achieved a Recall@5 of 0.55 on internal incidents, or 87% of GLM-5.3’s 0.63 score, while estimated to cost $0.003 per investigation versus $0.06 for the teacher, and it also outperformed both the base model and a heuristics-based ranker. Fine-tuning shifted the model from broad searches toward more focused collection of log, span, and metric evidence, reducing token use relative to the untuned model, though it remained more resource-intensive than fixed rules. The reported metric measures agreement with changes referenced by Bits Investigation rather than independently established causality, and Datadog plans to explore reinforcement learning using production-derived feedback signals while acknowledging that these labels can be imperfect.

Datadog’s RUM Remote Configuration enables teams to update supported browser, iOS, and Android SDK data-collection settings from the Datadog UI without changing or redeploying application code. It addresses delays caused by release cycles, app store reviews, and gradual user adoption, allowing teams to respond more quickly to incidents or performance regressions by adjusting options such as profiling and sampling rates, Session Replay, privacy controls, event tracking, tracing, and application attributes. New RUM applications include a remote configuration ID and have the feature enabled by default, while existing applications require adding the ID and enabling the setting in SDK initialization. Published configurations are retrieved when eligible SDKs initialize, and existing SDK values remain in effect until changes are reviewed, saved, and published. Platform-specific settings are managed separately, while distinct viewing, editing, and publishing permissions and modification records support controlled access and auditing.

Datadog Code Execution, now generally available through the Datadog MCP Server, enables AI agents to conduct multistep observability investigations by generating and running JavaScript in a sandboxed environment. Rather than sending raw results from each separate query through the model, agents can query Datadog APIs for logs, metrics, traces, and other data sources in parallel, apply control flow, correlate and aggregate results, and return only relevant evidence. In tests across 25 observability tasks and four AI models, Code Execution improved average answer correctness from 74.1% to 89.6%, while reducing input-token use by 73.2% and tool calls by 39.6% compared with Datadog’s Core toolset. The system keeps credentials outside the sandbox, with a trusted MCP service making requests under the caller’s existing permissions and policies. Users can enable the code-exec toolset after connecting the Datadog MCP Server to an AI client to investigate questions spanning multiple observability signals.

Datadog researchers describe using weak labeling to estimate agent user satisfaction from existing telemetry when users rarely provide explicit thumbs-up or thumbs-down feedback. Their public session classification skill combines Agent Observability traces, Real User Monitoring data, and Audit Trail records to infer whether a chat interaction was positive or negative, using conversation sentiment, user behavior, and resulting changes to Datadog artifacts. Tested against hundreds of manually labeled Bits Chat sessions, the classifier achieved 78% accuracy with traces alone, 80% after adding RUM, and 82% with Audit Trail data, indicating that additional telemetry improves its usefulness. Although it is not intended to replace direct user feedback, the approach can help teams prioritize sessions for review, generate approximate labels for evaluation datasets, and monitor agent performance when explicit feedback is limited.

Datadog Product Analytics’ Journey Paths feature expands funnel analysis by showing the ordered pages and actions users take between defined steps, helping teams understand why users convert or drop off rather than only measuring conversion percentages. It ranks common paths for successful and unsuccessful users, revealing inefficient but completed journeys, repeated backtracking, dead ends, and detours such as support-page visits that may pull users away from checkout. Teams can pair Journey Paths with Conversion Analysis to identify user attributes or segments associated with outcomes, Session Replay to observe representative experiences, and Real User Monitoring to check for technical factors including errors, latency, rage clicks, and dead clicks. Because it uses Datadog’s existing Browser and Mobile SDK instrumentation and can be saved to dashboards, the feature enables ongoing investigation of recurring behavioral patterns.

Datadog IaC Security custom rules allow security and platform teams to supplement default misconfiguration checks with organization-specific infrastructure policies, such as required tags, approved instance types, naming standards, network boundaries, and compliance requirements. Written in Rego and supported across Ansible, AWS CloudFormation, Dockerfiles, Kubernetes, Terraform, and GitHub Actions, these policies run alongside default rules in repository scans and can be configured by repository, path, severity, or individual rule. Users with the required permissions can create rules from scratch, clone existing policies, or use an AI-assisted natural-language rule creator that generates Rego code and sample configurations. Rules can be tested in the editor, saved as drafts, published when ready, and tracked through version history with comparison and rollback capabilities. Once published, violations appear in existing Datadog workflows, including pull request comments, IDE extensions, the findings explorer, PR Gates, and automated remediation pipelines, enabling teams to enforce internal standards before deployment without creating separate review processes.

Datadog Observability Pipelines introduces Microsoft Sentinel Packs, preconfigured integrations that normalize vendor-specific firewall, VPN, and network detection logs into Microsoft Sentinel table schemas before ingestion. Available initially for Palo Alto Networks, Fortinet, Cisco ASA, Cisco Meraki, and ExtraHop, the Packs map supported events into schemas such as CommonSecurityLog and Syslog, deriving fields including severity and device actions to support Sentinel analytics rules, workbooks, and investigations without requiring teams to maintain custom parsers. By applying consistent mappings across sources, security teams can investigate activity such as suspicious outbound traffic using shared fields like SourceIP, DestinationIP, and DeviceAction rather than stitching together disparate raw formats. The pipeline-based approach also enables organizations to limit Sentinel’s per-GB ingest to high-value events, retain complete raw logs in lower-cost storage, and filter repetitive or low-risk data, including ExtraHop detections below a configured risk threshold.

Datadog’s Bits Chat integrates its natural-language Bits AI interface into Slack to help incident responders investigate, coordinate, remediate, and document production issues without switching between communication, telemetry, code, and incident-management tools. When an alert triggers an incident, Datadog can create a dedicated Slack channel where users invoke @Datadog to launch a Bits Investigation, which analyzes relevant telemetry, runbooks, and prior incidents, posts progress updates, and returns likely root causes with supporting evidence and recommended actions. Teams can question the findings in the same channel, assess affected endpoints, regions, and downstream services, and use Bits Remediation to add responders, run workflows, send updates, page engineers, or create follow-up records. For code-related problems, Bits can pass context to Bits Code, which generates a proposed fix and pull request for review. After resolution, Bits can verify service recovery, close the incident, and create a postmortem notebook containing the incident summary, findings, remediation, and next steps, preserving context for future learning and response.

Datadog Cloud Cost Management integrates Cursor spending with cloud, SaaS, and other AI costs to help engineering and FinOps teams understand, monitor, and optimize AI coding expenses. Its AI Costs tab, Cloud Cost Explorer, and prebuilt dashboard provide breakdowns by user, model, billing group, Max Mode, headless mode, and other dimensions, while also showing spending trends, budget consumption, model shifts, and efficiency metrics such as accepted code suggestions and cost per accepted line. Automated anomaly detection identifies unexpected changes from historical spending patterns, and Bits Chat can assist with root-cause analysis by pinpointing contributing users, models, or features. Teams can also use savings recommendations, threshold-based monitors, and budgets to prevent overruns, while unified dashboards and service-cost views place Cursor expenditures in the broader context of organizational technology spending.

Distributed AI training workloads challenge Kubernetes’s default pod-by-pod scheduling model because workers require both low-latency topology-aware placement and simultaneous startup to avoid communication bottlenecks and idle GPU capacity. The post describes how Kueue provides admission control and topology-aware scheduling by assigning jobs to labeled hardware domains such as hosts, racks, or blocks, while the Coscheduling plugin implements gang scheduling by reserving resources until a job’s required set of pods can bind together. It explains how cluster topology affects GPU communication through NVLink, PCIe, and network fabrics, with cross-rack placement often increasing latency and reducing training throughput, especially for large distributed jobs. Effective validation and troubleshooting require correlating Kueue admission and eviction metrics, PodGroup gang states, workload topology assignments, GPU and network bandwidth, and framework-level throughput metrics. Datadog’s Kubernetes and GPU monitoring tools are presented as a way to centralize these signals, detect queueing, placement, hardware, or assembly issues, and retain diagnostic data for frameworks such as Ray, PyTorch, Kubeflow, and Horovod. Kubernetes is also adding native gang scheduling, workload-aware preemption, and topology-aware scheduling capabilities in versions 1.35 through 1.37, but these remain early-stage or disabled by default, leaving Kueue and Coscheduling as the described production approach.

As organizations scale autonomous agents across shared platforms, operational risks can emerge even when individual agents follow authentication, approval, and tool-use rules correctly, because concurrent workloads can strain model providers, queues, CI workers, sandboxes, APIs, and other dependencies. Datadog recommends mapping each agent’s full execution trajectory, measuring the actual limiting signals for every dependency—such as token throughput, concurrency, queue lag, utilization, quotas, retries, and latency—and linking capacity thresholds to customer and product impact. Platforms should attach consistent workload, owner, environment, service-class, and task identifiers to requests so they can prioritize critical traffic, isolate evaluation or background workloads, apply quotas and reserved capacity, and define overload behavior before contention occurs. Recovery mechanisms also require bounded retry budgets, exponential backoff, service-level concurrency controls, circuit breakers, and life-cycle-aware timeouts to prevent retries and deployment defaults from amplifying outages. Finally, stable non-human identities and propagated correlation IDs should connect agent tasks to tool calls, logs, traces, policy decisions, and downstream audit events, enabling enforcement, investigation, and accountability across system boundaries.

Datadog reports that Gartner named it the “Company to Beat” for observability platforms in August 2026 and positioned it as a Leader in the 2026 Magic Quadrant for Observability Platforms for the sixth consecutive year, while noting that Gartner’s research represents analyst opinion rather than factual endorsement. The company describes its strategy as providing a unified observability and security platform with more than 40 products for technology, security, and business teams managing increasingly complex AI-native systems. Key offerings include Bits Investigation for automated alert investigation and response, Agent Observability for monitoring AI agents and LLM applications, an MCP Server that enables AI agents to query telemetry data, and tools for OpenTelemetry collection, log management, data control, and cost predictability. Datadog also cites more than 100 capability launches at DASH 2026, the acquisition of Adaptive ML, and a partnership with Sakana AI. It identifies rising observability costs, data residency and sovereignty demands, and actionable governance for autonomous agents as major ongoing challenges, emphasizing continued investment and customer feedback as central to its approach.

NetFlow records can be duplicated when the same traffic conversation is exported by multiple interfaces or network devices, inflating volume measurements and distorting capacity planning and top-talker analysis. Ingress and egress duplication can be avoided by configuring monitoring only on egress interfaces, while multi-exporter duplication requires selecting authoritative devices or filtering records according to device roles and network topology. For internet-bound traffic, central border routers are often suitable reporting sources, whereas internal east-west traffic may require monitoring spine switches or load balancers, and cross-site traffic may need additional site-based filters to prevent counting flows at both tunnel endpoints. Datadog NetFlow Monitoring supports this approach by allowing users to tag network devices, such as with border, core, or site roles, and filter dashboards by those tags to obtain more accurate traffic views without necessarily disabling monitoring across the network.

Datadog has introduced an Experiments plugin for ChatGPT Work that lets teams query running or completed A/B tests in natural language without leaving their existing workflow. The plugin provides continuously updated, statistically grounded results including lift, confidence intervals, sample sizes, segment breakdowns, tested metrics, and guardrails, while linking responses back to Datadog for deeper analysis and related RUM or Session Replay data. It can combine causal experiment findings with warehouse, CRM, and other business context to support decisions such as prioritizing customer cohorts or preparing executive summaries. The integration also surfaces test-health issues such as sample ratio mismatches, traffic imbalances, instrumentation errors, and performance regressions before they require a rerun. Access is governed through ChatGPT Work’s role-based controls and existing Datadog permissions, while Datadog Audit Trail records plugin queries and cited experiments provide traceability into the underlying methodology and evidence.

Datadog rebuilt Eppo’s experimentation product as Datadog Experiments, integrating A/B test analysis with warehouse data, Real User Monitoring, Product Analytics, and observability signals to help teams make release decisions more quickly. The platform extends CUPED variance reduction to audience segments and percentile metrics such as p90 page-load time, enabling more statistically efficient results and earlier detection of performance regressions. It also redefines global lift as the transparent product of local lift and coverage, including for windowed metrics, making estimated company-wide impact easier to interpret and verify. For warehouse-native metrics, a Copy SQL feature provides the underlying query used for reported results, while diagnostic checks flag issues such as traffic imbalances or missing data. Near-real-time behavioral and performance metrics from RUM, Product Analytics, APM spans, and Session Replay provide early signals before slower warehouse metrics arrive, complementing longer-term business measures such as revenue and retention.

Datadog Kafka Console is presented as a centralized tool for monitoring, troubleshooting, and optimizing Kafka environments by collecting health and performance data directly from clusters. It helps teams investigate issues such as consumer lag, offline or under-replicated partitions, configuration changes, schema changes, and message deserialization failures through views of clusters, topics, brokers, partitions, consumer groups, and messages. The platform can alert users when lag approaches retention limits, correlate operational problems with configuration or schema updates, identify affected producer and consumer services, and connect teams with ownership information. Authorized users can inspect topic messages and schema versions to distinguish infrastructure failures from data-related problems, while configuration recommendations support performance tuning. Kafka Console can be used independently or alongside Datadog Data Streams Monitoring to combine Kafka infrastructure visibility with application-level flow and latency context, and it integrates with Bits AI and the Datadog MCP Server to assist investigations and recommend potential fixes.

AI Golden Paths are standardized workflows and configurations intended to make AI-assisted development more reliable, efficient, secure, and cost-conscious across teams. Datadog’s Frontend Augmented by AI guild develops these paths by identifying undesirable agent behaviors, selecting the least expansive effective controls—such as interface configurations, automated tests and hooks, skills, or documentation—and defining ownership and implementation scope. The approach emphasizes data-driven evaluation, using controlled experiments for broad or complex controls and simpler observation for low-risk changes, while dashboards and monitoring help track cost, duration, token use, output quality, and consistency over time. In one experiment, removing bloated root-level AGENTS.md documentation improved average run duration by 13%, reduced input tokens by 16%, and lowered spending by 10%, although output consistency fell by about 7%; the team plans to recover this through more targeted controls. The process aims to prevent unnecessary context bloat while continually refining agent guidance as models, repositories, and development practices evolve.

Datadog Product Analytics Launches is designed to help product teams plan, instrument, monitor, and evaluate feature releases through a connected workflow centered on product briefs and feature flags. It uses Bits AI to help generate measurement questions, tracking plans, and KPI dashboards, identifies missing events and properties, and can create pull requests for required instrumentation before release. During rollout, it combines Feature Flags, Experiments, RUM, Session Replay, behavioral analytics, and support-ticket context to compare treatment and control groups, detect experiment issues such as sample-ratio mismatches, and investigate user friction across segments including devices, screen sizes, and locations. A launch command center consolidates rollout exposure, business KPIs, experiment results, technical health, funnels, user-experience findings, and support signals around a specific feature change. The measurement plans and dashboards can remain in use after full release, while Journey Monitoring can extend coverage to critical flows such as checkout or login by tracking conversion, performance, errors, uptime, and session behavior over time.

Datadog Workload Protection adds automated and manual runtime threat remediation to reduce the time between detecting malicious activity and stopping it. Automated Agent rules can terminate processes that match clear, unambiguous threats such as cryptomining, while manual actions allow security teams to investigate signals and then kill a process or container, isolate a workload, or combine both actions from the same interface. The system uses kernel-level visibility, namespace-independent process IDs, cgroup-based container mapping, and eBPF traffic filters to identify and target affected resources precisely without disrupting healthy workloads. Elevated permissions, audit logs, timestamps, and recorded signals provide authorization and accountability for each action, enabling organizations to contain threats while maintaining operational control.

Datadog Workflow Automation can now be created, executed, debugged, and updated through Bits Chat, AI coding agents such as Claude Code, Cursor, and Codex via the Datadog MCP Server, and Slack conversations. The feature is designed to convert recurring operational investigation and remediation tasks into context-aware workflows without requiring teams to switch tools or manually recreate information. Examples include workflows that investigate API error spikes, correlate failures with deployments, perform safe rollbacks, verify service recovery, open pull requests for code fixes, or escalate issues with attached findings when causes are unclear. AI agents can use existing Datadog monitors, service metadata, integrations, traces, and workflows to generate and refine automations conversationally, while Bits Chat can diagnose failed workflow runs and suggest fixes. Custom agents built with Bits Agent Builder can also invoke workflows during incidents, enabling consistent human- and agent-initiated responses across Datadog, development environments, and Slack.

Datadog’s 2026 State of AI Engineering report found that system prompts accounted for 69% of input tokens in customer traces in March 2026, highlighting the cost and latency impact of repeated agent instructions, tool definitions, guardrails, and reference material. Prompt caching reduces this overhead by storing the intermediate model state for an identical prompt prefix and reusing it on later requests, though provider implementations differ: Anthropic offers automatic and explicit cache breakpoints, while OpenAI automatically caches eligible prefixes and supports explicit breakpoints on newer models. Effective caching depends on keeping shared prompt prefixes stable, placing dynamic user messages and tool results after cache boundaries, and potentially caching prior conversation history separately in multi-turn workflows. Because cache writes can cost more than uncached inputs and minor changes can invalidate cached content, teams should track cache-hit rates, token usage, costs, and latency using API metrics and request traces. Common causes of cache misses include reordered tools, context compaction, changing summaries or memory, and timestamps or request-specific metadata inserted into otherwise static prompt sections.

Datadog developed dd-trace-rs, an open-source, opinionated Rust distributed tracing library built on the OpenTelemetry SDK, to address fragmented traces, inconsistent context propagation, and uneven sampling across its growing fleet of performance-sensitive Rust services. Rather than relying solely on flexible but complex OpenTelemetry configurations or extending Rust’s tracing crate beyond its intended scope, the library applies Datadog defaults for metadata, propagation, and sampling while retaining compatibility with existing OpenTelemetry instrumentation. Datadog also contributed upstream improvements that synchronize tracing and OpenTelemetry contexts, including a stack-based context model that fixes broken parent-child relationships and improves performance. Its custom pipeline supports deferred sampling decisions and buffers spans into trace-level chunks, allowing complete traces to be retained when valuable while exporting aggregate metrics for dropped spans. Internal adoption reportedly reduced trace-ingestion volume by 20 times and tripled indexed spans per service, while Datadog plans to continue combining upstream OpenTelemetry contributions with targeted extensions for production-scale Rust observability.

Datadog Experiments introduces a CUPED adjustments visualization that explains why CUPED-adjusted experiment lift can differ from raw lift by breaking the change into individual covariate contributions. CUPED uses pre-exposure metrics and assignment properties to reduce variance and improve confidence intervals, but its adjustments can also alter the estimated treatment effect in ways that are difficult to interpret without additional context. Available from a CUPED-enabled metric’s analysis menu, the visualization presents raw and adjusted relative lift, estimated experiment run-time reduction, and a waterfall showing how each metric lookback or subject property raises or lowers the result. This helps users identify whether a difference is driven mainly by one covariate or several smaller ones, while clarifying that covariate adjustments account for observed imbalances rather than demonstrating causal effects.

Agentic AI systems require more than extensive telemetry to improve reliably; teams need to connect aggregate trace analysis, offline evaluations, and production experiments in a repeatable optimization loop. Trace signals such as latency bottlenecks, cost anomalies, quality scores, tool-selection accuracy, user feedback, and downstream outcomes can identify specific underperforming segments and support testable hypotheses. Candidate changes should first be evaluated on production-representative regression datasets and edge-case coverage datasets, using calibrated evaluators and segment-level analysis, before being tested through controlled, feature-flagged production experiments with predefined success metrics and guardrails. After rollout, continued monitoring and incorporation of new failure modes into evaluation datasets help detect quality drift and strengthen future tests. The post argues that integrating observability, datasets, evaluations, experimentation, and sensitive-data redaction within a shared platform, such as Datadog’s tools, can reduce manual work and help teams verify that agent changes produce durable improvements.