June 2025 Summaries

The text discusses the implementation of FinOps at Datadog to enhance cloud cost efficiency, emphasizing collaboration between FinOps and engineering teams. Initially, simple steps like eliminating unused resources offer quick savings, but managing costs becomes complicated with organizational growth and multi-cloud infrastructures. Datadog's approach involves building trustful partnerships based on reliable cost data, using custom dashboards to provide engineers with actionable insights for cost optimization. The company employs tools like Datadog Cloud Cost Management (CCM) for granular visibility and cost allocation, ensuring resources are appropriately tagged for precise cost attribution. The focus on unit economics aids in understanding the true cost of delivering business value, enabling teams to distinguish between beneficial growth and inefficiencies. Datadog's FinOps practice is supported by recommendations, analytics, and alerts that help prioritize and manage cost-saving efforts within Jira workflows. As part of an evolving FinOps journey, Datadog aims to integrate cost awareness into daily operations and contribute to the ongoing development of CCM by using real-world insights to refine the product.

As organizations expand their infrastructure, the challenge of managing monitoring systems becomes prominent, as an overload of alerts can lead to inefficiency and critical oversights. Datadog addresses this issue with its newly available monitor notification rules, which replace manual configurations with dynamic, tag-driven logic to streamline alert delivery. These rules help ensure that alerts are meaningful and directed to the appropriate individuals, reducing the risk of alert fatigue and missed incidents. By utilizing predefined conditions based on tags such as team, service, and environment, Datadog allows for centralized control of notifications, minimizing manual upkeep and complex logic. This approach enhances operational excellence by prioritizing business-critical alerts, thus supporting efficient incident management and safeguarding the customer experience. Adopting best practices like consistent tagging and clear ownership can further optimize the use of monitor notification rules, ensuring precise and effective alert routing.

Addie Beach emphasizes the crucial role of Service Level Objectives (SLOs) in enhancing user satisfaction, directing engineering priorities, and evaluating performance. Accurate and consistent reporting of Service Level Indicators (SLIs) is essential, with data ideally derived from production environments to ensure reliable assessments. However, employing SLOs in synthetic tests can preemptively identify potential issues that might affect users and business operations. Synthetic SLOs are instrumental in verifying the correctness of SLIs and anticipating performance challenges. The text outlines how to establish and monitor SLO-based synthetic tests, highlighting that SLOs set organization-wide reliability standards, particularly regarding application uptime. Datadog offers tools such as Time Slice SLOs and monitor-based SLOs to facilitate uptime monitoring and customization of alerts. By integrating with third-party tools and visualizing SLO metrics alongside other performance data, Datadog enables comprehensive monitoring and rapid issue identification, minimizing disruptions in production environments. The text underscores the importance of maintaining SLOs through synthetic testing to ensure application reliability and offers resources for monitoring SLOs with Datadog Synthetic Monitoring.

Datadog has introduced the OCSF Common Data Model within its Cloud SIEM to address the challenges posed by the fragmentation of log formats from diverse data sources in security environments. This model aligns security data with the Open Cybersecurity Schema Framework (OCSF), providing a unified schema that normalizes logs across platforms such as AWS CloudTrail, Okta, and GitHub. By automating the normalization process and enriching logs with standardized attributes, the OCSF Model enables security teams to apply prebuilt detection rules across varied data sources without manual parsing or field mapping. This standardization facilitates advanced analytics, expedites investigations, and allows for the detection of cross-platform anomalies, such as brute force login attempts, by correlating data from different services. The model enhances interoperability between tools and reduces the risk of missed signals due to inconsistent data, ultimately strengthening the security foundation by allowing for scalable and reusable detection rules. Datadog's commitment to transparency and adoption of the OCSF standard is further demonstrated by sharing its attribute remappings with the broader community through public GitHub repositories.

AWS Step Functions' Distributed Map state enables the coordination of massively parallel workloads, allowing up to 10,000 concurrent workflows to process millions of items efficiently, which is ideal for large-scale data processing tasks like image transformation and log ingestion. However, managing such scale introduces challenges in tracking execution progress and performance, requiring tools like Datadog for visualization, monitoring, and troubleshooting. Datadog provides a comprehensive view of Distributed Map executions, offering insights into parent-child workflow relationships, execution paths, and detailed metrics, thereby reducing mean time to resolution and aiding in identifying patterns and anomalies across parallel tasks. By integrating Datadog with AWS Step Functions, users gain enhanced visibility and control over complex workflows, making it easier to operate and optimize large-scale serverless applications.

The Datadog CoScreen team developed an open-source noise suppression library named dtln-rs to enhance real-time collaboration tools for remote work, addressing the challenge of background noise in virtual meetings. This library was created after discovering existing solutions were either too resource-intensive or not easily integrable with WebRTC, the widely used real-time communication software. Dtln-rs is based on the Dual-Signal Transformation LSTM Network (DTLN) and is optimized for speed and portability, allowing it to run efficiently on client devices without server dependencies. Utilizing AI-powered noise reduction, dtln-rs can filter out background noise in real time by analyzing audio data to distinguish between desired signals and noise. This solution, which supports native clients and WebAssembly, aims to provide developers with an accessible tool for embedding high-quality noise suppression in their WebRTC applications. The project underscores the potential of AI in enhancing real-time audio experiences and emphasizes the importance of making advanced technologies available to the open-source community.

The text outlines the development of Datadog's Log Forwarding feature, drawing parallels to package delivery services like UPS and FedEx. It highlights the challenges faced in efficiently and reliably forwarding logs to multiple external endpoints, comparable to delivering packages to various addresses. The system built by Datadog is designed to handle issues such as endpoint unavailability and data loss, using cloud object storage and Kafka topics to manage log traffic and ensure reliable delivery. This approach allows for low-latency and high-throughput log forwarding, isolating failures to specific endpoints without affecting overall system performance. The design also prioritizes real-time log delivery while gradually addressing backlogs, akin to prioritizing fresh deliveries in a package system. The successful implementation of this system has proven its scalability and reliability, prompting Datadog to adopt similar strategies for other services like Log Archives.

Datadog faced significant engineering challenges in efficiently managing and propagating user-specific configuration data across thousands of workload containers handling millions of logs per second. The core issue was ensuring low-latency and reliable configuration updates in a complex, multi-tenant distributed environment. Initial solutions relied on direct database connections and Kafka for cache invalidation, but these proved inadequate as Datadog scaled, leading to database overloads and unreliable notifications. A new architecture was developed using a gRPC context service and a context-publisher system, which allowed each workload container to maintain a local, frequently updated replica of the necessary configuration data, drastically reducing database load and improving reliability. This approach, which leverages the small size and slow update rate of context data, has been successfully generalized to handle various use cases, enhancing Datadog's resilience to database outages while maintaining swift configuration updates. The project underscores the importance of understanding data properties to devise scalable solutions, and Datadog continues to expand this system to support more types of context data, inviting engineers to join their team in tackling such challenges.

Datadog has been addressing the challenges of operating a large, shared relational database by transitioning towards independently owned database instances. This shift is driven by the limitations encountered as the organization grows, where the database's monolithic nature creates operational risks and hinders agile development. The process of splitting the database involves defining functional boundaries, minimizing migration risks, and implementing specialized tooling to ensure a safe transition. Datadog's strategy includes three phases: logical separation, access reduction, and physical separation, supported by platform investments like the Rapid framework and OrgStore managed platform. These initiatives aim to mitigate the operational burden, enhance scalability, and enable more autonomous development, while maintaining collaboration and communication with product teams to ensure the migration's success. Despite the complexity and time required, Datadog's approach highlights the potential for deeply entrenched systems to evolve through careful planning, automation, and partnership across teams.

The integration of OpenAI's Codex CLI with Datadog's MCP server offers an innovative solution for DevOps teams, enabling AI to assist in real-time incident troubleshooting by accessing live observability data. This collaboration facilitates the Codex agent to perform tasks such as retrieving logs, metrics, and updating incidents through natural language queries, thereby enhancing the efficiency and speed of incident response. The Codex CLI, now written in Rust for better performance and security, seamlessly integrates with Datadog's tools, providing on-call engineers with critical insights and automation capabilities directly within the terminal. Enhancements such as structured UIs and visual outputs in the terminal aim to make AI-generated insights more actionable, supporting engineers during high-pressure situations. This integration not only aids in generating and modifying code but also allows the agent to become context-aware, bridging the gap between code and runtime behavior, thus accelerating infrastructure automation and incident management with AI-driven intelligence.

The text discusses how security teams can effectively manage detection rules using Datadog Cloud SIEM by implementing "detection as code," which allows them to automate and streamline workflows. This approach, supported by infrastructure-as-code tools like Terraform, helps automate rule management, minimize errors, and integrate with CI/CD pipelines, enhancing the consistency and speed of deployments. Key practices include validating rule formats with linting, testing detection logic using mock and historical data, and converting detection rules to Terraform for standardized deployments. Datadog provides options for deploying rules via API or Terraform, enabling fast, repeatable deployments while tracking changes with rule versioning to maintain visibility over modifications. By enabling large teams to manage detection rules efficiently, Datadog's approach ensures that security measures can keep up with evolving threats in dynamic environments, thus supporting security teams in reducing errors and maintaining clarity in detection strategies.

The recent DASH keynote highlighted significant advancements in Datadog's platform, focusing on AI-driven insights to enhance observability and streamline workflows across various domains, such as infrastructure monitoring, security, and AI workload management. The newly introduced features aim to break down silos, simplify complex infrastructures, and expedite the resolution of performance and security issues, offering tools like LLM Observability for visualizing AI agent interactions, GPU Monitoring for optimizing AI infrastructure, and Bits AI for natural-language-driven application development. Further, Datadog's platform now supports comprehensive data observability, efficient log management, and advanced analytics through features like Archive Search, Flex Frozen, and AI-driven data analysis tools. These enhancements enable teams to accelerate software delivery, unify incident response, and improve operational efficiency, all while maintaining a centralized, seamless experience across Datadog's ecosystem.

Datadog LLM Observability's new Experiments feature provides a comprehensive framework for testing and evaluating LLM applications during pre-production. This tool allows users to optimize various parameters by running experiments that help in understanding application responses to complex inputs and managing experiment datasets effectively. With the Experiments SDK, users can build, manage, and analyze experiments while ensuring high-quality test data. The platform supports dataset version control, enabling teams to collaborate efficiently and troubleshoot issues by tracking changes. Users can monitor experiments through detailed traces and evaluations, facilitating insights into application performance and opportunities for optimization. Additionally, Datadog's Experiments allows for comparing different models to find the best fit for specific tasks, offering granular visibility into LLM outputs for deeper analysis. The tool emphasizes refining model parameters and evaluating application behavior under diverse conditions, currently available as a limited preview for those interested in enhancing their LLM testing processes.

Engineering leaders often face challenges in effectively monitoring their teams' performance and ensuring the reliability of their services due to scattered metrics and inconsistent reporting. Datadog addresses these issues by offering tailored insights through its Internal Developer Portal (IDP), which includes Engineering Reports and a Developer Overview page. These tools provide engineering teams with a unified view of service level objectives (SLOs), incident trends, and compliance with engineering standards, enabling them to track reliability, adherence to best practices, and delivery performance using DORA metrics. The centralized platform helps minimize context switching for developers by consolidating crucial information like Jira tickets and GitHub pull requests into a single dashboard, thereby improving focus and task completion. By offering comprehensive visibility and actionable insights, Datadog's IDP enhances engineering effectiveness, allowing leaders to make informed decisions and drive improvements in service health and software delivery.

As organizations expand their Kubernetes environments, managing the complexities of dynamic workloads and interdependent services becomes increasingly challenging, particularly in detecting and resolving incidents swiftly due to the deluge of telemetry data and alerts. Many application developers lack the necessary Kubernetes expertise, turning platform teams into bottlenecks for issue resolution. Datadog Kubernetes Active Remediation, currently in Preview, aims to address these challenges by offering clear contextual guidance and suggested actions to preempt business-impacting incidents, with its latest enhancement featuring AI-powered explanations for deeper insights into the root causes of issues. By streamlining the troubleshooting process, this tool enables faster incident response, reduces the mean time to resolution (MTTR), and enhances the self-sufficiency of developers, ultimately allowing platform teams to focus on more critical tasks.

The text discusses the integration of Datadog's Data Jobs Monitoring (DJM) with Databricks' serverless compute to improve the management of data processing workloads. As teams migrate to serverless computing for better performance and cost efficiency, challenges such as job failures, suboptimal queries, and cost management persist. DJM enables users to monitor serverless jobs alongside traditional cluster-based jobs, offering tools to set alerts for job failures, monitor job execution, and evaluate cost trends. This integration aids in maintaining service level objectives and identifying areas for optimization by providing detailed visibility into job performance metrics. By leveraging DJM, users can proactively address issues like data freshness, optimize job queries, and adjust resources to prevent inefficiencies, ultimately enhancing the reliability and performance of serverless data processing tasks.

At the DASH event, Datadog unveiled a variety of new features aimed at enhancing service monitoring, optimization, and security for teams. Key updates include the introduction of RUM Recommendations to identify user friction points, Automated Analysis for real-time interpretation of profiling data, and one-click suggestions for improving test reliability in CI pipelines. Datadog has also introduced cost management tools, product analytics capabilities, and enhanced database monitoring to facilitate faster issue detection and resolution. Moreover, innovations such as Tag Analysis and Topological Correlation help teams diagnose and resolve performance issues more efficiently by providing data-driven insights and reducing alert fatigue. With these advancements, Datadog aims to offer a comprehensive and integrated platform that enhances operational efficiency, cost management, and user experience across cloud environments.

Datadog Archive Search, now available in preview, addresses the challenges organizations face with accessing historical log data stored in separate systems by allowing in-place searches across archived logs without the need for costly and time-consuming rehydration. This feature helps teams conduct faster audits and investigations by providing direct access to archived data stored in Flex Frozen or customer-owned cloud storage, such as Amazon S3, using the familiar Datadog query language and interface. It supports cross-telemetry analysis and enhances organizational visibility, enabling teams to correlate archived logs with other data sources, such as RUM events and metrics, without switching tools. The integration with Datadog Log Workspaces and Observability Pipelines ensures that log data remains fully searchable, audit-ready, and cost-effective by allowing transformation and routing of logs to different storage destinations. This approach reduces operational overhead and supports compliance and performance monitoring, providing an end-to-end workflow that enhances control and visibility over log data.

Datadog's Oracle Cloud Infrastructure (OCI) integration offers a streamlined method for monitoring OCI environments by collecting metrics and logs across the entire OCI stack within a unified platform, alongside third-party technologies. The newly introduced OCI QuickStart facilitates a rapid setup process, enabling users to gain full visibility into their OCI infrastructure and applications in just a few clicks, with support for over 30 OCI services. This integration automatically discovers and monitors new resources, ensuring comprehensive observability without manual updates, and is particularly designed for enterprise-scale environments. The setup allows for customizable monitoring parameters for regions, services, and compartments, while Datadog's automated scaling and maintenance eliminate operational overhead. Additionally, the QuickStart provides in-depth insights into OCI resources like compute instances and storage, offering detailed metadata through the Resource Catalog to aid in troubleshooting and reducing resolution time for issues. This fully managed approach allows for continuous monitoring adaptation as the OCI environment evolves, enhancing the overall efficiency and effectiveness of cloud infrastructure management.

The Bits AI Dev Agent, developed by Datadog, is an AI-powered tool designed to streamline the troubleshooting process for developers by autonomously monitoring observability data and generating production-ready pull requests with context, tests, and human-readable explanations. This agent reduces the time developers spend on investigating issues by automatically diagnosing problems, assessing their impact, and suggesting code fixes or explanations. It is integrated within the Datadog platform, enabling seamless triage of telemetry data and facilitating efficient problem resolution without the need for developers to switch tools. The agent supports various Datadog products, including Error Tracking, with plans to expand its capabilities and coverage. It aims to enhance productivity by allowing developers to focus more on building code rather than fixing errors.

The text discusses the importance of integrating security measures when using generative AI tools, particularly Amazon Bedrock, to prevent risks associated with misconfigurations. Datadog Cloud Security has developed a set of out-of-the-box detections to help organizations identify and remediate these misconfigurations, which can lead to unauthorized access, data leaks, and other vulnerabilities. The partnership between Datadog and AWS aims to secure cloud infrastructure while maintaining innovation, offering a unified security workflow that supports regulatory compliance and guided remediation. Datadog's infrastructure-aware severity scoring system aids in prioritizing risks, and the platform provides tools for detecting and resolving these issues, ensuring AI deployments remain secure and compliant with evolving standards. Additionally, the text highlights Datadog's extensive integration with AWS and its capability to monitor and enhance compliance posture for organizations adopting AI solutions.

Datadog's Flex Logs offers a cost-efficient solution for storing and analyzing large volumes of logs without driving up expenses, by decoupling storage and compute costs. This system allows teams to store logs for up to 15 months and choose compute sizes based on their querying needs. Flex Logs works in tandem with Standard Indexing, allowing teams to determine which logs are needed for real-time troubleshooting and which can be retained for ad hoc analysis. The Flex Logs Controls page provides new compute usage graphs that help monitor performance, identify slowdowns, and optimize usage. These insights allow teams to manage their compute capacity, identify slow queries, and adjust storage strategies according to log importance and volume. Best practices include improving query efficiency, optimizing dashboards, and scaling compute size to enhance performance and responsiveness. Flex Logs thus provides transparency and flexibility in log management, offering a 14-day free trial for new users to explore its capabilities.

The introduction of Bits AI SRE represents a significant advancement in incident response management by automating and streamlining the process through artificial intelligence. Unlike traditional methods that rely on manual intervention and siloed expertise, Bits operates autonomously to investigate alerts, coordinate incidents, and learn from past responses without requiring human prompting. It processes telemetry data, generates and tests multiple hypotheses, and often identifies the root cause of issues before engineers can review them, thus saving substantial time and resources. Bits also enhances collaboration by integrating seamlessly with existing systems like Slack, ServiceNow, and Jira, providing real-time incident summaries and updates that facilitate effective communication and decision-making. Additionally, it supports post-incident analysis by capturing follow-up tasks and generating initial drafts of postmortems, allowing teams to focus on strategic improvements. Overall, Bits AI SRE aims to empower engineering teams to concentrate more on building and shipping software by handling the complexities of incident management autonomously and efficiently.

Over the past year, Datadog has successfully integrated its Site Reliability Engineering (SRE) and Security teams into a unified group to enhance its operational and security posture. This integration aimed to improve visibility, align shared objectives, and encourage proactive risk management by combining the traditionally distinct roles of security, which focuses on preventing external threats, and SRE, which ensures system reliability. Led by Tanner Prynn and Angela Beteta, the initiative involved refining processes to scale permissions, enhance audit logging, and strengthen organizational governance, ultimately resulting in more efficient workflows and a robust internal trust framework. The iterative governance strategy employed by Datadog emphasized collaborative solutions, blameless incident remediation, and close partnerships with product teams to prototype features that align with both internal and customer goals. As a result, Datadog not only addressed existing reliability challenges but also established a foundation for continuous improvements, with other teams across the organization adopting and evolving this combined approach to redefine collaboration and enhance security and reliability.

Antoine Dussault discusses the challenges of identifying the root causes of service reliability issues, such as latency and errors, highlighting the inefficiencies of traditional trial-and-error methods. He introduces Datadog's Tag Analysis as a solution that automatically identifies statistically significant tags correlated with performance issues, allowing users to quickly pinpoint and address root causes without the need for prior grouping guesses. The tool simplifies the investigation process by providing data-driven insights into latency or error spikes, helping users focus their efforts on relevant attributes and avoiding unnecessary trial-and-error. By showcasing examples where Tag Analysis identifies issues related to specific service versions or customer segments, Dussault emphasizes how this feature can prevent broader impacts and improve user experience. The Tag Analysis feature is available in preview, with further resources offered for those interested in exploring Datadog's application performance monitoring capabilities.

As organizations expand their cloud operations, managing object storage costs and performance becomes increasingly challenging, especially with the vast amounts of data generated by AI and data-intensive workloads. Datadog Storage Monitoring offers a solution by providing visibility into cloud storage usage, helping to optimize costs, enhance performance, and improve operational efficiency across platforms like Amazon S3, Google Cloud Storage, and Azure Blob Storage. With new prefix-level metrics for Amazon S3, users can better manage lifecycle and retention policies, identify cost-saving opportunities by moving infrequently accessed data to lower-cost tiers, and resolve performance bottlenecks caused by high-traffic prefixes. This centralized approach allows organizations to enforce storage policies more effectively and make informed decisions based on detailed storage usage patterns. Storage Monitoring is currently in preview, with additional features for Google Cloud Storage and Azure Blob Storage on the horizon, and offers a free trial for new users.

The OpenAI Agents SDK is a Python framework designed to facilitate the creation of agentic applications capable of decision-making, tool utilization, input validation, and task delegation. Key features include orchestration primitives, built-in tracing, and guardrails that support developers in debugging complex AI workflows. The integration of Datadog LLM Observability with the OpenAI Agents SDK enhances monitoring capabilities by automatically capturing insights like agent decisions, tool usage, and model calls without requiring code changes. This integration allows for efficient troubleshooting through end-to-end tracing, evaluation of agent outputs for quality and safety, and tracking of operational metrics such as token usage and latency. Datadog provides a comprehensive view of agent performance, enabling users to monitor, optimize, and ensure the reliability of their applications. With minimal setup, developers can gain valuable insights into their AI agents' behavior, supported by Datadog's LLM Observability features, available to all customers along with a 14-day free trial for new users.

The traditional process of collecting and forwarding Azure platform logs to Datadog has been labor-intensive, requiring manual configuration of Azure Monitor diagnostic settings, Event Hubs, and Azure Functions apps, which increases setup time, maintenance burden, and costs. Datadog's new automated log forwarding feature simplifies this by automating the setup and configuration needed to send Azure platform logs, such as Microsoft Entra ID sign-in logs, audit logs, and activity logs, to Datadog. This automation reduces operational overhead, accelerates setup, improves reliability, and optimizes Azure costs by automatically scaling log forwarding services based on log volume. The system employs a control plane for orchestrating log forwarders across subscriptions and individual log forwarders within each subscription to collect and buffer logs before transmission. To set up automated forwarding, users configure an ARM template via Datadog's Azure integration, allowing for efficient log management and visibility in Datadog's Log Management system.

The new features in Datadog Incident Response streamline the management of incidents by allowing users to transition seamlessly between different stages of the incident life cycle. The platform's AI voice interface provides real-time summaries and notifications that help responders quickly assess situations and take action, reducing the need for context switching and minimizing response times. Enhanced notifications allow for efficient handoff and collaboration among team members, while customizable status pages keep users informed about the status of service components without diverting focus from remediation efforts. These features work together to improve communication and response efficiency, allowing responders to dedicate more time to troubleshooting and faster problem resolution.

Infrastructure as Code (IaC) involves managing and provisioning cloud infrastructure using code rather than manual processes, offering advantages such as scalability, enforceability, testability, and version control. However, it also introduces potential security risks due to misconfigurations, such as overly permissive security settings or public-write access on resources. Open source tools like Open Policy Agent (OPA), Checkov, and Keeping Infrastructure as Code Secure (KICS) help identify these risks by scanning IaC files for vulnerabilities and compliance issues. Datadog enhances these tools by providing integrated observability and security features, allowing for proactive issue resolution with real-world context. Through continuous scanning and actionable insights, Datadog aids teams in maintaining secure and compliant cloud environments, correlating static code analysis with runtime data to prioritize significant risks effectively.

The Datadog remote MCP Server is designed to enhance AI-powered tools by bridging the gap between AI agents and Datadog's comprehensive monitoring resources, addressing common challenges such as error handling and complex data parsing. By integrating with AI agents like OpenAI's Codex, the MCP Server allows for seamless interaction with Datadog's systems, enabling real-time troubleshooting, incident management, and performance monitoring directly from a terminal. This collaboration simplifies the process of retrieving logs, metrics, and incident details, allowing for a more efficient response to service errors, such as those experienced with Redis latency spikes, by dynamically selecting appropriate tools to address specific queries. The MCP Server is currently available in preview and aims to support a wide range of applications by extending AI capabilities within familiar environments.

As AI and large language model (LLM) workloads increasingly rely on GPU infrastructure, efficient GPU management becomes crucial for maintaining performance and cost-effectiveness. Inefficient GPU usage can lead to longer runtimes and increased expenses due to idle or underutilized resources. Datadog GPU Monitoring provides a comprehensive solution by offering real-time insights into GPU fleet health, utilization, and cost, allowing teams to quickly identify and resolve issues such as overprovisioning, resource contention, and hardware failures. The platform enables users to monitor cloud and on-premises deployments, track spending, and optimize GPU allocation to enhance operational efficiency. By addressing both workload inefficiencies and underlying hardware constraints, Datadog ensures that AI workloads run smoothly without unnecessary financial strain, thus supporting scalable and high-performing AI infrastructure.

As data systems become increasingly complex and integral to business operations, issues related to data quality and reliability pose significant challenges, often requiring manual checks and ad hoc solutions that are inefficient and reactive. Current data observability tools tend to highlight only surface-level problems, lacking the context needed to trace these issues back to their origins within data pipelines. Datadog Data Observability aims to address these challenges by offering comprehensive visibility that spans the entire data life cycle, from ingestion to downstream usage. The tool enables real-time detection, diagnosis, and resolution of data issues by monitoring infrastructure elements like Kafka streams and Spark jobs, and performing checks for data quality metrics such as freshness and accuracy. By providing end-to-end observability, Datadog fosters better collaboration between data and software engineers, helping teams to quickly identify and address complex data issues, such as schema drift or resource underprovisioning, thereby maintaining trust in data systems and minimizing business disruptions. This is facilitated through built-in metrics and alerts, machine learning anomaly detection, and visibility into data lineage, allowing for a more proactive and coordinated approach to data management.

The Datadog Security Graph is a tool designed to enhance cloud security by modeling cloud environments as dynamic, relationship-aware graphs, addressing the challenge of understanding how compute, storage, identity, and networking components interact. Traditional security tools often struggle with visualizing multi-hop access paths, making it difficult to assess potential vulnerabilities such as which EC2 instances can access sensitive S3 buckets and whether they are exposed to the internet. The Security Graph provides a comprehensive view of cloud resources as interconnected nodes and relationships, offering security teams the ability to visualize, query, and analyze access paths, identity policies, and risk exposure effectively. By revealing both direct and transitive relationships, it helps identify potential attack paths, misconfigurations, and overly permissive access, allowing for proactive risk management and remediation. With features like prebuilt queries, a no-code query builder, interactive visualizations, and Access Insights, the Security Graph empowers users to investigate and respond to cloud security threats swiftly, providing a holistic understanding of cloud risks. Currently in preview, the tool is available to Datadog Cloud Security users, with options for new users to explore through a free trial.

Datadog Code Security offers an integrated approach to managing software vulnerabilities by continuously scanning code and libraries throughout the software development life cycle, from development to production. It addresses the challenges organizations face in prioritizing and remediating vulnerabilities by providing actionable insights and fixes, thereby reducing alert fatigue and fostering developer trust. The platform prioritizes risks based on runtime context and service impact, allowing teams to focus on the most critical issues and enabling a seamless remediation process within developers' existing workflows. By automatically associating vulnerabilities with responsible teams and providing real-time visibility into active threats, Datadog ensures efficient resolution of security risks, enhancing overall application security without hindering development speed.

Security analysts are encountering significant challenges in the cloud environment due to a shortage of staff in security operations centers (SOCs) and the increasing complexity of threats, particularly those driven by generative AI. These challenges are compounded by high false positive rates, leading to alert fatigue and slower threat detection, which increase the mean time to resolution (MTTR). Datadog Cloud SIEM has developed various features to support SOC teams, such as integrations, detection rules, and advanced visualizations, but the introduction of Bits AI Security Analyst marks a significant advancement. This new capability autonomously triages security signals, conducts thorough investigations, and provides actionable recommendations without human intervention. It leverages the MITRE ATT&CK framework for investigations, offering evidence-based conclusions and recommended remediations that are access-controlled. Bits AI Security Analyst, while tested and refined for accuracy, is now available in Preview, enhancing how security teams handle security signals in the cloud.

The text discusses the role and challenges of AI agents and multi-agent systems, which use large language models to execute complex tasks autonomously by collaborating and making decisions based on intermediate outcomes. These systems, though powerful in automating workflows, pose challenges in monitoring due to their non-linear and dynamic nature, with traditional visualization tools often falling short. The text highlights the diversity in frameworks like OpenAI's Agent SDK, LangGraph, and CrewAI, which complicate the observability due to differences in control flow and agent behavior. To address these challenges, Datadog's LLM Observability provides a solution by offering a clear visualization approach that captures agent operations, tool usage, and decision-making processes, helping teams understand, debug, and optimize their agentic systems. This includes tracking quality and performance metrics, ensuring functional correctness, and providing insights into the execution flow, thereby enabling developers to build, debug, and scale AI agent applications with improved accuracy and efficiency.

The text discusses Datadog Resource Policies, a tool designed to enhance cloud infrastructure management by allowing organizations to define and enforce best practices for resource configuration. As companies increasingly adopt multi-cloud architectures, Datadog offers a unified platform to oversee configuration compliance, which helps reduce the risks associated with misconfigurations. It provides out-of-the-box Policy Templates, shaped by industry standards and customer experiences, and allows for the creation of custom policies tailored to specific organizational needs. The tool enables real-time evaluation of infrastructure compliance, offering visibility into noncompliant resources and enabling efficient remediation through automated workflows and detailed instructions. By providing these capabilities, Datadog aims to streamline governance, increase operational efficiency, and maintain innovation without compromising control.

Proactive App Recommendations is a tool designed to help organizations manage the complexities of scaling operations by turning extensive telemetry data into actionable insights. As companies grow, they often face increased telemetry data from metrics, traces, logs, and frontend interactions, which can overwhelm teams trying to maintain service reliability and efficiency. This tool analyzes data from Application Performance Monitoring (APM), Real User Monitoring (RUM), Continuous Profiler, and Database Monitoring (DBM) to automatically identify performance and reliability issues, offering specific suggestions for fixes. It addresses backend inefficiencies by detecting patterns like sequential API calls that could be optimized, and it helps resolve frontend issues by identifying user frustration signals such as dead and rage clicks. Additionally, it provides code-level fixes for performance problems identified by the Continuous Profiler. Proactive App Recommendations streamlines the process of addressing telemetry data, minimizing the time required to implement necessary changes and enhancing the performance and reliability of services, all integrated within the Datadog platform at no additional cost to existing subscribers.

Datadog has introduced new features and enhancements to bolster security, compliance, and governance for cloud infrastructure. These updates include tag-based Data Access Controls and secret scanning to protect sensitive data, governance tools like the Governance Console and Fleet Automation, and the Security Graph for visualizing risks. The company has also improved its Cloud SIEM capabilities with anomaly and sequence detection to identify advanced threats. Secret Scanning, now in preview, helps detect embedded secrets in source code to prevent breaches. Datadog's Sensitive Data Scanner has been updated to identify human names in logs using machine learning, and the FIPS Agent ensures compliance with federal security standards. Datadog's Observability Pipelines support the Open Cybersecurity Schema Framework (OCSF) for standardized data processing, and Resource Policies allow for proactive governance of infrastructure with customizable templates. Additionally, Fleet Automation simplifies the process of upgrading Datadog Agents across infrastructure, and the new Cloudcraft Observability Overlay aids in identifying and managing observability blind spots. These features collectively aim to enhance security operations, compliance, and efficient resource management across cloud environments.

At the DASH event, Datadog introduced several new features aimed at enhancing automation and operational efficiency for organizations. Key innovations include Bits AI, which enables faster workflow creation through natural language, and Kubernetes autoscaling to optimize infrastructure performance and costs. The platform now offers AI-powered remediation for Kubernetes and real-time inspection of Amazon SQS dead-letter queues, aiding in quicker issue resolution. Datadog has also integrated direct action capabilities for AWS Cloud Cost Management, significantly streamlining the process of implementing cost-saving recommendations. Enhancements in database monitoring, Azure log automation, and deployment gating further improve operational workflows and reduce context switching. Additionally, the introduction of Datastore, a native database system, allows for seamless data management across applications and workflows. These updates reflect Datadog's commitment to simplifying infrastructure management and boosting organizational productivity.

Datadog Database Monitoring (DBM) is a tool designed to help teams proactively identify and address query regressions, which are unintended increases in query execution time that can cascade across services, affecting performance and user experience. By establishing historical baselines and utilizing anomaly detection, DBM automatically detects deviations in query performance and categorizes them based on severity, allowing teams to prioritize their efforts effectively. It provides visibility into the underlying causes of regressions by correlating them with contextual signals such as changes in query workload, user activity, schema modifications, and recent deployments. This approach enables database owners to maintain system reliability and performance without excessive resource allocation or unnecessary query optimization efforts. Through the use of DBM, users can receive timely alerts and recommendations, helping to manage database query performance and mitigate potential disruptions before they escalate into larger issues.

Datadog Cloud SIEM has introduced Content Anomaly Detection, a novel method for identifying unusual activities by analyzing the content of logs rather than just their frequency or recency. This new detection approach aims to uncover subtle deviations in user behavior or unexpected command patterns that traditional methods may miss. By establishing a baseline of normal log content during a learning period, the system can detect anomalies through Jaccard similarity, implemented with MinHash and Locality Sensitive Hashing, and trigger alerts based on configured thresholds. It allows security teams to fine-tune detection rules with parameters such as similarity threshold, minimum similar items, and evaluation window to reduce noise and enhance precision. This method enhances the ability to detect risks, like unauthorized access or configuration changes, that might otherwise go unnoticed, providing deeper visibility into log content beyond their structure or volume.

Datadog has enhanced its platform with three powerful tools—Sheets, DDSQL Editor, and advanced analysis features in Notebooks—designed to enable users to perform in-depth analysis and visualization of telemetry data without needing to export it to external tools. Sheets offer a spreadsheet-style interface ideal for users familiar with spreadsheets, allowing them to manipulate and visualize large datasets such as logs and infrastructure metrics. DDSQL Editor caters to those who prefer using SQL for precise and flexible querying, facilitating complex analyses like cross-source joins and cost optimization across cloud environments. Notebooks provide a collaborative space for building multistep queries and documenting investigations, which is particularly useful for incident analysis and correlating data across multiple sources. Together, these tools cater to a range of user preferences and workflows, enabling seamless exploration and analysis directly within the Datadog platform.

The Datadog Internal Developer Portal (IDP) is designed to enhance software delivery by providing developers with tools for efficient and confident software deployment. It centralizes engineering knowledge, self-service tools, and live observability data, facilitating a streamlined workflow by reducing the need for developers to switch contexts frequently. The portal's Software Catalog offers a dynamic view of software systems and APIs, helping teams easily access system architecture, ownership details, and deployment workflows, which are crucial during incidents or when onboarding new developers. Self-Service Actions allow developers to execute common tasks like provisioning infrastructure or configuring CI/CD without waiting for platform engineering approvals, thereby improving development velocity. Meanwhile, Datadog Scorecards provide criteria for production-readiness, ensuring that services meet essential standards before deployment. These features collectively help organizations maintain consistency across platforms and improve the reliability of their services, with the portal being easily integrable into existing systems for those already using open-source IDPs.

Bowen Chen Cursor is an AI-enhanced code editor designed to boost developer productivity through features like contextual code completion and an agent that assists with commands and error fixes. The recent integration of the Datadog Cursor extension enriches this experience by providing developers with access to Datadog's observability tools, such as Error Tracking and Live Debugging, within the Cursor IDE. This extension uses the Datadog Model Context Protocol (MCP) Server, allowing seamless access to Datadog APIs and reducing context switching. Through an example involving a checkout error in an online store, the extension demonstrates its capability to identify and troubleshoot issues by utilizing dynamic logpoints and capturing real-time data. By generating unit tests based on live production data, developers can validate fixes and ensure application reliability. This integration, currently in preview, aims to streamline the debugging process and enhance workflow by offering direct access to real-time observability data and testing tools.

As organizations expand their cloud operations, managing object storage costs and performance becomes critical, especially with the vast amounts of data generated by AI and data-intensive workloads. Datadog Storage Management provides a solution by offering visibility and optimization tools across Amazon S3, Google Cloud Storage, and Azure Blob Storage, focusing on cost and performance optimization and operational efficiency. The tool provides prefix-level metrics for Amazon S3, enabling users to manage life cycle and retention policies, quickly identify cost-saving opportunities, and resolve performance bottlenecks by pinpointing high-traffic prefixes. For instance, it can highlight data that is rarely accessed and better suited to lower-cost storage tiers like Glacier, or identify hot prefixes that cause latency issues, thereby allowing for better workload distribution. This granular visibility into storage usage patterns and configurations allows organizations to make informed decisions, ensuring efficient cloud storage management. Datadog Storage Management is available for Amazon S3 and includes object size metrics for Google Cloud Storage and Azure Blob Storage, with additional request-related metrics for these platforms expected soon.

Migrating historical logs to a new logging platform, such as Datadog, can be a challenging task, especially when dealing with both active and archived logs, but Observability Pipelines offers a seamless solution. The platform's dual-shipping capability allows active logs to be routed to the new platform without disrupting existing workflows, while historical logs, crucial for security investigations and compliance, require a tailored approach due to real-time data limits in many logging tools. Observability Pipelines facilitates this by enabling the routing of historical logs through customizable pipelines that can correct timestamps and forward logs to cloud storage for future rehydration. The process involves extracting logs from archives like Elasticsearch, Splunk, or cloud storage services such as Amazon S3, Google Cloud Storage, and Azure Blob Storage, using custom scripts and processing tools to transform and store them. The platform's Custom Processor can automatically adjust timestamps and apply transformations, ensuring logs are ready for long-term storage and eventual rehydration into Datadog for detailed analysis. Observability Pipelines supports various data processing features, including Grok parsing, JSON parsing, and sensitive data redaction, providing flexibility and control over log migration and management.

Datadog's new Cloudcraft feature offers a dynamic visualization platform for AWS cloud infrastructures, addressing challenges faced by teams in managing and understanding complex cloud environments. By providing automated, real-time diagrams, Cloudcraft serves as a cohesive source of truth, enhancing visibility and facilitating navigation through key observability, cost, and security insights. It includes features like automatic diagram generation, search-and-highlight capabilities, and customizable views to organize resources by various criteria. Additionally, overlays highlight agent status, security misconfigurations, and potential cost optimization opportunities, enabling teams to proactively address issues directly in the diagram interface. Cloudcraft integrates seamlessly with the wider Datadog ecosystem, offering a comprehensive approach to cloud infrastructure management and decision-making.

Datadog has released Go tracer v2.0.0, an upgrade from its previous version, which offers improved security, stability, and a simplified API. This update results from feedback from both Datadog's community and internal teams, with all internal services already migrated to the new version. The Go tracer v2 introduces a leaner API, enhances security by isolating integration dependencies, and shifts its package repository from gopkg.in to github.com for scalability. It also supports configurations using a single struct to optimize performance. For users unable to fully transition to v2, a transitional version, v1.74.0, is available, allowing the gradual migration of services. Datadog encourages users to upgrade to v2 to leverage its new features and improvements, offering a migration guide and tool to assist in the process. The company remains committed to incorporating community feedback for future updates and invites users to try the new version and provide feedback.

Organizations often use a variety of platforms for observability and security, such as Datadog Log Management for DevOps and SIEM platforms like Microsoft Sentinel and Google Security Operations for security teams. To streamline processes, many companies aim to standardize log processing to ensure consistent formatting and enrichment. Datadog Log Management offers a solution by centralizing log processing, allowing teams to parse, enrich, and forward logs efficiently. It enables the creation of log pipelines that structure and standardize logs, enrich them with contextual data, and generate metrics for real-time performance tracking. Logs can then be forwarded to platforms like Microsoft Sentinel or Google SecOps, reducing infrastructure overhead and maintaining workflow continuity. This integration allows security teams to focus on threat detection and incident investigation without managing complex infrastructure, while also optimizing security analytics. Organizations can start using Datadog to enhance their log management and security operations with a 14-day free trial or by configuring their existing systems for seamless integration.

The text discusses the development and implementation of Datadog's Automatic Faulty Deployment Detection, a feature designed to quickly identify problematic software deployments in services monitored by Datadog Application Performance Monitoring (APM). The process began with an unsupervised learning model due to challenges like label scarcity and data imbalance, as faulty deployments are rare and context-dependent. The team defined faulty deployments by significant increases in error rates and used an iterative framework to refine detection through statistical checks. Later, they transitioned to supervised learning, using weak supervision to improve label quality and model accuracy. This approach allowed for earlier detection of deployment issues, improving both precision and recall. The project highlights how data science and machine learning can enhance software deployment reliability and offers techniques applicable to other domains with similar challenges.

OpenTelemetry (OTel) is becoming the industry standard for observability data, and Datadog has integrated support for OTel-native data alongside its own native telemetry format to offer a consistent monitoring experience. This integration allows users to utilize OTel-native metrics with Datadog's dashboards, queries, and visualizations, facilitating a seamless adoption of OTel without disrupting existing workflows. Datadog now enables viewing of OTel-native metrics in its out-of-the-box integration dashboards, making it easier for teams to monitor systems like Kafka, Docker, and NGINX with minimal setup. The platform's Semantic Mode allows for comprehensive queries across OTel and Datadog metrics by automatically detecting equivalent metrics, eliminating the need for manual mapping or separate queries. These capabilities provide a unified observability experience, reducing overhead and speeding up insights for users with mixed telemetry environments.

Datadog offers a comprehensive approach to managing custom metrics, helping organizations maintain visibility while controlling costs as their telemetry data grows. By utilizing features such as Metrics without Limits, Datadog enables users to separate ingestion from indexing, allowing for significant reductions in custom metrics usage without sacrificing critical insights. The platform provides built-in alerts and a Metrics Volume Management page to help monitor and attribute costs, enabling proactive responses to usage spikes. Datadog also supports role-based access control (RBAC) and audit trails to safeguard observability, ensuring only authorized users can modify configurations. These tools empower teams to optimize their metrics governance and maintain budget control while scaling their observability capabilities.

The text introduces "RUM without Limits™," a new model for Real User Monitoring (RUM) by Datadog that addresses the limitations of traditional RUM tools, which often require trade-offs between capturing all user sessions and managing costs. Unlike fixed sampling rates that can overlook critical issues, this model allows for the capture of 100% of user sessions, providing complete visibility into user experiences while using dynamic filters to retain only high-value data. This approach enables teams to monitor application performance metrics accurately, detect and resolve issues in real-time, and optimize user experiences without incurring high storage costs. The tool offers over 30 precise metrics, retained for 15 months, that can be used in service level objectives (SLOs), dashboards, and alerts, and supports deep filtering and segmentation by key dimensions like OS, browser, and app version. By focusing on sessions that matter, such as those with errors or from high-value users, teams can prioritize and manage data retention effectively, enhancing digital experiences and aligning storage costs with business value. The RUM without Limits™ model is now generally available for Datadog users, offering a unified monitoring platform with a 14-day free trial for newcomers.