Home / Companies / Datadog / Blog / March 2026

March 2026 Summaries

36 posts from Datadog

Filter
Month: Year:
Post Summaries Back to Blog
Round trip query latency often extends beyond the database itself, encompassing various elements along the data path such as connection pools, load balancers, and proxies. This complexity can make it challenging to pinpoint the true source of latency, as traditional database monitoring tools typically focus only on database-centric metrics. Datadog's correlated Application Performance Monitoring (APM) and Database Monitoring (DBM) enable users to dissect round trip latency into its components, distinguishing between database execution time and other overheads, thereby facilitating targeted troubleshooting. By employing Datadog's "Round trip overhead analysis," users can identify whether latency issues stem from within the database or from external factors, such as a saturated connection pool like PgBouncer. This approach is critical in modern database architectures, allowing engineers to avoid misdirected efforts and focus on the actual bottlenecks in the data query process.
Mar 27, 2026 1,079 words in the original blog post.
Caching is essential for enhancing the performance of modern JavaScript applications by speeding up data retrieval and improving user experience, particularly for apps with resource-intensive operations like animations or API calls. Choosing the right caching method can be challenging due to the diverse types of data handled by JavaScript apps. Popular caching strategies include in-memory caches for fast but temporary storage, Web Storage API for more persistent data with limitations, and IndexedDB for large, structured datasets. For both static and dynamic data, advanced methods like the Cache API and Origin-Private File System offer flexible storage and retrieval options. Each caching method comes with trade-offs in terms of performance, staleness, security, and complexity, necessitating a balanced approach. Tools like Datadog can assist in identifying which resources to cache by monitoring performance metrics and optimizing cache strategies accordingly to enhance speed and reliability in web applications.
Mar 27, 2026 3,224 words in the original blog post.
Datadog's Bits AI Dev Agent for Code Security addresses the challenge of managing increasing code vulnerabilities due to the rapid adoption of AI-assisted development by automating the remediation process. Integrated with Datadog's Code Security, the Dev Agent efficiently reduces vulnerability backlogs by automatically generating fixes, creating pull requests (PRs), and allowing for batch remediation of vulnerabilities like SQL injections. This system maintains control and transparency by providing detailed session tracking and real-time monitoring, enabling security and engineering teams to efficiently oversee remediation progress. By combining rule-based and AI-native static code analysis, Datadog enhances vulnerability detection accuracy, filters out false positives, and allows for scalable vulnerability management practices that align with the accelerated pace of software development.
Mar 26, 2026 954 words in the original blog post.
Nutanix is a hyperconverged infrastructure (HCI) platform that integrates compute, storage, and virtualization into a single software-defined stack, simplifying the management of virtualized workloads. Prism Central manages clusters, providing insights into health, performance, and capacity, which necessitates a nuanced approach to monitoring and troubleshooting, as performance issues can arise from various sources like cluster resource pressure or inefficient VM allocations. Datadog's integration with Nutanix enhances this troubleshooting process by collecting telemetry data and Prism Central alerts, allowing for comprehensive monitoring of both the infrastructure and the applications it supports. This integration helps in identifying root causes of performance issues, such as latency spikes, by correlating infrastructure and application data, enabling swift resolution without the need to switch tools. The article emphasizes the importance of analyzing cluster health, storage performance, and capacity trends to maintain optimal operations and provides a real-world example of resolving a latency issue by investigating and rebalancing resources, illustrating the integration's efficacy in maintaining system performance and reliability.
Mar 26, 2026 1,454 words in the original blog post.
As AI-powered products become integral to organizational operations, the necessity for responsible AI governance is increasing, driven by demands from customers, partners, and regulators for assurance of responsible AI system management. Datadog has achieved ISO 42001 certification, marking it as compliant with the international standard for AI management systems, which ensures ethical, transparent, and lawful AI operations. This certification assures stakeholders of Datadog's comprehensive control over the AI lifecycle, from development to monitoring, aligning with international standards for risk management and continuous improvement. With many organizations subject to new AI regulations, Datadog's certification simplifies vendor assessment and supports companies in bolstering their AI governance frameworks by offering a reliable, third-party-verified benchmark. Datadog's dedication to trust and accountability is further demonstrated by its suite of certifications, including ISO 27001, ISO 27701, HIPAA, PCI, and TISAX, underscoring its commitment to security, privacy, and responsible operations.
Mar 26, 2026 380 words in the original blog post.
The updated Datadog Host Map provides a comprehensive, real-time visual representation of modern infrastructure, incorporating hosts, clusters, pods, and containers into a single interactive view. This redesign addresses the increasing complexity of infrastructure due to the shift towards Kubernetes and containerized workloads. It enables users to quickly assess infrastructure health, monitor rollouts and migrations, and identify patterns by grouping resources based on various dimensions and coloring them by relevant metrics. The Host Map introduces hierarchical views for better understanding of relationships between infrastructure components, aiding in efficient troubleshooting and providing clear insights during infrastructure changes. Enhanced search capabilities allow users to construct precise queries to locate specific resources, making the Host Map adaptable to diverse workflows and investigative paths. Overall, the new Host Map serves as a critical tool for visualizing and managing infrastructure, facilitating both high-level awareness and detailed analysis.
Mar 25, 2026 994 words in the original blog post.
Juniper Mist, an AI-powered networking platform, enhances business operations by providing critical insights into wireless environments, focusing on access point performance and radio frequency health. The integration of Juniper Mist with Datadog, currently in Preview, allows for the collection and analysis of device, client, port, and traffic metrics via the Juniper Mist API, enabling teams to correlate user experience with application and infrastructure data. This integration assists engineering teams in monitoring network health across sites, analyzing port performance, and tracking network throughput and traffic patterns to identify and address connectivity issues before they affect users. The out-of-the-box dashboard in Datadog presents high-level device and client metrics, allowing teams to quickly assess network status and isolate connectivity problems. By examining port and PoE telemetry data, teams can detect overloaded endpoints and uncover root causes of network issues. Additionally, monitoring network throughput and traffic patterns aids in identifying performance bottlenecks and planning capacity needs in distributed enterprises. Teams can set alerts to detect abnormal behavior, ensuring issues are addressed proactively. The integration ultimately connects network telemetry data with application metrics, helping teams understand how network behavior impacts user experience. To utilize the Juniper Mist integration, interested parties can join the Preview by contacting their Datadog representative or signing up for a free trial.
Mar 25, 2026 672 words in the original blog post.
Organizations utilizing Oracle Fusion Cloud Applications often face challenges in monitoring performance due to limited access to the underlying Oracle Cloud Infrastructure. Datadog's integration with Oracle Fusion addresses this by collecting metrics and logs from Enterprise Scheduler Service (ESS) jobs, enabling teams to monitor application performance, track data flows, audit user activity, and test user experiences. This integration provides insights into job behavior and user activity, allowing for early detection of performance regressions and security risks. Datadog's Synthetic Monitoring further enhances visibility by simulating real user interactions with Oracle Fusion applications, providing comprehensive performance data without direct infrastructure access. The out-of-the-box dashboards and audit logs help teams analyze and respond to anomalies, ensuring business continuity and security compliance.
Mar 23, 2026 953 words in the original blog post.
Datadog has announced a Preview of native OpenTelemetry (OTel) support in its Kubernetes Explorer, enhancing its existing OTel metrics capabilities. This development allows users to explore and troubleshoot Kubernetes environments using their OTel data without sacrificing vendor flexibility or data ownership. The Kubernetes Explorer facilitates understanding of resource health and correlates metrics, logs, and traces by providing a unified platform for viewing clusters, nodes, and workloads. It addresses challenges related to metric variations and semantics by automatically translating OTel metrics into Datadog-standard representations. The Explorer also aids in root cause analysis by mapping metrics to Kubernetes resources, allowing users to diagnose issues more efficiently across hybrid environments. This new feature aims to streamline the troubleshooting process by enabling seamless navigation from high-level health signals to specific container issues, ensuring users can maintain data integrity while using their preferred telemetry pipeline.
Mar 23, 2026 1,026 words in the original blog post.
Datadog Cloud SIEM has introduced new capabilities to address the challenges faced by security teams in an evolving threat landscape dominated by AI-driven attacks and identity misuse. Traditional SIEMs struggle with scaling modern data and require manual correlation, leading to increased detection and response times. Datadog's enhancements, including AI-powered investigations and Bits AI Security Analyst, automate signal correlation and analysis, significantly reducing investigation times and analyst workloads. The platform integrates threat intelligence and behavioral analytics to provide real-time context, helping teams identify high-risk threats and compromised identities more effectively. Additional features like UEBA detect subtle signs of compromise, while Risk Insight Notifications prioritize threats by monitoring risk scores. The platform also supports enterprise-scale security operations with SQL-based detections, content packs, and integrated Security Case Management to streamline investigations and improve response times. These advancements aim to modernize security operations by delivering speed, actionable insights, and scalability, ultimately allowing teams to automate processes and respond to threats with confidence.
Mar 23, 2026 1,034 words in the original blog post.
At Datadog, a new upsert query designed to track the last time hosts were seen unexpectedly doubled disk writes and quadrupled Write-Ahead Logging (WAL) syncs due to row locks, despite most operations being no-ops. This occurred because even non-updating upserts lock rows, which is recorded in the WAL. With Postgres's single-writer limitation, this increased disk usage was unsustainable, especially at the scale of Datadog's operations. To address this, Datadog's engineers analyzed the WAL using the pg_walinspect extension and rewrote the query to avoid locking rows unnecessarily by employing a data-modifying Common Table Expression (CTE). This change reduced overhead by preventing unwanted WAL entries while maintaining the system's correctness. The exploration highlighted the importance of understanding WAL behavior and optimizing queries to manage database performance effectively, even for operations expected to be no-ops.
Mar 23, 2026 2,419 words in the original blog post.
Datadog LLM Observability introduces Automations and Annotation Queues to enhance the quality evaluation of large language models (LLMs) in production environments, addressing the challenge of detecting subtle quality failures that traditional metrics might miss. Automations allow for the automatic routing of production traces to datasets or annotation queues based on configurable rules, ensuring that high-signal requests are prioritized for review without overwhelming the system. Annotation Queues facilitate systematic human review by providing a structured workspace where domain experts can apply consistent labels and qualitative feedback, leveraging a shared labeling schema to ensure reliable and comparable evaluations. This framework supports a continuous quality improvement loop by using human annotations as ground truth to calibrate automated evaluators, build and maintain golden datasets, and track failure patterns over time, ultimately enabling teams to refine models and prompts effectively. By integrating human judgment with automated processes, Datadog ensures that LLM evaluations remain aligned with real user behavior and production traffic, fostering ongoing improvements as applications evolve.
Mar 23, 2026 857 words in the original blog post.
The release of Datadog Terraform provider v4.0.0 introduces significant enhancements aimed at improving reliability, security, and governance for managing Datadog configurations as code. This major update offers predictable monitor governance, allowing permissions to persist as defined in configurations, and introduces a unified AWS integration resource to streamline management by replacing four separate legacy resources. The new version also emphasizes credential security by tightening application key workflows, removing the datadog_application_key data source, and ensuring compatibility with one-time read application keys. It further upgrades to support Terraform protocol v6, enabling more efficient future improvements to schemas and validation. As teams prepare to upgrade, they are encouraged to verify their Terraform CLI version and follow migration guidance, with the option to continue using older configurations if they choose to remain on v3.
Mar 20, 2026 790 words in the original blog post.
The 2025 State of Containers and Serverless report highlights that while 64% of organizations use Kubernetes Horizontal Pod Autoscaler (HPA) to manage workload capacity, only 20% use custom metrics for scaling, relying instead on resource metrics like CPU and memory. However, these resource metrics often fail to accurately reflect real-time demand, leading to delayed scaling and potential performance degradation. The report suggests that custom metrics, such as queue depth, request rate, and tail latency, provide more effective scaling signals for specific workload patterns like event-driven workers, high-throughput APIs, latency-sensitive services, and database-backed services. Datadog Kubernetes Autoscaling (DKA) is presented as a solution that integrates metrics, observability, and autoscaling into one platform, allowing for more responsive scaling based on application metrics. DKA enables teams to define scaling signals using Datadog's query language, providing tools to tune scaling behavior and safeguard against custom metric delays by reverting to local CPU-based scaling when necessary. The article emphasizes the importance of aligning scaling strategies with business logic, encouraging the use of custom metrics to avoid overprovisioning and ensure high performance.
Mar 18, 2026 1,496 words in the original blog post.
Cloud environments are inundated with security signals, necessitating precise determination of their legitimacy and posing a challenge to threat analysis. AI can enhance this process by identifying patterns and risks through User and Entity Behavior Analytics (UEBA), which relies on historical behavioral patterns of cloud identities. However, AI's effectiveness depends on consistent telemetry data and well-defined security controls, as it can falter with incomplete data or unfamiliar behaviors. To optimize AI's threat analysis, cloud environments must provide clear guidelines and context through logs and metadata tags, enabling AI to accurately trace and correlate activities. This structured approach ensures AI can distinguish between normal and anomalous behavior, focusing investigations on significant security risks.
Mar 18, 2026 1,398 words in the original blog post.
Aruba Central offers a centralized, cloud-based platform for managing distributed wired and wireless network infrastructures at scale, essential for modern organizations operating across multiple locations. The Datadog integration with Aruba Central, currently in Preview, enhances network management by providing a unified dashboard that offers visibility into device health, performance metrics, and network usage. This integration allows teams to monitor access points and switches, analyze network performance, and track application usage, which are critical for maintaining reliable connectivity and efficient operations. By surfacing key metrics such as device uptime, CPU utilization, and application traffic trends, teams can quickly identify and address network issues, correlate them with application and infrastructure signals, and make informed capacity planning decisions. The integration is designed to improve incident response and reduce user disruption by enabling faster identification and resolution of network-related problems, ultimately facilitating better management of infrastructure across distributed environments.
Mar 17, 2026 717 words in the original blog post.
Datadog's integration with ServiceNow offers a comprehensive solution for incident management by streamlining the process of identifying, investigating, and resolving technical issues while maintaining governance and compliance. This bidirectional integration allows engineers to conduct investigations in Datadog with automatic synchronization of updates to ServiceNow, ensuring that records remain complete and accurate without manual intervention. Datadog enhances incident response with AI and automation, providing features such as automatic event correlation, real-time collaboration, and automated remediation, which help reduce context switching and mean time to resolution (MTTR). It generates AI-assisted postmortems and telemetry-focused analytics, enabling teams to understand systemic issues and improve reliability. The integration also supports communication tools like Slack and Microsoft Teams for seamless collaboration and keeps stakeholders informed with real-time updates, minimizing downtime and administrative overhead while maintaining an audit trail for compliance purposes. By using ServiceNow as the system of record and Datadog as the system of resolution, organizations can balance governance with operational efficiency.
Mar 17, 2026 1,285 words in the original blog post.
In the constantly challenging landscape of technical environments, proactively managing risks is crucial to prevent disruptions and data breaches. The Datadog Risk team tackles this by using Datadog Case Management to centralize risk data into a single repository, enhancing visibility and response capabilities. This approach involves collecting data from distributed systems to create a comprehensive risk register, which is then enriched by a custom AI tool that assesses and prioritizes risks based on factors like impact and likelihood of exploitation. The team employs LLM Observability to ensure accurate risk information and utilizes Workflow Automation to streamline remediation processes, assigning tasks and updating case statuses efficiently. Role-specific dashboards provide stakeholders with tailored insights into the organization's risk posture, enabling swift identification and mitigation of risks. This comprehensive strategy not only standardizes responses across different risks but also adapts to their varying severities and timelines, thereby maintaining a robust defense against potential security incidents.
Mar 17, 2026 1,018 words in the original blog post.
Load balancers serve as crucial entry points in distributed applications by directing incoming traffic to specific endpoints, optimizing throughput and ensuring scalability. Beyond basic load distribution, load balancers perform complex tasks like request analysis, traffic routing based on various parameters, implementing security mechanisms, and conducting health checks, resulting in significant log generation. The logs from application load balancers (ALBs) and network load balancers (NLBs) provide valuable insights into system performance, with ALBs focusing on HTTP/gRPC requests and NLBs on transport-level connections. Monitoring these logs helps identify error spikes, latency outliers, traffic anomalies, and SSL/TLS handshake failures. Datadog's Observability Pipelines and Log Management tools assist in processing high-volume logs, reducing noise, and managing costs by transforming logs before routing them to downstream tools, ensuring efficient management of load balancer activities. These tools provide predefined Packs for popular load balancers, enabling teams to parse, normalize, and analyze logs to maintain performance and security in distributed systems.
Mar 17, 2026 2,655 words in the original blog post.
David Lentz Karpenter is a highly adaptable Kubernetes autoscaler designed to enhance cluster performance and cost efficiency by dynamically requesting or reducing compute resources based on workload needs. Developed by AWS, Karpenter is gaining traction across various cloud platforms beyond Amazon EKS. It distinguishes itself from the traditional Cluster Autoscaler (CAS) by interacting directly with cloud provider APIs, enabling faster provisioning and better cost optimization through techniques like workload consolidation and just-in-time resource allocation. Karpenter's architecture is characterized by a separation of concerns between provider-agnostic NodePools and provider-specific NodeClasses, allowing it to efficiently manage compute capacity while respecting operational constraints and application performance. It further optimizes costs by using a priority hierarchy for capacity types, preferring Reservations, Spot, and then On-Demand instances, and also provides mechanisms for managing node lifecycles through expiration and disruption policies. While both Karpenter and CAS manage scaling, Karpenter offers more flexibility and responsiveness, particularly in dynamic environments, although it introduces a level of complexity that requires careful observability to ensure optimal cluster performance and cost-effectiveness.
Mar 11, 2026 2,748 words in the original blog post.
Karpenter's architecture facilitates just-in-time provisioning and active node consolidation, making it crucial to monitor specific metrics to understand its behavior and performance. These metrics, which are categorized as STABLE, BETA, ALPHA, or DEPRECATED, provide insights into Karpenter's scheduling, disruption, cloud provider interactions, controller internals, and cost optimization processes. For instance, metrics such as karpenter_pods_startup_duration_seconds and karpenter_scheduler_scheduling_duration_seconds help gauge the efficiency of the scaling process and the potential causes of latency. Additionally, metrics like karpenter_voluntary_disruption_eligible_nodes and karpenter_nodeclaims_termination_duration_seconds reveal opportunities for cost savings and highlight challenges in node management. Monitoring cloud provider metrics, such as karpenter_cloudprovider_errors_total and karpenter_cloudprovider_duration_seconds, can help identify issues stemming from API failures or latency. Finally, metrics related to controller performance, including controller_runtime_reconcile_time_seconds and workqueue_depth, provide insights into Karpenter's ability to manage cluster changes efficiently. By correlating these metrics, users can ensure that Karpenter maintains optimal performance and cost-efficiency.
Mar 11, 2026 2,066 words in the original blog post.
As product analytics becomes more integrated with AI and data centralization, the role of product managers (PMs) remains crucial in interpreting data and forming actionable insights. The complexity of product analytics lies not in reading dashboards but in extracting meaning from evolving data while mitigating human bias. Two PMs can draw different conclusions from the same data due to their distinct hypotheses and goals, underscoring the importance of aligning on definitions and expectations before analysis. Effective product management involves bridging quantitative data with qualitative insights, segmenting data with intent, and selecting appropriate interventions based on the data narrative. Tools like Datadog Product Analytics enhance visibility into user journeys, facilitating informed decision-making by connecting frontend behavior with backend performance. Ultimately, the key to successful analytics is not merely accumulating data but fostering a shared understanding of what the data communicates, supported by clear hypotheses, consistent metrics, and informed human judgment.
Mar 11, 2026 1,703 words in the original blog post.
Karpenter, an open-source Kubernetes node provisioning tool, facilitates efficient cluster management through just-in-time provisioning and active node consolidation. This article outlines how to leverage vendor-agnostic tools like Prometheus and Grafana to monitor Karpenter's performance by capturing and visualizing key metrics such as provisioning latency, disruption behavior, and batching efficiency. Kubernetes-native commands and tools like kubectl can be used for real-time audits and troubleshooting, while Prometheus serves as a robust backend for storing and querying Karpenter metrics. Grafana complements this by enabling customizable visualizations and alerts to monitor trends and potential issues. Additionally, Karpenter's structured JSON logs offer granular insights into its operational decisions, which are crucial for diagnosing performance problems. The article emphasizes the importance of deep visibility into Karpenter's decision-making processes to maintain cluster efficiency and suggests that while self-hosted observability tools like Prometheus and Grafana are valuable, scaling them can be complex. The next installment in the series will explore using Datadog for a more comprehensive and managed observability solution.
Mar 11, 2026 1,667 words in the original blog post.
This article explores how Datadog aids in monitoring and optimizing Karpenter, an autoscaling tool for Kubernetes clusters, by providing a comprehensive integration that tracks key metrics and correlates them with cloud costs. It details the process of enabling Karpenter integration in Datadog, which involves configuring the Datadog Agent to scrape Karpenter's telemetry data and setting up alerts for performance issues. The article highlights the role of Datadog Cloud Cost Management (CCM) in understanding the financial impact of Karpenter's scaling activities, enabling users to visualize cost efficiency, identify high-cost workloads, and optimize cloud spend. By tracking NodeClaim life cycle metrics and monitoring reconciliation latency, users can proactively address scaling issues and enhance application performance. The integration facilitates better cost management by providing insights into resource utilization and offering tools to monitor and govern cloud expenditures, ensuring that every node provisioned by Karpenter is both performance-optimized and cost-effective.
Mar 11, 2026 1,602 words in the original blog post.
Observability data plays a crucial role in understanding system behavior during outages or security incidents by providing essential context for reconstructing attack paths and correlating security signals. This data, comprising metrics, events, logs, and traces (MELT), is increasingly vital in complex environments with AI applications, where fragmented tooling can obscure critical insights. By integrating observability data with AI, teams can expedite threat detection, investigation, and response, as seen in the Cloudflare breach case, where the data facilitated a comprehensive understanding of the incident. Datadog exemplifies this approach by merging SRE and security teams to unify observability and security practices, enhancing system resilience and accuracy in monitoring. Furthermore, integrating observability data throughout the software development lifecycle helps prioritize genuine risks amidst AI-assisted development and reduces false positives by providing necessary context for code analysis. Datadog's systems leverage observability data to process vast amounts of information, improving security event interpretation and remediation across a wide array of customer environments, thus demonstrating the importance of treating observability data as a foundational element in cloud security monitoring.
Mar 10, 2026 1,181 words in the original blog post.
The Datadog Model Context Protocol (MCP) Server, now generally available, enhances developer workflows by connecting Datadog tools with AI agents used in platforms like GitHub Copilot and Visual Studio Code. This integration allows developers to automate processes such as onboarding to Datadog products, detecting unused services, correlating incidents with feature flag changes, and monitoring cloud costs. For onboarding, a custom agent can use the MCP Server to guide developers through best practices, while another agent identifies and decommissions services with no real traffic, reducing unnecessary cloud spending. In incident management, the MCP Server helps correlate alerts with feature flag changes, enabling faster response times and minimizing customer impact. Additionally, organizations use it to monitor cloud costs, alerting teams of unexpected spikes to prevent them from becoming the norm. The MCP Server facilitates these capabilities by providing enriched context, thus improving efficiency and reducing manual workload for engineering teams.
Mar 09, 2026 1,593 words in the original blog post.
Datadog's exploration of securing open-source contributions highlights the challenges posed by malicious actors leveraging AI agents to exploit vulnerabilities in continuous integration (CI) pipelines. The company faced attacks from an AI agent named hackerbot-claw, which targeted Datadog's public repositories by attempting to inject malicious code and exploit GitHub Actions workflows. Despite the sophistication of these threats, Datadog's robust defensive strategies, including their LLM-driven code review system BewAIre, effectively identified and mitigated these risks. The incidents underscored the importance of implementing comprehensive security measures, such as minimizing token permissions, avoiding insecure code execution vectors, and enforcing CI security best practices. Datadog emphasizes the necessity of a defense-in-depth approach, which combines proactive detection with strict privilege scoping to limit potential compromises, and highlights the evolving capabilities of AI in both offensive and defensive security contexts.
Mar 09, 2026 3,602 words in the original blog post.
Ming Chen and Sesh Nalla explore the challenges and advancements in optimizing distributed systems, particularly focusing on the use of BitsEvolve for autonomous, real-time code optimization in Datadog's Unicron service. They demonstrate how AI-assisted development can produce verifiably correct and more efficient distributed systems by utilizing a five-stage pipeline that includes specialization, LLM evolution, formal verification, shadow evaluation, and live hot-swapping of WebAssembly modules. The study reveals significant performance improvements, with optimizations leading to message throughput increases of up to 541% in tested workloads. The method relies on a two-server architecture, where the evolution server continuously optimizes code, generating WASM modules that the aggregation server can integrate without downtime. This framework highlights the potential of LLM-driven optimization to discover fundamentally different algorithms that traditional methods might not easily achieve, while maintaining safety and correctness through rigorous verification processes.
Mar 09, 2026 1,944 words in the original blog post.
AI agents have revolutionized software development by drastically accelerating the code generation process, but this has shifted the bottleneck to verifying the correctness of that code. At Datadog, a strategy known as harness-first engineering has been adopted, where automated checks replace extensive human review to ensure code reliability. This involves using deterministic simulation testing, formal specifications, and observability-driven feedback loops to verify AI-generated code. Two projects, redis-rust and Helix, illustrate the approach's effectiveness; both achieved significant performance improvements and maintained correctness without the need for traditional code reviews. Redis-rust demonstrated an 87% memory reduction, while Helix improved produce latency significantly compared to a baseline Kafka cluster. The harness-first methodology allows for quick iterations, with the human role focused on defining system ideas and strengthening verification processes, highlighting a shift in engineering towards designing checks rather than inspecting outputs. The integration of formal methods and automated pipelines has inverted the traditional balance between scalability and rigor, enabling AI agents to handle tasks that previously required significant human oversight. Observability ensures that any discrepancies between modeled behavior and actual performance can refine the verification process over time, marking a significant step toward industrializing software engineering.
Mar 09, 2026 2,666 words in the original blog post.
Nick Vecellio, Principal Engineer and Co-founder of NoBS, a Premier Datadog Partner, emphasizes the importance of treating Datadog migrations not just as a technical task but as an opportunity to rethink observability strategies. By viewing the migration as a chance to redesign observability around flexibility, reuse, and actionability, teams can avoid the common mistake of simply transferring existing dashboards and alerts. Key principles for successful migration include migrating only necessary resources, using tagging as a foundational strategy, consolidating dashboards around common questions, and ensuring all alerts are actionable to prevent alert fatigue. This approach results in a more streamlined, efficient, and strategically valuable observability setup, with fewer redundant dashboards and more reliable alerts. Moreover, a mindset focused on redesign rather than mere relocation ensures long-term success and adaptability within the Datadog environment.
Mar 09, 2026 962 words in the original blog post.
Bits AI SRE, an autonomous Site Reliability Engineering agent, has been updated to enhance its capabilities in investigating and resolving complex alerts by integrating more seamlessly with existing workflows and tools. The latest version boasts a faster, more intelligent agent with broader data access and improved triage and remediation features, allowing it to effectively navigate complex observability environments. The updated agent can plan investigations, evaluate hypotheses in real-time, and complete investigations twice as fast as before. It now offers expanded access to Datadog data sources, enabling comprehensive troubleshooting across the full stack by correlating signals from various domains, such as user experience, infrastructure, and network layers. The new Agent Trace view provides transparency into the investigation process, supporting internal reviews and building confidence in autonomous investigations. Additionally, Bits AI SRE supports direct triage actions within chat interfaces, reducing context switching and speeding up response times. The agent can also initiate automated remediation within the Datadog platform and integrate with third-party tools, helping teams accelerate incident response and enhance system reliability.
Mar 05, 2026 1,138 words in the original blog post.
Datadog's Resource Catalog has introduced new enhancements to address the complexities of managing modern cloud environments, which often involve thousands of resources across multiple providers. It provides a centralized inventory of cloud infrastructure, supporting more than 1,000 resource types, allowing organizations to effectively manage compliance, security, and cost optimization. A key feature is the natural language querying (NLQ) capability, which enables users to ask questions in plain English and have them automatically translated into structured queries. This facilitates easier exploration and management of cloud resources without the need for deep technical knowledge or provider-specific syntax. The Resource Catalog integrates data from various cloud providers, including AWS, Azure, Google Cloud, and Oracle Cloud, and supports a wide range of resources such as Amazon Route 53 and AWS Transit Gateways. By allowing users to save and share useful views, it also enhances collaboration and supports repeat workflows for tasks like compliance checks and cost reviews, making cloud infrastructure more accessible to nontechnical team members.
Mar 04, 2026 746 words in the original blog post.
Reilly Wood discusses the evolution and design improvements of Datadog's Model Context Protocol (MCP) server, which is tailored for AI agents to enhance observability. Initially, the server was a simple API wrapper, but real-world usage revealed inefficiencies, such as context window overloads and resource mismanagement. To address these, Datadog optimized data formats by using CSV and YAML over JSON, implemented token-aware pagination, and enabled SQL querying for more efficient data handling. These changes reduced context space usage and costs, while improving accuracy. Wood also highlights the importance of guiding agents with specific error messages and accessible documentation, and outlines approaches like flexible tools and layering to manage tool count and complexity. The article contrasts the general-purpose MCP server with specialized agents like Bits AI SRE, emphasizing the trade-offs between flexibility and workflow-specific optimizations. The insights gained from these developments are shaping Datadog's approach to building scalable, agent-friendly systems, with an eye on future advancements in the field.
Mar 04, 2026 1,743 words in the original blog post.
The 2026 State of DevSecOps study, analyzing thousands of applications, uncovers trends and best practices in software security, highlighting prevalent vulnerabilities in deployed services and challenges in keeping libraries updated. The report emphasizes the need to secure GitHub Actions against supply chain attacks by pinning actions to specific commit SHAs and using tools like Datadog's Security features for CI/CD pipelines. It also advocates for using DORA metrics to enhance security by aligning deployment velocity with security goals, thereby reducing exposure to vulnerabilities. While upgrading software quickly is important, the study warns against automatic upgrades without safeguards due to risks demonstrated by recent supply chain incidents involving malicious packages. The use of tools like GuardDog and Supply-Chain Firewall is recommended for identifying and blocking malicious dependencies. Additionally, the study highlights the risk of name-confusion attacks on Amazon Machine Images (AMIs) and suggests using AWS's "Allowed AMIs" feature for security. The report calls for improved prioritization of vulnerabilities by using application and vulnerability context, as only a fraction of "critical" vulnerabilities are genuinely critical when evaluated in context. Datadog's updated severity scoring model incorporates runtime context and exploit intelligence to refine vulnerability prioritization, helping organizations focus on exploitable threats. Overall, the study emphasizes the integration of security and observability through platforms like Datadog to drive DevSecOps adoption and security improvements.
Mar 04, 2026 1,836 words in the original blog post.
Organizations are increasingly adopting multi-cloud architectures for reasons such as compliance, business strategy, and resilience, and with the addition of Oracle Cloud Infrastructure (OCI) support, Datadog Cloud Security now offers enhanced visibility and actionable context across these environments. This integration allows security teams to utilize familiar Datadog capabilities, already trusted for securing AWS, Microsoft Azure, and Google Cloud, within a single platform. By centralizing security findings from OCI alongside other cloud providers, Datadog eliminates visibility gaps and reduces tool sprawl, ensuring that critical operational insights are not siloed. Security findings are integrated directly into existing workflows, making them accessible to DevOps and engineering teams for immediate remediation, with detailed resource information and severity of risks provided. Additionally, the platform offers bidirectional Jira syncing to keep findings and tickets aligned, facilitating faster response times and reducing manual follow-up. This unified approach aids security teams in assessing risks consistently across major cloud providers, prioritizing findings with appropriate context, and accelerating response times to resolve security issues before they escalate.
Mar 03, 2026 540 words in the original blog post.
Datadog Synthetic Monitoring provides a comprehensive approach to troubleshooting digital experiences by validating key workflows such as authentication and checkout through scheduled API and browser tests, correlating failures with logs, traces, and backend telemetry data. It introduces new features like Test Suites and AI Failure Summaries to help teams organize tests into meaningful collections and quickly understand the context of failed runs. These capabilities transform Synthetic Monitoring from a collection of individual tests into a structured method for identifying and resolving issues across the entire user journey, reducing alert fatigue and improving response efficiency. By centralizing test results and offering targeted alerts, Datadog enables teams to efficiently trace failures back to their origin, ensuring a seamless user experience.
Mar 03, 2026 1,120 words in the original blog post.