August 2025 Summaries

In April 2018, the IRS faced a significant outage on Tax Day due to a hardware failure in a Tier 1 storage array, disrupting 59 production systems, including the Modernized e-File platform, which delayed tax filings and payments and resulted in a 24-hour extension of the tax deadline. The incident highlighted the complexities of maintaining government IT systems, which often rely on outdated infrastructure and lack centralized monitoring, leading to delayed responses and widespread impact. Following the outage, the IRS invested in additional storage to increase failover capacity, but the event underscored the need for better observability and automation in government IT environments to prevent similar occurrences. Datadog suggests that its unified observability platform could enhance resilience by providing real-time infrastructure monitoring, correlating alerts, automating response workflows, and offering proactive testing, all of which could help government agencies manage their complex systems more effectively and reduce downtime during critical events.

As organizations increasingly adopt multi-cloud strategies, managing and reducing cloud spending has become complex, prompting the need for efficient solutions like Datadog's Cost Recommendations. This feature provides actionable insights and customized recommendations based on billing and observability data to help organizations optimize costs across AWS, Azure, and Google Cloud environments. By highlighting opportunities to eliminate waste, modernize outdated technologies, and rightsize resources, Cost Recommendations helps align cloud spending with business priorities. The tool offers built-in workflows and integration with Jira, enabling teams to implement cost-saving actions directly from Datadog's interface without manual scripting. Customizable settings allow organizations to tailor recommendations to their specific needs, ensuring they remain relevant and effective. The centralized platform allows users to manage and optimize cloud resources, leading to significant cost savings while maintaining performance.

The text discusses the challenges and solutions in maintaining code quality when using AI-assisted tools in software development, particularly within microservice architectures. It highlights the tension between increasing developer velocity and ensuring code quality, emphasizing that poor code quality can lead to significant issues in performance, reliability, and security. To address this, the text suggests using automated tools to shift-left quality checks, reducing reliance on manual reviews. It illustrates these points through a case study of a path tracer application developed using AI, identifying common issues such as inefficient code and unused functions, and explaining how static and dynamic analysis tools, along with AI-assisted reviews, can help detect and correct these problems. The text underscores the importance of integrating quality control practices into AI-assisted development workflows to prevent costly errors and improve code standards.

The article examines the issue of hallucinations in large language models (LLMs), which are instances where these AI systems fabricate information, leading to significant challenges in deploying them in sensitive applications. Datadog has developed a real-time hallucination detection feature, especially for retrieval-augmented generation (RAG) scenarios, focusing on faithfulness—ensuring LLM-generated answers align with a given context. The company employs black-box detection methods, particularly LLM-as-a-judge approaches, to evaluate the accuracy of LLM outputs without accessing the model's internal workings. This involves a structured prompting strategy that breaks down tasks into smaller guided steps, improving accuracy by leveraging the LLM's strengths in guided summarization. Datadog's technique has shown promising results, particularly in challenging human-labeled benchmarks, and highlights the significant impact of prompt design over just model architecture in detecting hallucinations effectively. The company continues to refine its approach and invites interested individuals to join their team.

Kubernetes v1.34, set for release on August 27, 2025, introduces significant enhancements focused on scheduler visibility, life cycle observability, and resource management. The update aims to address challenges such as opaque scheduling decisions, inefficient resource use, and limited insight into pod placement and container shutdowns. Key features include asynchronous scheduler API calls to reduce bottlenecks, the NominatedNodeName field for improved pod placement prediction, and new container stop signals for better shutdown clarity. The release also advances the Dynamic Resource Allocation (DRA) framework, allowing for more consistent management and monitoring of specialized hardware like GPUs and FPGAs, and introduces device binding conditions and consumable capacity to enhance visibility and resource allocation efficiency. These improvements aim to provide faster insights into system health and performance, thereby supporting more efficient and reliable Kubernetes operations.

The text discusses the challenges and strategies associated with scaling observability systems in growing organizations, emphasizing the importance of structured and intentional design for dashboards and monitors. As infrastructure and teams expand, the initial simplicity of observability setups can become fragmented, leading to redundant dashboards, noisy alerts, and unclear ownership, which can overwhelm teams and degrade incident response. To manage this complexity, the text suggests shifting from ad hoc setups to purpose-driven, scalable systems with consistent naming conventions, effective tagging, and clear ownership. It highlights strategies like dynamic thresholding, ownership-based alert routing, and continuous refinement of monitors to maintain an effective signal-to-noise ratio. Additionally, the text underscores the role of well-designed dashboards in supporting real-time decision-making and cross-team alignment, advocating for shared interfaces, consistent naming conventions, and reusable templates to facilitate scalable and effective observability practices. Tools like Datadog are recommended for implementing these strategies, offering features that support scalable monitor and dashboard management, ultimately leading to improved response times and greater confidence in system performance.

The text introduces Datadog Network Path as a solution to the challenges of identifying network slowdowns in modern, distributed infrastructures that span on-premises data centers and multi-cloud environments. Traditional troubleshooting methods are often inadequate, as they require piecing together fragmented data and guessing the origins of latency or packet loss. Network Path addresses these issues by providing a visual representation of packet journeys, hop by hop, from source to destination, which helps in pinpointing network problems swiftly. It integrates with Datadog's other monitoring tools to offer comprehensive insights into network and application performance, enabling quicker resolution of issues by correlating network behavior with application anomalies. Additionally, it aids corporate IT teams in improving end-user experiences by identifying slowdowns across networks, ISPs, and cloud infrastructures, thereby enabling targeted solutions and enhancing overall network reliability.

The rapid evolution of modern application development is marked by the adoption of serverless containers and microservices, with Azure Container Apps playing a key role by allowing developers to deploy containerized applications without managing infrastructure. However, this infrastructure abstraction can make it difficult to gain insights into application behaviors in production. To address this, Datadog has integrated with Azure to offer visibility into critical metrics emitted by Azure Container Apps, and now provides an easy solution for deeper insight through a lightweight sidecar container that operates alongside the app container to collect metrics, logs, and traces without affecting the core application logic. This innovative approach simplifies the traditionally complex process of application instrumentation, allowing developers to set up the sidecar directly within the Azure portal or using YAML and Terraform templates, facilitating a comprehensive all-in-one monitoring experience. Additionally, Azure Container Apps supports a managed OpenTelemetry agent, offering an alternative for sending telemetry data to multiple endpoints, thereby enhancing observability and enabling faster troubleshooting and reduced downtime. With Datadog's sidecar, developers can achieve deep observability for both single container apps and distributed microservices, and those new to Datadog can explore these capabilities with a 14-day free trial.

As organizations increasingly utilize large language models (LLMs), LiteLLM has emerged as a tool to streamline access to various LLM providers and models, albeit with challenges in understanding model selection and performance due to its abstraction layer. To tackle these issues, Datadog has introduced an Agent integration and SDK with LiteLLM, providing comprehensive observability across LLM workflows. This integration allows users to monitor, troubleshoot, and optimize applications by tracing every LLM request, offering insights into model performance, token usage, latency, and cost. The Datadog Agent further enhances this by capturing high-level metrics about the LiteLLM proxy service, aiding in performance trend tracking and ensuring reliability. Together, these tools offer full-stack visibility, empowering teams to make informed decisions about host sizing, model selection, and resource allocation, ultimately supporting data-driven optimization of LLM-powered applications.

As organizations expand, managing observability resources becomes crucial, with an increase in dashboards, tags, API keys, and costs. Observability governance provides a framework for handling these challenges by managing data access, attributing costs, enforcing policies, and ensuring effective product usage. A well-structured governance strategy includes best practices such as tagging, access and key management, cost control, and onboarding alignment. Datadog offers tools to implement these strategies, helping organizations manage permissions, control costs, and align teams through features like tagging, usage attribution, and workflow automation. The Datadog Governance Console aids in visualizing and addressing compliance gaps, ensuring that observability data is managed securely and efficiently, allowing for scalable and cost-effective monitoring efforts.

Datadog's efforts to optimize cloud costs led to significant savings through a collaborative initiative involving their FinOps and engineering teams. By utilizing the Datadog Cloud Cost Management (CCM) tool, the FinOps team initially identified storage inefficiencies in Amazon S3 usage and recommended migrating to a more cost-effective storage class. This collaboration sparked further investigation by the engineering team, which uncovered additional savings opportunities related to managing non-current object versions. The implementation of new S3 Lifecycle rules, designed to reduce unnecessary storage costs, resulted in an estimated annual savings of $1.5 million. This initiative not only exemplified the potential for substantial cost reductions but also fostered a culture of cost-consciousness across the organization. By promoting transparency and trust in data, engineering teams were empowered to prioritize cost efficiency alongside performance, reliability, and security. The success of this initiative has encouraged ongoing optimizations and sharing of cost-saving achievements to motivate other teams within Datadog.

Many engineering organizations, including Datadog, are integrating agentic AI-based coding tools and large language models (LLMs) to enhance development velocity, though the transition can be challenging for developers who encounter derivative or faulty solutions. Datadog developers suggest strategies such as implementing a planning phase, improving context understanding, and optimizing model token usage to achieve better results. They emphasize the importance of clearly defining problem statements, execution plans, and constraints to guide AI agents effectively, while also leveraging AI for exploring alternative solutions and conducting cost-benefit analyses. Additionally, connecting AI clients to MCP servers can extend an agent's capabilities by allowing it to access external systems and tools, thereby improving problem-solving efficiency. Datadog continues to explore advancements in AI technology to enhance the output of agentic AI, offering insights and tools through their blog and product offerings.

The rapid integration of generative AI (GenAI) into the software industry has brought new security challenges, particularly with threats targeting AI applications' infrastructure, supply chain, and interfaces. These vulnerabilities are increasingly significant as this technology evolves, with common threats involving infrastructure misconfigurations, such as overprivileged IAM roles, and the lack of adequate controls for authentication and authorization. The article discusses how attackers often exploit these vulnerabilities by using techniques like credential access and discovery to infiltrate AI systems, highlighting examples such as the exploitation of public-facing applications and the mishandling of retrieval-augmented generation (RAG) systems. Moreover, it emphasizes the importance of effective logging and monitoring, as demonstrated by tactics like LLM jacking, which involves unauthorized access to cloud-hosted large language models. The piece underscores the necessity of minimizing risks through strategies that include the use of tools like Datadog Cloud SIEM to detect compromised credentials and abnormal activities, ultimately aiming to mitigate potential threats to AI infrastructure.

The text outlines the challenges and solutions related to managing the costs of foundation models, particularly as AI adoption increases and teams utilize complex models like Anthropic's Claude Opus and Claude Sonnet. As organizations face difficulties in tracking and controlling AI-related expenditures, Datadog's Cloud Cost Management (CCM) offers a solution by integrating with Anthropic's Usage and Cost Admin API to provide detailed insights into Claude usage and costs. This integration helps organizations allocate costs more effectively, allows engineers to manage spending, and optimizes usage by normalizing data in a unified format, thus enhancing visibility and accountability. The system includes features like monitoring, alerting for cost anomalies, and aligning usage with service ownership, all of which facilitate informed decision-making and improved cost governance. Additionally, by correlating Claude usage data with broader infrastructure metrics and application performance, Datadog enables organizations to address inefficiencies and optimize their AI investments seamlessly within existing workflows.

The text explores the vulnerabilities and threats that target AI interfaces, such as chatbots and assistants, within generative AI applications. It highlights the importance of AI interfaces as critical entry points that attackers exploit through tactics like prompt injections, which can manipulate AI models to leak sensitive data or maintain unauthorized access. The text also maps these threats to MITRE's Adversarial Threat Landscape for Artificial Intelligence Systems (ATLAS), discussing how attackers use these methods to extend their influence over AI systems. Furthermore, it emphasizes the need for effective monitoring, detection, and response strategies to mitigate these threats, such as implementing prompt input sanitation, output filtering, and controlling model permissions. The discussion underscores the evolving attack surface that prompts represent and the necessity for robust security measures to protect AI applications and their data from malicious exploitation.

The AI ecosystem's rapid growth brings specific challenges in securing the supply chains that support AI applications, particularly focusing on software and data artifacts like training datasets, pre-trained models, and third-party libraries. The text discusses how attackers exploit these vulnerabilities by targeting AI-specific resources, often employing tactics outlined in MITRE's ATLAS framework. Attackers develop and stage attacks outside organizational visibility, making it difficult to detect early threats. They gain initial access by exploiting supply chain components, sometimes embedding malware in AI models to control affected systems. The need for comprehensive security measures is emphasized, including maintaining visibility into supply chain artifacts and employing vulnerability scanning tools to detect risks. Additionally, the importance of setting guardrails around the access and sharing of AI libraries and data is highlighted to prevent unauthorized use. The text underscores the necessity for organizations to proactively monitor AI artifacts to mitigate the risks posed by potentially vulnerable third-party packages as the AI landscape continues to evolve.

Datadog is hosting its first French-language summit in Paris on September 30, designed to celebrate and engage its community of engineers, developers, and system reliability engineers. The event will include product announcements, customer stories, and a fireside chat with CEO Olivier Pomel. Attendees can participate in hands-on workshops led by Datadog experts, covering topics like OpenTelemetry integration and optimizing application performance. The summit will also feature presentations from customers such as Leboncoin, who will discuss their use of Datadog to enhance service observability. Networking opportunities will be abundant, with interactive sessions and an AWS GameDay event. Preceding the summit, a Partner Day on September 29 will offer workshops and keynotes for Datadog Partner Network members, focusing on LLM observability and business growth strategies.

Windows event logs, crucial for security and compliance in Microsoft-based environments, often present challenges due to their XML format, which is verbose and difficult to process. Many industries, including financial services and logistics, still rely on XML for data exchange, complicating the analysis and storage of these logs. To address these issues, Datadog Observability Pipelines offers a Parse XML processor that converts XML logs into structured JSON, facilitating more efficient analysis and storage. This transformation reduces log volumes, enhances event prioritization, and enables better visibility into security activities. By converting logs to JSON, teams can automate log processing, enrich data with tags and metadata, and integrate insights across various observability and security tools. Additionally, the processor supports remapping logs to an open source schema, aiding in threat detection and metric generation. This capability is particularly beneficial for large systems where XML logs are prevalent, allowing organizations to derive actionable insights and improve their security posture.

Adyen is a comprehensive global payment platform that simplifies transaction processes across various channels, but managing payment workflows can be complex due to multiple steps like authorization and refunds. Datadog's integration with Adyen transforms webhook events and API responses into structured logs, providing merchants with real-time visibility into their payment data. This integration allows for efficient monitoring and troubleshooting of failed payments, detection of fraud signals, and the ability to correlate payment errors with infrastructure performance data. Datadog offers dashboards that visualize transaction, dispute, and payout data, enabling users to filter and inspect logs for insights into payment activities. The integration also includes monitor templates for proactive alerting on issues such as authorization failures and charge-backs, thus helping merchants maintain customer trust by addressing potential problems swiftly.

Datadog faced significant challenges in scaling its real-time data pipeline for its Processes and Containers products due to the increasing demand for live data views, which necessitated handling millions of data points per second. Originally, the system collected data from all hosts in a tenant's infrastructure, leading to inefficiencies and high resource usage. To address this, Datadog refined its approach, focusing on enabling real-time data collection only for actively viewed hosts, thus reducing data traffic by 100x and cutting infrastructure costs by 98%. This optimization not only enhanced the system's efficiency but also improved the user experience by decreasing latency and CPU usage. By adopting a more targeted data collection strategy and leveraging the standard 10-second data intervals for sorting, Datadog was able to maintain system performance while significantly reducing resource consumption. This approach demonstrated the importance of scaling systems based on user needs rather than sheer data volume, leading to substantial performance gains and cost savings.

Serverless platforms like Azure Functions and Azure Container Apps simplify application scaling by eliminating the need to manage infrastructure, but they require careful planning to address challenges like cold starts, unpredictable scaling, and secure data handling. The Azure Well-Architected Framework provides a foundation for building robust serverless applications by focusing on reliability, performance, security, cost optimization, and operational excellence. Datadog supports these pillars by offering insights into serverless environments, helping ensure applications are secure and efficient. Reliability involves managing high availability and function failures, while performance can be improved by minimizing cold starts and optimizing function processes. Security is enhanced through role-based access control, authentication, and encryption, while cost optimization requires selecting the right pricing plans and reducing idle execution. Operational excellence is achieved by integrating observability into applications, and Datadog's tools provide comprehensive monitoring, tracing, and security features to help maintain and improve serverless application performance on Azure.

The text outlines the security challenges and limitations associated with monitoring remote user sessions in Kubernetes environments due to dynamic pod scheduling, which can lead to increased vulnerability when user credentials are compromised. Traditional methods, such as Kubernetes audit logs, provide limited visibility, while capturing standard input of remote sessions often fails to offer comprehensive insights and context needed for effective security investigations. Datadog Workload Protection offers a novel approach by utilizing eBPF to correlate system activity with Kubernetes user identities, enabling enhanced visibility into remote sessions. This approach overcomes challenges associated with traditional monitoring by attributing kernel-level events to the correct users, providing a more robust and actionable security framework. Workload Protection also offers dashboards to visualize trends and detect suspicious activity, facilitating better incident management and risk mitigation in Kubernetes deployments.

Mobile application development poses unique challenges, particularly in testing, due to the diverse range of devices, operating systems, and network types that need to be supported. Developers must ensure a consistent user experience by testing their apps across multiple device models and platforms, which can be both costly and time-consuming. The underlying structure of mobile apps, differing from web apps, necessitates specific testing tools, such as emulators, simulators, and real devices, to evaluate both software and hardware configurations effectively. While virtual devices, like emulators, offer speed and flexibility for early-stage testing, they fall short in replicating real-world conditions, making real devices more reliable for performance and usability testing. Furthermore, mobile app testing involves identifying UI elements using unique locators provided by iOS and Android, such as accessibility IDs, predicates, and resource IDs. Datadog Synthetic Monitoring aids developers by offering real-device testing, self-healing locators, and support for OS-specific locators, streamlining the testing process and ensuring accurate and comprehensive app evaluation.

Asynchronous communication in distributed systems allows services to independently produce, route, or consume events and messages without requiring immediate responses, enhancing scalability and fault tolerance but complicating traceability. Unlike synchronous systems where events follow a predictable sequence, asynchronous systems involve operations that are temporally decoupled, making it difficult to trace the flow and origin of events. Distributed tracing helps bridge this gap by using context propagation, which involves passing metadata like trace IDs across service boundaries to identify and correlate activities. However, asynchronous systems often face challenges in maintaining trace continuity, especially when intermediaries strip trace metadata, requiring manual injection. The choice between parent-child relationships and span-link methods for context propagation impacts the visibility and analysis of operations. Parent-child relationships offer a unified, hierarchical view suitable for tightly controlled environments, while span links provide flexibility in loosely coupled systems but may result in fragmented visibility. Implementing these methods involves choosing appropriate tools and strategies based on the system's architecture and control level, with platforms like Datadog and OpenTelemetry offering support for both methods to improve trace visibility in asynchronous environments.

Datadog IaC Security is designed to mitigate risks associated with the rapid deployment of modern infrastructure by identifying and fixing misconfigurations in cloud-native applications. It integrates directly with Git-based workflows to scan configuration files for security issues, detect policy violations, and prevent insecure defaults, providing real-time feedback to developers without disrupting delivery. The tool offers features such as Quality Gates to block risky code merges, daily full scans for existing misconfigurations, and integration with Jira for streamlined remediation processes. Additionally, it supports various notification channels and includes a dashboard for tracking IaC security posture over time, helping security and platform teams to align their goals with engineering output and maintain compliance.

The text discusses the importance of understanding cloud spending through unit economics, which links each dollar spent to specific business outcomes. By analyzing costs in terms of unit economics, organizations can measure the efficiency of cloud resource usage and make data-driven decisions to optimize spending. This approach helps distinguish between waste and valuable investments, enabling teams from finance, engineering, and product divisions to collaborate more effectively. Unit economics allows organizations to identify valuable activities and calculate the corresponding cloud costs, leading to improved cost forecasting, product pricing strategies, and overall business growth. Adopting unit economics fosters a shared understanding of cloud investments, ensuring that rising costs reflect increased business value rather than inefficiency.

The text explores the critical role of caching in modern software development, particularly within Continuous Integration (CI) and Continuous Deployment (CD) pipelines. It highlights the importance of caching for reducing computational costs, speeding up job runtimes, and minimizing environmental impact. The discussion emphasizes the necessity of effective cache purging strategies to ensure that users receive up-to-date content and avoid issues such as bugs or outdated information. Different caching patterns are examined, including content-based hashing, time-to-live policies, and least recently used eviction strategies, all aimed at maintaining efficient and reliable systems. The text also delves into runtime caching for delivering fast user experiences post-deployment, covering strategies for managing static assets and application configurations through content delivery networks (CDNs). Best practices for cache invalidation are suggested, such as simulating purges before application, monitoring purge failures, and ensuring consistent caching behavior across different environments. Overall, the text underscores the importance of a comprehensive caching strategy to enhance application performance and reliability.

Datadog has developed a sixth-generation real-time timeseries database, designed in Rust, to handle growing data volumes, complex queries, and high-cardinality workloads efficiently. This new system, named Monocle, employs a log-structured merge tree (LSM tree) architecture, enabling high throughput and low latency while utilizing a shard-per-core model to manage data ingestion and storage. The evolution from previous systems like Cassandra, Redis, MDBM, and a Go-based B+ tree to this unified Rust-based engine has enhanced performance, offering a 60x increase in ingestion speed and 5x faster query execution. Key design features include sharded ingestion pipelines, a shared radix-tree buffer for memory-efficient aggregation, and a unified series cache to reduce query latency. The architecture's modular nature allows for reuse across Datadog's infrastructure, promoting consistency and reliability. Future improvements aim at dynamic load balancing and potentially unifying indexing and timeseries storage to further adapt to Datadog's growing demands and customer needs.