November 2024 Summaries

Datadog has developed Orchestrion, a tool that processes Go source code at compilation time and automatically inserts instrumentation to produce Datadog APM traces. This simplifies the tracing setup and eliminates the need for manual instrumentation in Go applications. Orchestrion interfaces with the standard Go toolchain to inspect and modify the source code as it's being sent to the compiler, allowing unrestricted access to all behavior of the application while preventing errors that could result from direct modification of the compiled binary. The approach is based on compile-time instrumentation, which was chosen over alternative techniques such as binary patching and eBPF due to its safety, reliability, and data quality benefits, as well as its level of automation and performance overhead. Orchestrion offers capabilities not available in eBPF, including blocking function calls for security protection. The tool simplifies monitoring by instrumenting Go applications at build time, enabling quick setup with Datadog APM.

Datadog has introduced the capability to ingest historical metric values for custom metrics, allowing users to submit data with timestamps older than one hour, which can be crucial for long-term data analysis and various use cases such as outage recovery and data science. This feature enables businesses to backfill data during network outages, correct erroneous metrics, and ensure comprehensive monitoring without gaps. Users can configure historical metrics ingestion either for individual metrics or in bulk for multiple metrics sharing a common namespace. The Metrics Summary page in Datadog now includes tools to track and manage historical metrics usage, including a facet box for filtering metrics by their ingestion status and a dedicated metric for monitoring historical metrics configuration. This enhancement supports maintaining compliance with real-time analytics requirements by alerting users to delays in metric emissions and facilitating prompt troubleshooting and remediation. Datadog's historical metrics ingestion feature is accessible to all users, with options to explore further through documentation or a 14-day free trial for new users.

Amber Tunnell and Kevin Gallardo from Datadog introduced Workflow Automation and App Builder, which enable users to act on insights directly within the platform, saving time during critical moments. The new AI assistant for Workflow Automation and App Builder uses generative AI to automate much of the code-writing process, making it easier for users without coding knowledge to build workflows and apps. Users can create custom JavaScript functions and troubleshoot code with the AI assistant's guidance. The tools allow users to add remediation actions such as restarting a VM or toggling feature flags, and enable self-service workflows that can be triggered by team members.

The Open Cybersecurity Schema Framework (OCSF) is an open-source, vendor-neutral schema designed to standardize event formats for security data. It establishes a common taxonomy, simplifying the correlation of Tactics, Techniques, and Procedures (TTPs) and enabling modular schemas. The OCSF model consists of Data Types, Attributes, and Arrays, Event Categories and Classes, and Profiles and Extensions. Datadog Observability Pipelines enables users to easily remap any log from any vendor or source to OCSF format, standardizing security data and supporting taxonomy requirements and security strategies without vendor lock-in. This feature is available in Observability Pipelines with no subscription to Datadog Log Management or Datadog Cloud SIEM.

Datadog's vLLM (Virtual Large Language Model) integration is a high-performance serving framework that optimizes token generation and resource management for large language models, enabling low-latency, scalable performance for AI-driven applications such as chatbots, virtual assistants, and recommendation systems. The integration provides comprehensive visibility into the performance and resource usage of LLM workloads, allowing organizations to monitor key performance indicators like response times, throughput, and resource consumption in real-time. By collecting these metrics, Datadog enables organizations to quickly identify issues and optimize infrastructure usage for cost efficiency, detect critical issues before they impact production, and rightsize their infrastructure to balance performance with cost-efficiency. With the integration's out-of-the-box dashboard, users can seamlessly begin monitoring their LLM workloads in Datadog and gain end-to-end visibility into how efficiently their LLM models are processing requests.

Datadog APM has enhanced its distributed tracing for serverless applications to provide visibility into additional serverless patterns. This includes Span Auto-linking, which automatically identifies traces related to S3 Event Notifications or DynamoDB stream events and connects them to provide end-to-end visibility. Additionally, Datadog now supports enhanced AWS Step Functions trace propagation, allowing users to link up- and downstream Lambda function spans to their state machines and visualize the full execution path of a request in a flame graph or waterfall view. These enhancements aim to provide deeper insights into application performance and enable faster troubleshooting of issues in serverless applications.

PWAs are modern frontend architectures designed to provide similar user experiences to native iOS, Android, or platform-specific apps. They are built using common web technologies like HTML, CSS, and JavaScript and are intended to run in browsers, be accessed from the web, and installed on users' devices for offline access. As PWAs grow in popularity, they present unique monitoring challenges, including cross-browser testing, service worker performance, and offline experience testing. To address these challenges, developers can use progressive enhancement methodology, feature detection, error tracking, RUM tools, and synthetic testing to monitor PWA compatibility, service worker activity, and cache performance. Additionally, automated testing in CI/CD pipelines can help ensure the reliability and performance of PWAs by detecting regressions and flaky tests.

The most common cloud security incidents are the result of compromised credentials for either human or non-human identities. Microsoft 365 is a popular target due to its numerous integrations with other platforms and services, making it a central point of access for sensitive data. Attackers often use phishing campaigns to gain initial access, and once they have control, they may manipulate settings, download data, or modify permissions to escalate their privileges. Understanding the various stages of an attack on Microsoft 365 services is crucial to detect malicious activity, such as increased login attempts, atypical IP addresses, changes to account passwords, and suspicious inbox rules. Datadog Cloud SIEM provides a Microsoft 365 content pack to simplify monitoring of suspicious behavior captured in logs, offering comprehensive visibility into user activity and interactions with important resources and data.

We analyzed the design and testing of a distributed message queuing system called Courier, developed by Datadog. The team aimed to tackle challenges in analyzing complex systems by applying formal modeling and lightweight simulations techniques, which they used to verify the correctness properties of Courier before its implementation. Formal modeling involved using TLA+ (Temporal Logic with Allen's Intervals), a high-level specification language that allows for precise definition of system specifications, capturing both structure and behavior, and verifying system properties against those definitions. Lightweight simulations were used to describe statistical properties under various real-world conditions and workloads, providing insights into performance characteristics. The team also applied chaos testing to validate the system's robustness in failure scenarios. After implementing Courier, they performed performance testing to optimize its design, including adding a sequencer process to order messages for delivery. The TLA+ model was updated to reflect this change, but it still did not guarantee perfect alignment with the implementation due to subtle differences. Deterministic simulations, which involve executing an actual implementation of a system in tightly controlled environments, are considered for future use to verify every release and cover a large number of test cases.

Datadog` applies the principles of `AIOps` (Artificial Intelligence for Operations) to its monitoring solutions, enabling proactive detection of issues across entire technology stacks. This approach combines big data and machine learning to automate IT operations processes, such as event correlation, anomaly detection, and causality determination. The company's AIOps-powered products help users proactively detect anomalies early on, reducing time spent investigating and resolving issues, consolidating related alerts, building automated troubleshooting workflows, and more. Specifically, `Datadog` focuses on proactive anomaly detection, using machine learning algorithms to scale anomaly detection effectively. These algorithms, including `Basic`, `Agile`, and `Robust`, enable users to detect anomalies early, before they become full-blown incidents. The `Basic` algorithm is suitable for unpredictable metrics, while the `Agile` algorithm is better suited for metrics with seasonal patterns. The `Robust` algorithm is designed for metrics with stable, recurring seasonal patterns. Additionally, `Watchdog`, an AI-powered engine, uses these algorithms to automatically flag anomalies and outliers, forecast potential bottlenecks, conduct automated business impact analysis and root cause analysis (RCA), and detect faulty code deployments. Users can customize anomaly detection, outlier detection, and forecasting alerts with `Datadog` monitors, configuring custom alerts based on their chosen metrics. By leveraging these features, users can proactively address issues before they become incidents, prevent unplanned downtime, and protect their environment.

AWS Lambda supports a wide range of programming languages through serverless functions, but when a runtime is deprecated, developers must adapt to ensure their applications continue to function. Options include packaging functions as container images, migrating to custom runtimes, or rewriting code in a supported language. In practice, many Datadog customers have opted for container images, which offer portability and reduced operational overhead compared to custom runtimes or code rewriting. However, container images pose challenges in maintaining and inspecting for vulnerabilities. Datadog aids in managing this transition by identifying Lambda functions using deprecated runtimes, providing warnings and contextual information to facilitate decision-making on runtime updates.

The Windows Blue Screen of Death (BSOD) is triggered when the operating system detects a critical issue that compromises system stability, causing it to shut down immediately and restart. These errors are often caused by problematic or incompatible device drivers, including updates that introduce bugs or conflicts within the system's kernel. To quickly resolve BSOD errors, tools such as Datadog's Windows Crash Detection integration can help detect them and provide context for troubleshooting. This integration generates an event via the Agent each time it detects a Windows system crash on startup, providing information such as the time of the crash, the offending module responsible for the crash, and a bugcheck code. By analyzing this data, organizations can identify potential causes, roll back problematic driver updates, and take corrective action to prevent further disruptions. Additionally, Datadog's integration offers additional visibility into Windows Kernel memory usage, allowing for more granular analysis of system failures.

The text provides a comprehensive guide on managing AWS Identity and Access Management (IAM) policies effectively and securely. It discusses the challenges of scaling IAM policies without compromising security and highlights the importance of granting least-privilege permissions to avoid excessively permissive or restrictive access. The document outlines various types of IAM policies, including identity-based, resource-based, Service Control Policies (SCPs), and permissions boundaries, explaining their roles in managing access. It details the key elements of IAM policies, such as Effect, Action, Resource, and Condition, and explains the evaluation logic, emphasizing the precedence of explicit Deny over Allow and the concept of effective permissions. The text also explores tools like the AWS Policy Generator, IAM policy simulator, and IAM Access Analyzer to create, test, and validate policies. Additionally, it addresses the use of wildcards and Attribute-Based Access Control (ABAC) for scalability, the pros and cons of inline and managed policies, and the benefits of posture management solutions like Datadog for maintaining security across large environments.

Addie Beach and Shah Ahmed discuss the challenges of parsing metadata and user interaction information to understand user behavior, particularly when combining various analytical lenses. They advocate for a subtractive model that involves ingesting all usage data upfront and filtering it afterwards, enabling flexibility in adapting to changing project goals or realizing the need for new data. Organizing usage data into a hierarchical taxonomy helps analysts link and query data more easily, while verifying and normalizing data ensures accuracy and consistency. To effectively query product data, identifying useful events and parameters is crucial, considering both user interactions and metadata. The authors suggest using Datadog Product Analytics to visualize and query product usage data, providing various visualizations and enabling natural language queries. By collecting, normalizing, and verifying data, analysts can build accurate insights that ground UX decisions in concrete trends.

The text explores the vulnerabilities of large language model (LLM) applications, particularly those utilizing chain-based and agentic architectures, to prompt injection attacks that can lead to sensitive data exposure. These attacks, which can take the form of direct or indirect prompt injections, exploit the model's access to privileged data and resources, making them attractive targets for attackers. Techniques such as jailbreaking are used to trick LLMs into ignoring moderation guardrails, while indirect injections may utilize hidden instructions in linked assets. To mitigate these threats, the text suggests implementing data sanitization, monitoring for injection attempts, and employing protective measures like prompt guardrailing and least privilege principles. Additionally, tools such as Datadog LLM Observability are recommended for tracking and analyzing potential attacks to enhance security and prevent data breaches effectively.

The use of custom resource definitions (CRDs) in Kubernetes production environments enables users to tailor resources to their specific applications or infrastructure needs, automate processes through operators, simplify the management of complex applications, and integrate with non-native applications such as Kafka and Elasticsearch. Datadog encourages the use of CRDs via the Datadog Operator, which provides benefits including ease of use, clarity of ownership, and more seamless workflows. Monitoring CRDs for resource management, availability, autoscaling configuration, and state validation is crucial to ensure stability and performance. The Datadog Operator includes CRDs that help manage Datadog resources such as dashboards, monitors, metrics, SLOs, and more. CRDs can impact the stability and performance of the entire Kubernetes cluster and any applications using them, so it's essential to monitor them with tools like Datadog Container Monitoring.

The text discusses various tools and solutions in the Kubernetes ecosystem for scaling applications and infrastructure. It highlights the importance of careful configuration and selection of the right tool to achieve prudential scaling, minimizing waste and ensuring optimal cost efficiency and performance optimization. The key points covered include: using Horizontal Pod Autoscalers (HPA) for workload scaling, Vertical Pod Autoscalers (VPA) for dynamic resource adjustments, combining HPA and VPA for comprehensive solution, Datadog Kubernetes Autoscaling for right-sizing workloads, Cluster Autoscaler for scaling nodes, Karpenter for intelligent node provisioning, and the Watermark Pod Autoscaler for handling variable traffic. The text emphasizes the need to choose the right tool based on specific application requirements and workload patterns.

The text discusses the challenges and strategies involved in managing Kubernetes workloads, focusing particularly on the issues of overprovisioning and the importance of autoscaling to optimize cloud costs. It introduces Datadog’s Watermark Pod Autoscaler (WPA) as an enhancement to the traditional Horizontal Pod Autoscaler (HPA), offering more flexibility with scaling decisions through high and low watermarks. The text highlights the significance of proper vertical and horizontal provisioning before implementing autoscaling to achieve high resource utilization and cost savings. It emphasizes the necessity of choosing the right metrics for scaling, such as using request queue length over CPU utilization for more proactive scaling decisions. The article underscores that successful autoscaling requires continuous monitoring and tuning of parameters like scaling velocity and cooldown periods to maintain a balance between performance and cost efficiency. Finally, it suggests that autoscaling is not a "set it and forget it" solution, necessitating regular adjustments to adapt to evolving workloads and traffic patterns, and promotes Datadog's fully managed Kubernetes Autoscaling service for simplifying scaling strategies.

Datadog's Kubernetes Active Remediation feature aims to assist teams in analyzing and responding to dense telemetry data related to Kubernetes orchestration layer. This feature provides clear contextual recommendations and next steps for application teams to confidently launch investigations, gather context, and determine the root causes of Kubernetes issues. By using this new feature, teams can get clarity about how to proceed with Kubernetes troubleshooting, gain visibility into related issues, and take immediate action to resolve them. The feature offers guided actions to efficiently stabilize the environment by providing investigation areas, associated next steps, and an opportunity to perform those steps within the same context. This solution expands organizations' ability to address business-critical Kubernetes issues, enabling engineering teams to quickly understand Kubernetes alerts, engage in guided root cause analysis, and take immediate action.

Datadog Cloud SIEM provides security teams with enhanced insights into their performance by delivering comprehensive security operational metrics such as mean time to detect (MTTD), mean time to acknowledge (MTTA), and mean time to resolve (MTTR). These metrics offer clear visibility into a team's effectiveness, enabling them to streamline processes, address threats promptly, and continuously optimize operations. Security teams can visualize these metrics in dashboards, explore and track progress using tags and custom dashboards, and receive weekly digest reports that provide a consolidated overview of key metrics and operational insights. By leveraging security operational metrics, teams can respond effectively to evolving threats, maintain a stronger security posture, and ensure their cloud environments remain secure and resilient.

Datadog's integration with Cisco Umbrella provides a platform for monitoring and maintaining DNS-layer security across networks. The integration enables the collection, processing, and visualization of DNS and proxy logs, allowing security teams to detect behavior like DNS hijacking, spoofing, and other attacks. Datadog Cloud SIEM automatically detects threats by continuously scanning DNS logs and generating security signals if it detects potentially malicious activity. Security teams can create custom detection rules or use out-of-the-box rules developed by the dedicated security research team to automate alerts and remediation for suspicious activities. The integration also provides two customizable out-of-the-box dashboards that deliver a high-level view of DNS and proxied network activity, offering essential insights into the health and security of the environment.

Azure AI Search is Microsoft Azure's managed search service that includes AI-powered features and is highly interoperable with other Azure services. Datadog has integrated with Azure AI Search, allowing users to track the performance and usage of their AI Search services using Datadog dashboards and monitors. With this integration, users can monitor query performance, track outages, and analyze query throughput and latency for each search service in their environment. The integration also includes an out-of-the-box monitor that provides timely notifications when query throttling reaches a problematic threshold, helping engineers scale out their search services to prevent dropped queries and mitigate customer impact. Users can create custom monitors to track other Azure AI Search metrics, such as search latency, to quickly spot issues and optimize their services. The integration is now generally available for all Datadog customers, allowing them to monitor their Azure AI Search services alongside the rest of their Azure infrastructure and application stack.

As organizations scale up the use of Kubernetes, they often encounter increased complexity in maintaining workloads and services, primarily due to siloed expertise in DevOps teams and the overwhelming influx of telemetry data and alerts. Datadog's Bits AI Kubernetes Remediation, now generally available, offers a solution by providing AI-driven root cause analysis and automated workflows to address common infrastructure issues before they escalate into significant incidents. This tool helps teams troubleshoot more efficiently by aggregating context-rich information, offering remediation guidance, and allowing direct action on recommendations to reduce the mean time to resolution. By integrating with GitHub, the tool facilitates seamless updates to configuration files, enabling users to resolve issues quickly. As a result, teams of all experience levels can enhance their response times to Kubernetes incidents, decreasing the number of escalations and improving system uptime. Bits AI Kubernetes Remediation is part of Datadog Container Monitoring, and users can sign up for a free trial to explore its capabilities.

Datadog Cloud Cost Management (CCM) is a tool that provides real-time insights into cloud expenditures, enabling government organizations to understand their total cost of ownership, optimize resources, and identify cost changes. By deploying CCM in the Datadog for Government region, public sector organizations can safely rely on it to manage their cloud costs effectively. With CCM, FinOps personnel and platform engineers can correlate costs with performance and usage data to find opportunities for optimization, such as unused resources or unexpected spikes in usage. The tool also provides a unified view of cloud infrastructure usage, breaking down costs by facets including service, region, and host. Additionally, CCM enables agencies to set up custom monitors and dashboards using CCM metrics, which can be easily shared across an organization. By integrating CCM with other Datadog tools, such as Infrastructure Monitoring, APM, Service Management, and Incident Management, organizations can bring together context-rich observability insights and cost data within a unified platform, providing a holistic view of application performance and cost.

The use of Retrieval-Augmented Generation (RAG) in Large Language Models (LLMs) enhances the accuracy and relevance of responses by retrieving information from external datasets, such as websites, financial databases, or company policy guidelines. However, integrating RAG into LLM applications introduces complexity in managing latency, ensuring the relevance of retrieved data, and maintaining model accuracy. To mitigate these challenges, developers can take steps to reduce latency, implement hybrid search to limit irrelevant responses, use vector databases to exclude outdated information, and scan prompts and responses to prevent accidental exposure of sensitive data. By implementing effective metadata filtering and regular updates, RAG systems can deliver contextually relevant information at scale while maintaining trust with users. Additionally, developers can utilize tools like Datadog LLM Observability to identify and troubleshoot issues in their RAG-based LLMs.

Datadog has released several new features, including unified Error Tracking, which allows users to view and search for errors across RUM, APM, and Logs in one place. Security Operational Metrics is another new feature that helps teams assess their effectiveness at responding to threats by capturing mean time to detect, acknowledge, and resolve. Additionally, Datadog has introduced a new Serverless feature that enables users to retry or redrive failed AWS Step Functions executions directly from the platform. The company's LLM Observability has also been integrated with Google Gemini, allowing for automatic capture of LLM requests and end-to-end tracing capabilities. Other updates include new App Builder blueprints, optimization of Ruby garbage collection activity, and improved management of RUM apps with Service Catalog. These features are available to all customers today, and users can learn more by visiting the release notes or signing up for a 14-day free trial.

Datadog has introduced Exploit Prevention, a new security feature in its Application Security Management (ASM) that uses runtime application self-protection (RASP) to protect systems from runtime threats before code execution. This feature analyzes the input/output entry points in an application's code and uses request data as context to determine if vulnerabilities are being exploited. It can detect and block attacks targeting vulnerabilities such as local file inclusion, server-side request forgery, SQL injection, and command injection, providing deep visibility into application behavior and blocking malicious requests before they can be executed. Exploit Prevention is integrated with Datadog's In-App web application firewall (WAF) and monitoring tools, enabling organizations to analyze clusters of attacks, correlate them with specific endpoints in their application, and provide business logic security rules and recommendations. The feature is available in .NET, Go, Java, NodeJS, and Python tracers for all Datadog customers using ASM, and can be deployed and enabled through the company's documentation.

Scorecards, a feature of the Datadog Software Catalog, enables organizations to gauge the performance and observability of their services, guided by industry standards as well as custom rules, providing actionable feedback to service owners on an ongoing basis. Before implementing Scorecards, Datadog relied on a manual production-readiness review process that was not scalable for growing teams and services. By leveraging Scorecards, Datadog has established and maintained important standards for security, reliability, and performance at scale, eliminating knowledge silos and providing robust guidelines throughout the software development life cycle. The rollout of Scorecards has been progressive, with a focus on promoting a culture shift and communicating the "why" behind each rule and its outcomes. Key metrics have shown that Scorecards are helping to reduce unexpected setbacks, promote best practices early in the development process, and improve the overall efficiency of service launch timelines.

The Datadog Architecture Center is a collection of detailed guides and reference diagrams that help teams efficiently deploy observability solutions to their cloud environment. It provides customizable blueprints, descriptions of benefits and challenges, reference architecture diagrams, use cases, and step-by-step explanations for deploying Datadog to gain visibility into various components of the stack. The center covers log management, infrastructure monitoring, observable cloud environments, and more, offering a roadmap for meeting both immediate and long-term observability needs and avoiding common pitfalls in complex deployments. It provides a frictionless path to deploying enterprise-ready observability with minimal expertise required, allowing teams to tap into Datadog's deep expertise as their environment expands and needs evolve.

Datadog Database Monitoring (DBM) now supports MongoDB, enhancing the existing integrations with MongoDB and MongoDB Atlas to provide comprehensive visibility into database performance, critical for managing modern applications relying on these NoSQL databases. This support includes insights into slow operations, replication state changes, and detailed operation samples, which help diagnose and resolve performance bottlenecks. DBM facilitates high availability by tracking replication health and node status, and it optimizes query performance using explain plans and detailed metrics. The tool offers visualizations of cluster health, detects slow queries, and presents execution plans to identify inefficiencies. By unifying MongoDB monitoring with infrastructure metrics, DBM enables collaborative troubleshooting across teams, ensuring database and application stability. Overall, DBM provides a full-stack view that simplifies monitoring efforts, allowing organizations to manage multiple database types within a single platform.

** The authors of a test impact analysis library aimed to speed up the testing process by selectively running tests based on their impact. They considered several approaches, including using existing code coverage tools and TracePoint, but found them to be either incompatible or too performance-intensive. Instead, they created their own solution that uses interpreter events to track which source files are executed during test runs. This approach allows for a highly optimized solution with a median performance overhead of 25% (up to 70%), making it suitable for use in continuous integration pipelines. The library also addresses limitations such as "code-less classes" and external dependencies, providing workarounds for these cases. By collecting test impact data on every test run, the tool ensures that the correct set of tests is skipped each time, creating a fully deterministic experience.

Datadog's Service Management offerings aim to simplify issue detection and resolution by providing a unified platform that integrates monitoring, DevOps, IT operations, and app development tools. This allows for faster incident remediation, improved reliability, and streamlined engineering processes. By leveraging features such as Incident, Case, and Event Management, users can quickly detect issues, triage tasks, and track progress without leaving the platform. Additionally, Datadog's automation capabilities help streamline workflows, automate critical actions, and reduce alert fatigue, enabling teams to focus on more complex problem-solving. The platform also provides a centralized hub for managing service data, reducing tool sprawl, and improving communication among stakeholders.