February 2024 Summaries

The integration of Google Cloud Armor in Datadog provides insights into malicious attacks aimed at your Google Cloud deployments through visualizations and contextual data. With the out-of-the-box (OOTB) Google Cloud Armor dashboard, users can easily visualize network security patterns, including which firewall policies have been triggered most and which source IP addresses have triggered them. The integration also shares data with the Google Security Command Center, enabling users to consolidate and view a broad range of security information about their Google Cloud environment in a single dashboard. This helps improve response to network attacks by providing easy access to security data from Google Cloud Armor within the Datadog platform, allowing users to supplement that data with contextual information and functionality from other Datadog features and products.

Google Cloud Armor is a network security service that helps protect applications from attacks by filtering incoming traffic and automatically mitigating common threats. The Datadog Google Cloud Armor integration provides insights into these attempts, enabling users to visualize network patterns, detect malicious activity, and improve their response to network attacks. With the integration, users can view detailed dashboards, monitor key findings, and enhance a native feature related to Google Security Command Center. By combining this data with additional Datadog features, such as Log Management and monitors, users can better investigate, analyze, detect, and remediate threats, ultimately strengthening their application's security profile.

Stream-based log management is gaining popularity due to its ability to support real-time troubleshooting and minimize data storage costs. Datadog's new Live Search feature allows users to search across all ingested logs for the past 15 minutes, providing full visibility into their logs without needing to retain them long-term. This feature can be used to correlate directly between live traces and logs, verify new deployments, streamline CI/CD troubleshooting, and gain insights during peak traffic times or major events. Datadog's Log Management is designed to handle data at petabyte scale, enabling users to view and query all ingested logs for troubleshooting and analysis without any pressure to retain them.

Watchdog Insights and Alerts are now available for Datadog Live Processes to help identify anomalous workload behavior in processes such as memory or CPU anomalies. These insights enhance understanding of workload performance, enabling users to quickly investigate and resolve issues before they impact the business. The tool generates stories specifically tailored for process-level issues in common open source integrations like Redis, Elasticsearch, NGINX, Kafka, and more. Each Watchdog story includes service, environment, and other tags to help understand the scope of the problem within infrastructure. Additionally, it provides a wealth of additional telemetry related to the issue being viewed, allowing users to troubleshoot performance issues surfaced by Watchdog.

Watchdog Insights and Alerts help identify anomalous performance trends in processes, enabling teams to quickly investigate and resolve issues before their impact spreads. These insights enhance understanding of workload performance and provide detailed information about anomalies. Watchdog stories surface specific issues such as memory or CPU anomalies in common integrations like Redis, Elasticsearch, NGINX, Kafka, and more. The stories include tags and a detail side panel highlighting the time frame of the anomaly to aid investigations. Additionally, Watchdog provides related telemetry and metrics to troubleshoot performance issues, allowing teams to pivot to processes and infrastructure for further insight.

The blog post provides a comprehensive guide on monitoring etcd with Datadog, detailing the process of setting up and configuring the Datadog Agent and Cluster Agent to send monitoring data to a Datadog account. It explains the integration of etcd with Datadog using the Datadog Operator in Kubernetes, which simplifies the installation and management of the Agent, thereby enabling efficient collection of etcd metrics and logs. The post highlights tools like the out-of-the-box etcd dashboard for visualizing cluster performance and alerts for detecting issues. It also discusses the use of tags for filtering and aggregating metrics, the importance of monitoring resource utilization to address performance degradation, and the collection and exploration of etcd logs for troubleshooting. Additionally, it emphasizes the broader visibility into Kubernetes environments provided by Datadog's integration capabilities and offers a free 14-day trial for new users.

In this text, we learn about monitoring etcd clusters using tools like Prometheus, Grafana, and etcdctl. We explore how to view a snapshot of etcd metrics via the /metrics endpoint and expand our visibility with Prometheus and Grafana. Additionally, we discuss checking endpoint health and performance with etcdctl and using journalctl to view etcd logs for greater context around metric data. Finally, the text mentions that in Part 3 of this series, it will show how Datadog provides a unified platform for monitoring etcd clusters alongside Kubernetes control plane and workloads, as well as infrastructure.

Etcd is a distributed key-value data store that provides highly available, durable storage for distributed applications. In Kubernetes, etcd functions as part of the control plane, storing data about the actual and desired state of the resources in a cluster. Key metrics to monitor include resource metrics like process_open_fds, process_max_fds, and process_resident_memory_bytes; disk metrics such as etcd_disk_backend_commit_duration_seconds, etcd_disk_wal_fsync_duration_seconds, and etcd_mvcc_db_total_size_in_bytes; network performance metrics like etcd_network_peer_round_trip_time_seconds and grpc_server_handled_total; watch metrics including etcd_debugging_store_watchers and etcd_debugging_mvcc_slow_watcher_total; Raft metrics such as etcd_server_leader_changes_seen_total, etcd_server_proposals_failed_total, etcd_server_proposals_committed_total, and etcd_server_proposals_applied_total; and Kubernetes metrics like etcd_request_duration_seconds. Monitoring these key metrics can help ensure the health and performance of your etcd cluster and by extension, your Kubernetes cluster.

You can collect and visualize etcd metrics using tools like Prometheus, Grafana, and `etcdctl`. The `/metrics` endpoint provides a snapshot of current values for all available metrics. You can secure communication with the `/metrics` endpoint by passing authentication information with your request. Prometheuse allows you to track performance over time, while Grafana provides history, trends, and patterns in the values of metrics stored in Prometheus. Etcd's `etcdctl` command-line tool allows you to execute simple tests and query the status of your etcd nodes. You can use `etcdctl` to gather information about your cluster for troubleshooting and analysis. The `journalctl` command provides a way to view etcd logs, which contain information about the etcd process, activity, and other data that can help with troubleshooting.

The text discusses Datadog's integration with the Windows Registry, highlighting its ability to collect and monitor registry key values in real-time. This integration allows users to track unexpected changes in registry values, potentially indicating performance issues or security vulnerabilities, and ensures compliance with security best practices. Users can use Datadog to create alerts for critical changes, such as certificate expirations, and employ Cloud SIEM for detecting suspicious activities, like the disabling of Windows Defender. The integration provides deep insights into Windows host activities, enabling proactive system monitoring and enhanced security measures, and encourages new users to explore these capabilities through a free trial.

Microsoft SQL Server is a popular relational database management system that provides various performance and reliability features to support business-critical applications. As workloads scale and increase in complexity, it becomes challenging to monitor all components and identify issues affecting database performance. Datadog Database Monitoring (DBM) offers deep visibility into SQL Server instances, allowing users to optimize stored procedures, identify latch contention and deadlocks, and monitor tempdb usage. DBM also helps track query performance trends and provides insights at the stored procedure level. By using DBM, users can gain comprehensive visibility into SQL Server index usage, memory utilization, performance counters, and more to ensure optimal database performance.

Microsoft SQL Server is a relational database management system that provides performance and reliability features to support business-critical applications. Datadog Database Monitoring (DBM) offers deep visibility into SQL Server instances, allowing users to monitor tempdb, index usage, and more, and gain insights into slow queries and costly explain plans. DBM can help identify latch contention and deadlocks, optimize stored procedures, and monitor tempdb usage, providing comprehensive visibility into SQL Server performance and enabling quick troubleshooting before issues affect users.

Mitigating application vulnerabilities is crucial in the software development life cycle (SDLC), especially as applications increasingly rely on third-party open source software (OSS). Datadog SCA, a comprehensive software composition analysis solution, enables DevOps and security teams to efficiently identify, prioritize, and resolve vulnerabilities in their application services. By providing full visibility into application vulnerabilities, complete context for efficient prioritization, and improved remediation for the most critical vulnerabilities, Datadog SCA helps teams effectively balance time between reducing application security risk and meeting delivery goals.

Datadog's Retention Analysis helps measure the long-term success of an application by tracking user engagement over time through high-level cohort graphs. This data enables developers to assess overall user satisfaction and identify pain points in their UX that can be optimized. By pivoting to other tools like Heatmaps and Session Replay, developers can troubleshoot these issues and create smoother user journeys. Retention Analysis also allows for the analysis of both user views and actions, as well as filtering based on various factors such as session type, browser, or country.

Datadog SCA is a comprehensive software composition analysis solution that helps DevOps and security teams efficiently implement and scale vulnerability management for their applications. It provides full visibility into application vulnerabilities, complete context for efficient prioritization, and improved remediation for the most critical vulnerabilities by continuously monitoring libraries running in production, offering detailed information about vulnerabilities, including multiple upgrade recommendations, and integrating with the larger Datadog platform.

Datadog Retention Analysis is a tool that helps measure the long-term viability of an application by tracking fluctuations in user engagement over time. It provides high-level cohort graphs to study how many users are returning to specific features in the application, enabling the measurement of overall user satisfaction and identification of pain points in UX that can be optimized. By analyzing retention rates, developers can troubleshoot UX issues using Datadog Heatmaps and Session Replay, gaining a closer view of user behavior and identifying problematic elements driving low retention rates. The tool provides insights into user demographics and allows for filtering data based on characteristics such as session type, browser, or country to spot pain points that may be affecting retention rates.

The Datadog Documentation team has adopted Vale, an open-source command-line linting tool, to automate the enforcement of their style guidelines and maintain consistency in their documentation. By integrating Vale into their CI workflow and authoring environment, they aim to reduce editing time, mental toll on writers, and improve overall quality of their documentation. The team has also created a set of rules for prose linting, including rules for avoiding jargon, mismatched tenses, and gendered language. To make it easier for contributors to follow the style guide, Vale provides automated comments and suggestions in the GitHub Files Changed tab. By automating these processes, the Datadog team can ensure that their documentation is high-quality, consistently styled, and easily maintainable.

The Datadog .NET continuous profiler is a tool designed to help developers understand their application's performance by collecting profiling samples. The profiler aims to provide insights into CPU and wall time usage, as well as thread behavior. It achieves this through sampling threads at regular intervals, getting their call stacks, and assigning durations to the samples. The profiler also measures the impact of the garbage collector on CPU consumption and provides labels to add context to each sample, including thread IDs, names, and process information. The implementation involves handling special native threads such as those created by the garbage collector, using signal handlers for Linux, custom stack walkers for Windows, and symbolization to map instruction pointers to method metadata. The profiler's impact on CPU consumption is discussed, with measures taken to minimize its effect, especially in production environments.

Datadog Case Management is a centralized ticketing system that helps organizations track, triage, and troubleshoot issues related to their infrastructure and applications. It provides a single view for all cases, allowing users to prioritize and delegate tasks effectively. The platform integrates with various observability data sources such as alerts, security signals, and error tracking issues, enabling teams to investigate and resolve problems quickly. Datadog Case Management also supports collaboration across different teams and can be used to manage operational tasks outside of the platform. It is available at no additional cost for existing Datadog customers and can be accessed through a 14-day free trial for new users.

Trace Queries in Datadog APM allows developers to filter and analyze traces based on trace-level attributes, service relationships, endpoints, and other properties. This feature helps users pinpoint the root causes of performance issues, measure end-to-end latency, and track the business impact of application performance. By enabling quick analysis of the impact of performance issues, Trace Queries assists in prioritizing troubleshooting efforts and optimizing infrastructure based on business interests and KPIs.

Datadog APM's Trace Queries feature provides indispensable insights into the state and performance of distributed applications by allowing users to filter and analyze traces based on various attributes, such as trace-level attributes, service relationships, endpoints, and other properties. This enables application developers to quickly turn granular visibility of microservices into bigger-picture insights on the health of application requests and the business impact of service performance. With Trace Queries, users can pinpoint the root causes of performance issues, measure end-to-end latency and other trace-level attributes, track the business impact of application performance, and put application performance data in context. By doing so, they can expedite troubleshooting, identify slow requests, analyze dependencies, and focus optimization efforts on high-priority issues.

Change Overlays is a feature introduced by Datadog that helps visualize deployments tracked via APM and RUM within any graphs on your Datadog dashboards, making it easier to determine the impact of specific changes on system health and performance. By using version tags on APM services and RUM events, Change Overlays automatically identifies deployments and places them in context with your metrics. This feature simplifies troubleshooting by enabling users to quickly identify faulty deployments and revert them, minimizing their end-user impact. Additionally, it provides enhanced context for tracking metrics and monitoring the impact of deployments at a glance. Change Overlays is now available in open beta.

Datadog Summit London is an upcoming event on March 26 that aims to celebrate and bring together the community of engineers, developers, and other professionals who use Datadog's observability platform. The summit will feature customer talks from companies like Dunelm and Electrolux, discussing their experiences with engineering challenges and how a strong observability culture has helped them achieve their goals. There will also be hands-on workshops covering topics such as infrastructure monitoring, distributed tracing, security, and more. Additionally, several sessions from the Datadog team will cover product updates, SRE best practices, and insights into running stateful workloads in Kubernetes. The event will provide opportunities for attendees to network with other local Datadog users and learn about new ways to use the platform effectively. Registration is free but limited, so interested individuals should RSVP soon.

Meghan Lo and Aaron Kaplan introduce Change Overlays, a feature in Datadog that helps engineers identify and revert faulty deployments, which account for around 70 percent of all application outages. By visualizing deployments within metrics graphs, Change Overlays enables teams to quickly pinpoint the impact of specific changes on system health and performance, simplify troubleshooting, track metrics with enhanced context, and monitor the impact of their deployments at a glance. This feature uses version tags on APM services and RUM events to automatically identify deployments, making it easier for engineers to contain user impact while investigating issues. With Change Overlays, teams can quickly determine the last stable version to roll back to, analyze changes' impact and status, and integrate deployment tracking with overall monitoring of applications.

The Datadog Summit London will be held on March 26 in London, bringing together the company's community of engineers and SREs to celebrate their contributions and achievements. The event features customer talks from notable companies like Dunelm and Electrolux, as well as hands-on workshops covering various topics such as infrastructure monitoring, distributed tracing, security, and web application development. Additionally, the summit includes sessions on product announcements, "Ask an SRE" with a Datadog engineer, and discussions on stateful workloads on Kubernetes. The event also features gamified challenges like AWS GameDay and CoTerm Battle to encourage collaboration and friendly competition among attendees. With opportunities for networking and learning from others in the community, the summit promises to be an engaging experience for those interested in observability, engineering culture, and software development.

Windows Performance Counters are built-in metrics in the Windows operating system that provide insights into CPU, memory, and disk usage, as well as other high-level facets of Windows subsystems, components, and native or third-party applications. These performance counters can be monitored using the built-in GUI utility or remotely through a unified monitoring solution like Datadog. By leveraging these metrics, system administrators, DevOps engineers, and developers can monitor resource usage, troubleshoot issues, optimize efficiency, reduce costs, and improve end-user experience. Datadog's Windows Performance Counters check is a configuration included in the Datadog Agent package that monitors Windows Performance Counters and streams them into Datadog. This integration allows users to view these metrics within the context of other key metrics and telemetry from across the stack, helping break down silos between teams. To effectively monitor Windows Performance Counters, it is essential to understand their conceptual building blocks: countersets (tables), counters (columns), and instances (rows). Users can choose which performance counters to map into corresponding Datadog metrics and configure the Agent file accordingly. Additionally, optional facets of the configuration provide more data and granularity for monitoring purposes. Microsoft's documentation provides guidance on which performance counters to monitor for specific technologies such as IIS, AD FS, ADO.NET, BizTalk, Failover Clustering, Exchange, SQL Server, and WCF. By leveraging Windows Performance Counters in Datadog, teams can gain deep visibility into the internal state of an application in a production environment, monitor resource usage, and design performant, effective apps that will satisfy customers.

Windows Performance Counters are built-in performance metrics exposed by the Windows operating system that provide a unified way to observe performance, state, and high-level facets of Windows subsystems, components, and native or third-party applications. These counters can be invaluable for monitoring resource usage and infrastructure health, as well as systems that services rely on. DevOps engineers and developers use performance counters to optimize efficiency, reduce costs, and improve end-user experience. While built-in GUI utilities allow users to monitor Windows Performance Counters, remote monitoring alongside other key metrics is often more practical with a unified monitoring solution like Datadog. Datadog seamlessly maps Windows native telemetry to its own metrics, which can be sliced, diced, sorted, filtered, and aggregated. To start collecting Windows Performance Counters in Datadog, users must conceptualize them at the individual counter level, understanding how they map to countersets and instances. Once configured, these counters stream into Datadog, allowing users to view them in the Metrics Explorer. Users can decide which metrics to collect based on their specific needs and goals, such as monitoring resource usage or network traffic. With Datadog's integration, Windows Performance Counters offer deep visibility into internal application state and system health, enabling teams to design performant apps that satisfy customers.

Amazon CloudWatch Network Monitor is a network monitoring service that enables customizable monitors for AWS to on-premises infrastructure via AWS Direct Connect (DX). It alerts users to connectivity issues and collects metrics such as packet loss and round-trip latency. Datadog's integration with this service allows users to send network monitoring metrics for Direct Connect paths to Datadog, offering deep visibility into network performance alongside monitors and telemetry from across hybrid systems. Users can create custom dashboards in Datadog to view alerts and metrics in one place, enabling quick identification and resolution of issues.

Amazon CloudWatch Network Monitor is a network monitoring service that enables customers to create customizable monitors for their network connectivity from AWS to on-premises infrastructure via AWS Direct Connect. This monitor alerts users to issues in the connectivity and collects metrics, allowing them to observe traffic patterns, diagnose problems, and resolve issues quickly. Datadog's integration with Amazon CloudWatch Network Monitor allows users to send network monitoring metrics for Direct Connect paths to Datadog, offering deep visibility into their network performance alongside monitors and telemetry from across their hybrid system. With this integration, customers can create custom monitors on traffic between AWS and their on-premises data centers via AWS Direct Connect, monitor these networks alongside all other monitoring data, and troubleshoot connectivity issues with additional context offered by Datadog products such as Cloud Network Monitoring (CNM), Application Performance Monitoring (APM), and more.

OpenStack is an open-source cloud platform that provides infrastructure-as-a-service functionality and additional services for orchestration, fault management, and service management. To monitor and manage OpenStack components effectively, extensive monitoring capabilities are required. Datadog has partnered with OpenStack and Rackspace Technology to expand its OpenStack Controller integration, allowing users to monitor more OpenStack components and gain visibility into their resource deployment. The integration works with Nova, Neutron, Ironic, Cinder, Octavia, Keystone, and Glance services. It enables users to identify connection issues, detect anomalies, and maintain service health through service checks and alerts. Datadog's OpenStack Controller integration provides comprehensive visibility into hypervisor load and status, server details, bare metal node status, and load balancer health, allowing users to optimize their environment for peak performance.

The OpenStack Controller integration enables customers to monitor and manage their OpenStack deployments more efficiently. This integration provides extensive monitoring capabilities, allowing users to detect and resolve issues quickly. With the preconfigured OpenStack Controller Overview dashboard, users can visualize metrics for a high-level overview of their deployment, identifying potential optimization opportunities and troubleshooting errors. The integration also includes service checks that report the health of OpenStack components, enabling users to take targeted action steps to troubleshoot issues. Additionally, users can set up monitors directly from the dashboard to alert on unexpected changes in their deployment, consolidating information into a centralized location.

Zendesk and Datadog have integrated their services to provide support teams with an enhanced solution for troubleshooting customer issues. The integration of Session Replay captures individual user sessions, allowing support engineers to reproduce problems directly from Zendesk tickets without relying on customers for detailed context. This feature minimizes the mean time to resolution (MTTR) and improves customer satisfaction by providing a clear picture of user experience. Support teams can also create playlists of user sessions to share with product and engineering teams, streamlining collaboration and issue resolution. The integration is available through Zendesk Marketplace, and new users can sign up for a 14-day free trial of Datadog.

Zendesk provides support teams with an integrated solution for processing customer inquiries and feedback. However, as organizations scale, support tickets can multiply, making it difficult to parse customer feedback and investigate issues promptly and thoroughly. To address this challenge, Zendesk has announced a new integration with Datadog Session Replay, which captures individual user sessions via telemetry and reconstructs them in a video-like playback interface. This integration enables support engineers to quickly reproduce issues by directly accessing relevant user sessions from any Zendesk ticket, eliminating their reliance on customers for detailed context when reporting a problem. With this integration, support teams can improve the speed and precision of their troubleshooting, minimize the mean time to resolution (MTTR) of customer tickets, and enhance overall reliability and customer satisfaction.