January 2018 Summaries

Datadog introduces a new feature called Host Info in its Application Performance Monitoring (APM) platform to provide immediate infrastructure-level context for service requests. The Host Info panel displays host system metrics and tags alongside request traces, allowing users to trace requests end-to-end across their infrastructure. This integration of APM and metrics enhances full-stack observability without the need for multiple monitoring tools. The Host Info panel also provides additional context by showing system metrics and tags related to a selected span in a trace. This feature aims to speed up application troubleshooting by reducing time spent searching for root causes of issues, ultimately improving system behavior understanding and guiding root cause analysis during performance problems. Existing Datadog customers can access the Host Info panel from APM, while new users can try it out with a free 14-day trial.

Datadog has introduced the Host Info feature in its Application Performance Monitoring (APM) platform, which provides immediate infrastructure-level context for service requests by displaying host system metrics and tags side-by-side with request traces. This new feature brings APM and infrastructure monitoring closer together, offering full-stack observability without the need to switch tools. The Host Info panel displays host system metrics and request traces on a single pane of glass, allowing users to easily correlate slow requests with system issues, such as overloaded database servers. By showing host data inline with request traces, users can reduce time spent searching for the root cause of an issue and gain increased visibility into their full system.

On January 3rd, an unexpected spike in CPU usage was observed across large Redis instances due to the disclosure of Meltdown and Spectre vulnerabilities and subsequent patches. The average impact on system.cpu.system was less than 1%, but the widespread issue affected millions of cores running varying workloads. Hosts with more system CPU usage experienced a higher-magnitude impact, while compute-optimized and general-purpose virtual machines were most affected. This large-scale analysis confirms the significant performance impact of these security patches and highlights the potential for future vulnerabilities in this class.

The incident highlights the unexpected performance impacts of the Meltdown and Spectre patches on CPU utilization, particularly in kernel space. A large-scale analysis by Datadog found a noticeable increase in `system.cpu.system` across millions of cores, with the impact being more pronounced in hosts that spend more time in kernel space. The study also revealed that compute-intensive workloads were most affected, while memory-optimized instances showed less significant increases. The widespread issue underscores the systemic effects of security patches and raises concerns about potential future performance impacts.

RabbitMQ is a message broker that implements a messaging architecture for loosely coupled microservices. It routes messages between producers and consumers, keeping them in queues until they can be delivered. The broker exposes metrics for its main components, allowing comprehensive monitoring of message traffic and system performance. RabbitMQ runs as an Erlang runtime node and supports multiple protocols, including AMQP. Producers publish messages to exchanges, which route them to queues. Queues wait for consumers to be available before delivering the message. Monitoring RabbitMQ is crucial for ensuring application availability and performance. Key metrics include exchange performance, connection performance, queue performance, and resource utilization. By tracking these metrics, developers can identify issues and optimize their messaging setup.

RabbitMQ provides several monitoring tools to help developers track the performance of their messaging infrastructure. The built-in CLI tool, `rabbitmqctl`, offers quick access to key metrics such as node-level resource usage and queue performance. The management plugin extends this functionality with a web server that reports metrics via UI and API, including exchange metrics, connection performance, and queue performance. Additionally, the Prometheus plugin enables users to leverage OpenMetrics for granular system monitoring, allowing for better data fidelity at the cost of heavier resource consumption. Two plugins, `rabbitmq_tracing` and `firehose`, provide logging capabilities for messages and events, respectively. The event exchange and firehose tools allow developers to decouple tracing from their application code by controlling the firehose through the management UI or API. By using these monitoring tools together, developers can gain a unified view of data, receive alerts, traces, and logs in a single place, and streamline their monitoring process as their messaging infrastructure scales.

The text discusses the integration of Ansible, an infrastructure automation solution, with Datadog's monitoring capabilities. It explains how users can leverage Ansible roles to quickly accomplish tasks such as installing a database and configuring its service. The integration allows for monitoring as code, enabling deployment of the Datadog Agent and other integrations while providing reporting and visibility into key metrics and events. The guide provides step-by-step instructions on how to use this Ansible role and callback plugin to deploy the Datadog Agent on a node and track Ansible performance metrics and events in Datadog. It also demonstrates how to configure Ansible playbooks for enabling specific Datadog integrations, such as NGINX integration. The text concludes by encouraging users already using Datadog and Ansible to explore the documentation for further integration benefits.

Ansible is an infrastructure automation solution that enables users to provision, deploy, and manage their infrastructure and applications. It provides roles similar to Puppet modules and Chef cookbooks, which can be downloaded from the open source Ansible Galaxy to accomplish tasks such as installing a database and configuring its service. Datadog's integration with Ansible allows users to implement monitoring as code by deploying the Datadog Agent and enabling other integrations. The integration provides real-time reporting and visibility into key metrics and events, including failed tasks, through an out-of-the-box dashboard. To use this integration, users need to source a set of environment variables, add node FQDNs to the hosts file, and install the Datadog role and callback plugin. The integration can be used to track Ansible performance metrics and events in real-time, correlate Ansible events with metrics, and configure Ansible playbooks for multiple roles.

Amazon Elastic Compute Cloud (EC2) is a core component of the Amazon Web Services platform that provides scalable cloud-based computing capacity. EC2 instances allow users to increase or decrease resource capacity within minutes, and they can be launched in specific parts of the world to match regional demand. EC2 can be integrated into other AWS components and features, including Auto Scaling, which automatically launches or stops instances to meet the demand on your application. Monitoring key metrics such as disk I/O, network, CPU usage, status checks, and events is crucial for maintaining optimal performance of your EC2 infrastructure.

Amazon's CloudWatch service is used to collect EC2 metrics and events. Users can access these metrics through the CloudWatch web console, AWS command line tool, or a program or third-party monitoring service that connects to the CloudWatch API. The service gathers metrics via hypervisor instead of reporting from instances themselves, so it doesn't collect all resource metrics such as memory usage statistics. To fill this gap, custom metrics can be forwarded to CloudWatch and monitored using the same methods outlined above. Additionally, comprehensive monitoring services that integrate with EC2 and the rest of a user's stack can provide deeper visibility into instances and infrastructure components.

Amazon CloudWatch is an excellent starting point for monitoring EC2 instances and AWS services. However, integrating CloudWatch with Datadog provides a more detailed and comprehensive view of the entire infrastructure. Datadog automatically collects performance metrics for EC2 and other AWS services while retaining data for 15 months at full granularity. Installing the Datadog Agent on instances enables additional system-level metric collection, including memory, disk latency, and others. With over 650 integrations, Datadog allows users to visualize, correlate, and alert on metrics from AWS and other systems in one place. EC2 + Datadog integration is beneficial for monitoring infrastructure, applications, and services. There are two ways to start monitoring EC2 instances with Datadog: enabling the AWS integration or installing the Datadog Agent. The former allows users to pull full AWS metrics into Datadog without installing anything on their instances, while the latter provides detailed monitoring of applications and infrastructure.

Amazon Elastic Compute Cloud (EC2) is a core component of Amazon Web Services that provides scalable cloud-based computing capacity, allowing users to provision compute resources in the form of virtual servers called instances. EC2 offers a wide range of instance types with different CPU, memory, storage, and networking capacities, as well as various operating systems and software configurations. Users can monitor their instances' performance using CloudWatch's metrics, including disk I/O, network throughput, CPU utilization, and status checks. Monitoring these metrics is crucial to ensure the health and performance of EC2 infrastructure, detect potential issues, and make informed decisions about instance scaling, upgrading, or downscaling. Additionally, users need to be aware of events that may affect their instances' lifecycle, such as stopping, retiring, or system maintenance. While CloudWatch does not report system-level memory metrics for instances, users can collect memory metrics using other methods. By tracking these metrics and understanding how to interpret them, users can optimize their EC2 infrastructure to meet their needs efficiently and cost-effectively.

Amazon's CloudWatch service provides a way to collect and monitor metrics and events from AWS resources, including EC2 instances. To access CloudWatch metrics, users can use the web console, the AWS command line tool, or a third-party monitoring service that integrates with the CloudWatch API. Additionally, users can supplement available CloudWatch metrics by running a monitoring agent that pulls system-level information not directly collected by CloudWatch. The service provides various options for filtering and aggregating data, including dimensions, periods, and statistics. Users can create dashboards to visualize multiple configurable graphs simultaneously and set alarms to notify them when certain thresholds are exceeded. The AWS CLI tool also allows users to query the full AWS API from the command line, enabling them to collect metrics and status checks programmatically. Furthermore, Amazon provides SDKs for major programming languages and mobile platforms to access the CloudWatch and EC2 namespaces via APIs. However, CloudWatch has limitations in terms of collecting resource metrics, such as memory usage statistics, which can be addressed using custom metrics or a comprehensive monitoring service like Datadog.

Maxim Brown explains that Amazon CloudWatch is an excellent starting point for monitoring EC2 instances and AWS services, but connecting it to Datadog provides a more detailed view of the entire infrastructure. The Datadog Agent collects additional system-level metrics from EC2 instances at 15-second resolution, including memory, disk latency, and others, and offers over 850 integrations with other systems for visualization and correlation. Datadog APM enables tracing requests through distributed services and cloud instances to troubleshoot bottlenecks in cloud applications. The Agent can be installed on instances to collect detailed metrics, and it also includes out-of-the-box support for log collection from AWS cloud services and popular technologies like Apache and Java. Datadog's Live Container view provides complete coverage of container fleets with metrics reported at two-second resolution, while Live Process monitoring offers visibility into all processes running across the entire distributed architecture. The integration allows users to create customizable dashboards on their Datadog dashboard list to visualize key metrics organized by instance resource usage, and it enables alerts for potential issues based on tags and events from AWS, allowing for precise scoping of alerts and dynamic monitoring.

Datadog has introduced a new feature called "dashboard lists" to help users organize and find their custom dashboards more easily. Dashboard lists can be created on the revamped Dashboards page, allowing users to group related dashboards together for sharing with other teams across an organization. The feature also improves collaboration by enabling users to explore and utilize dashboards they may not have known existed. Additionally, faster search capabilities and starring/bookmarking favorite dashboard lists are now available on the new Dashboards page.

Datadog has revamped its Dashboards page with a new feature called dashboard lists, which enables users to organize and share their custom dashboards across the organization. This feature allows users to group their favorite dashboards into meaningful lists that are easy to share with other teams, making it easier to collaborate and find what they're looking for. Dashboard lists provide valuable context around each dashboard's purpose and can be customized to fit individual team needs. The new Dashboards page also includes faster search capabilities, allowing users to quickly find specific dashboards or lists. Additionally, users can star their favorite dashboards and dashboard lists to reduce noise and improve focus.