November 2015 Summaries

Etcd is a distributed key-value store designed by CoreOS to manage configuration settings across clusters of Docker containers, machines, or both. By enabling the etcd integration within Datadog, you can ensure that configurations provided to your machines are consistent and up to date throughout your cluster. The dashboard includes throughput metrics tracking successful directory or key creation, writes, updates, deletes, and two atomic operations for "compare and swap" and "compare and delete". Additionally, it provides package and bandwidth send and receive rates, append requests, error failure rates, and latency between the leader and its followers. Depending on your application, you may also want to use metrics such as the number of queries watching for changes to a value or directory, and the rate at which values or directories are expiring due to TTL.

As a distributed key-value store, etcd is designed to manage configuration settings across clusters of Docker containers or machines. Activating the Datadog etcd integration ensures consistent and up-to-date configurations throughout your cluster. The integration provides real-time monitoring of etcd's throughput, including metrics on successful directory or key creation, writes, updates, deletes, and atomic operations. Additionally, it tracks package and bandwidth send and receive rates, append requests, and failure rates of the leader's Raft RPC requests to followers. Latency between the leader and its followers is also monitored, with metrics available for current, average, minimum, maximum, and standard deviation latency. The integration also provides additional metrics such as watcher counts, expire count, and auto-tagging with leader/follower status. With the Datadog etcd integration, users can set alerts on anomalous changes to their event stream and notify humans of large anomalies, ensuring the health of their etcd cluster.

Amazon Aurora is a MySQL-compatible database offered on Amazon RDS, providing features such as auto-scaling storage, low-latency replication, and rapid automated failover. To effectively monitor performance, it's crucial to track key metrics in areas like query throughput, latency, resource utilization (CPU, memory, network), and connection metrics (open connections, failed attempts). Monitoring these metrics can help identify bottlenecks, optimize queries, and ensure the database instance has sufficient resources. The Aurora-specific metric `AuroraReplicaLag` tracks the lag time between the source instance and read replicas, which should be investigated if consistently long. By monitoring these metrics, users can maintain good application performance, upgrade instance types as needed, or add read replicas to mitigate I/O limitations.

This post provides a comprehensive guide on how to collect and visualize metrics from Amazon Aurora's MySQL-compatible edition using CloudWatch, as well as four methods for collecting database engine metrics: querying server status variables, the performance schema, the sys schema, and using a MySQL-compatible monitoring tool. The post explains how to set up automated alerts when metric thresholds are crossed and provides an overview of how to use Datadog in Part 3 of this series.

This post explains how to integrate Amazon Aurora with Datadog for comprehensive monitoring of database health and performance. To start, you need to connect Datadog to CloudWatch, which involves setting up role delegation in AWS IAM, creating a new role for Datadog, and granting the Datadog role read-only access to your AWS services. Next, you integrate Datadog with Aurora's database engine by installing the Datadog Agent on an EC2 instance running the database, providing the Aurora instance endpoint, and tagging Aurora metrics with the DB instance identifier. Once set up, the Agent collects detailed metrics from the database instance, which can be unified under a single dashboard called "Amazon - RDS (Aurora)". This dashboard displays key metrics around query throughput, resource utilization, and replication lag, as well as database engine metrics from all instances configured via the MySQL integration. Additionally, enhanced monitoring for RDS instances running Aurora is available, providing more than 50 new CPU, memory, file system, and disk I/O metrics that can be collected on a per-instance basis as frequently as once per second. With Datadog's RDS Enhanced integration, you can monitor these high-resolution metrics in real-time and customize your dashboard to suit your needs.

The text discusses the growing adoption of Docker and the resulting complexity in container orchestration, highlighting Kubernetes as a solution for managing containerized applications across multiple hosts. It emphasizes the importance of monitoring in ensuring good application performance and introduces a new integration with Kubernetes that allows users to monitor clusters, containers, and applications in a unified platform. This integration simplifies resource sharing and management through Kubernetes' abstraction of containers into pods, enabling users to track performance metrics by various criteria. The integration also supports monitoring of Kubernetes itself, offering insights into system metrics and node status, while providing access to Datadog's visualization, alerting, and collaboration features. Existing Datadog users can easily set up this integration, and new users are encouraged to explore it through a free trial. The text concludes by noting ongoing efforts to enhance monitoring capabilities with new features and integrations.

In this third installment of a four-part series on monitoring Docker, the focus is on the methods of collecting Docker metrics, highlighting three key mechanisms: pseudo-files in sysfs, the stats command, and the API. Each approach has its own strengths and limitations in terms of the types of metrics they can access, such as CPU, memory, I/O, and network metrics. Pseudo-files offer a fast and lightweight way to access metrics without needing root access, though they have limitations regarding I/O and network data. The stats command provides a continuous live stream of basic metrics, enhanced in version 1.9.0 to include disk I/O metrics, while the API offers a more detailed live stream of metrics, accessible through local connections with potential for increased flexibility via custom socket bindings. The article emphasizes that while these tools offer valuable insights, they may not suffice for production environments, where a dedicated monitoring service is recommended for comprehensive system health and performance tracking.

K Young's article is the second in a series focused on monitoring Docker, specifically detailing the resource metrics available from Docker containers, which differ from traditional host metrics due to the container's operation within cgroups. The piece outlines key Docker resource metrics, including CPU, memory, I/O, and network, with specific attention to CPU throttling and memory utilization metrics such as resident set size (RSS) and cache memory. It emphasizes that some traditional host metrics are missing, like nice and idle CPU time, while offering insights into the unique metrics applicable to Docker containers. The article also previews the next installment in the series, which will explore three distinct methods for collecting Docker resource metrics, noting that the techniques differ from those used in traditional hosts.

The HAProxy monitoring series has covered the key metrics emitted by HAProxy, and now focuses on collecting these metrics. The most common method to access HAProxy metrics is through a static status page that can be viewed with any web browser, but this requires modification of HAProxy's configuration. An alternative is to use the Unix socket interface, which allows for dynamic modification of HAProxy's configuration and offers security benefits. The socket interface can be accessed using tools like netcat or socat, and provides a more practical method for scraping HAProxy metrics than the status page. Third-party tools like HATop are also available, offering human-readable metrics and management capabilities. Ultimately, while HAProxy's built-in tools provide sufficient data for spot checking, dedicated monitoring systems may be necessary for production environments.

HAProxy is an open-source solution for load balancing and reverse proxying that provides high availability and can handle both TCP and HTTP requests. It consistently performs well in benchmarks against other popular reverse proxies, making it a fundamental element in the architecture of many high-profile websites. Monitoring HAProxy's key metrics is crucial to ensure proper performance and operation, as poor load balancer performance can increase latency across the entire stack. Frontend metrics provide information about client interactions with the load balancer, while backend metrics measure communication between HAProxy and backend servers handling client requests. Health metrics expose information about the health of each front and backend server. Monitoring these metrics can help identify potential hotspots, diagnose issues, and optimize HAProxy configuration for better performance and availability.

This article discusses the integration of HAProxy with Datadog for monitoring and logging purposes. It covers how to verify that HAProxy is set to output metrics, install the Datadog Agent, configure the Agent to collect HAProxy logs and metrics, restart the Agent, and view the collected data in the Datadog account. The article also explains how to capture and visualize HAProxy logs, create custom dashboards, and set up alerts for potential issues. It provides a sample haproxy.cfg file and instructions on how to use it with the Datadog Agent, as well as information on log levels and filtering.

Datadog and Moxtra are two powerful tools that can be combined to enhance team collaboration and productivity in managing business-critical information. Datadog allows monitoring of infrastructure while Moxtra provides a unified platform for messaging, document sharing, video conferencing, and screen sharing. Alerts from Datadog can be configured directly to Moxtra binders, enabling quick action by teams. The combination of these tools streamlines the process of tracking performance and resolving issues efficiently.

Datadog × Moxtra is a powerful combination that enables teams to stay up-to-date on business-critical information by integrating Datadog's monitoring capabilities with Moxtra's collaboration solution. With Moxtra, teams can have a unified place to send messages, collaborate on files, and communicate in real-time, while also connecting all apps in one place for seamless accessibility and actionable content. This integration allows users to configure alerts directly to their favorite binder, enabling rapid response to unusual spikes or high CPU usage issues, such as the example where Jane from the Ops team receives a notification and promptly tackles the situation with her team. The combination of Datadog's informative dashboards and Moxtra's collaboration features enables teams to track performance in real-time, collaborate seamlessly, and instantly turn alerts into action.