October 2015 Summaries

This article discusses the challenges of monitoring containers in a production environment using Docker technology. Containers are lightweight virtual runtimes that provide software isolation and are ideal for microservice architectures and rapidly scaling environments. However, their frequent creation and deletion make them difficult to monitor effectively with traditional infrastructure or application performance monitoring tools. The article suggests an approach centered around layers and tags to simplify monitoring of highly dynamic container-based stacks.

Docker containers are becoming increasingly common in production due to their ability to provide an escape from software dependency hell and scaffolding for scalable software architectures. However, this also increases operational complexity by an order of magnitude as containers have a very short half-life and are used in large numbers. To address this issue, it's essential to monitor all layers of the stack together, without gaps, and tag containers so that they can be monitored as queryable sets rather than individual entities. This allows for powerful monitoring capabilities and effective visibility into infrastructure and applications.

Consul is a distributed configuration and service-discovery tool that provides datacenter-aware architecture and high availability. It allows clients to register services and dynamically discover those services, enabling easy scaling and fault tolerance. Consul uses the Raft consensus algorithm for strong data consistency and is built on top of a distributed client-server architecture. The tool offers health checks for both node-level and service-level checks, allowing users to monitor cluster health and detect potential issues. Additionally, Consul provides metrics on latency, cluster health, events, tagging, and alerts, enabling users to create custom monitoring solutions with Datadog. By integrating Consul with Datadog, users can gain a comprehensive view of their infrastructure performance and optimize their applications for scalability and reliability.

This article discusses how to monitor MySQL on Amazon RDS by collecting both standard RDS metrics and native MySQL metrics. It explains three ways to collect CloudWatch metrics: using the AWS Management Console, command line interface, or a monitoring tool with CloudWatch integration. Additionally, it outlines four methods for collecting native MySQL metrics: querying server status variables, querying the performance schema and sys schema, using the MySQL Workbench GUI, and employing a MySQL monitoring tool. The article concludes by stating that in the next part of this series, it will demonstrate how to use Datadog to collect, visualize, and set alerts on metrics from both RDS and MySQL.

This article discusses monitoring MySQL performance metrics on Amazon RDS (Relational Database Service). It highlights key metrics in areas such as query throughput, query performance, resource utilization, and connection metrics that provide a detailed view of the database's performance. The article also explains how to collect these metrics from CloudWatch and directly from MySQL instances. Additionally, it covers monitoring read replicas and provides recommendations for actionable alerts based on these metrics.

The key points of the text are that MySQL performance on Amazon RDS can be monitored using various metrics, including query throughput and errors, disk I/O operations, CPU utilization, memory usage, storage space, network traffic, and connection metrics. Key metrics include `Questions`, `Queries`, `Reads` (calculated), `Writes` (calculated), `Slow_queries`, `Query_errors`, `DiskQueueDepth`, `FreeStorageSpace`, `NetworkReceiveThroughput`, `NetworkTransmitThroughput`, `DatabaseConnections`, `Threads_connected`, `Threads_running`, `Aborted_connects`, and `Connection_errors_max_connections`. These metrics can be collected using Amazon CloudWatch and MySQL, and are essential for identifying performance bottlenecks and ensuring the optimal operation of the database instance. Monitoring these metrics is crucial to maintaining good application performance, and will help users identify when it is necessary to increase their instance storage, IOPS, or memory.

To collect both standard RDS metrics and native MySQL metrics, users can access RDS metrics via Amazon CloudWatch and query server status variables, the performance schema, or use a monitoring tool with a CloudWatch integration. The process involves connecting to the database instance remotely using tools like SSH, querying server status variables, enabling the performance schema, or using a monitoring tool that integrates with MySQL. These methods provide insights into MySQL performance, allowing users to set up alerts and visualize metrics for a comprehensive view of their database instance's health.

Datadog integrates with AWS CloudWatch to gather RDS metrics and provides a comprehensive view of database health and performance by correlating native MySQL metrics with RDS metrics, allowing for the identification of recent and long-term trends and performance problems. To connect MySQL RDS to Datadog, users need to configure the AWS CloudWatch integration and set up role delegation in AWS IAM, create a new role for Datadog, and attach a policy that grants read-only access to AWS services. The Agent integrates seamlessly with MySQL to gather key performance metrics, which are reported at a higher resolution than basic CloudWatch metrics. Users can enable enhanced monitoring for RDS instances running MySQL, which includes more than 50 new CPU, memory, file system, and disk I/O metrics that can be collected on a per-instance basis as frequently as once per second. Once integrated with Datadog, users can access a comprehensive dashboard called “Amazon - RDS (MySQL)” that gathers key metrics around query throughput and performance, resource utilization, database connections, and replication status.

Datadog introduces event-based alerts to identify specific occurrences such as failed deploys, incorrect job runs, and more. This feature works with any integration that sends events to Datadog. Users can customize filters for precise event detection, including string matching, status, priority, source, and tags. Event-based alerts also support aggregations and absence conditions. All existing alerting features are now available for events as well.

With Datadog's new event-based alerts, users can trigger notifications on specific events such as deploy failures, job errors, and third-party service outages, allowing for more precise and flexible alerting. The feature works with any integration that sends events to Datadog and allows users to combine filters to select specific events to alert on, including string matching, status, priority, source, tags, and aggregations. Additionally, the feature includes absence checks to notify when a critical event hasn't been reported in a certain time frame. This new capability enhances Datadog's existing alerting features, enabling users to set up custom alerts based on specific events and customize their notification conditions.

Amazon Elastic Load Balancer (ELB) is a widely used AWS service that ensures a smooth user experience and provides fault tolerance for web and mobile applications. It continuously checks for unhealthy EC2 instances and reroutes traffic to healthy ones. Key ELB performance metrics include load balancer metrics, such as RequestCount, SurgeQueueLength, and SpilloverCount, which measure the number of requests, queued requests, and dropped requests, respectively. These metrics are crucial to ensure the load balancer itself and EC2 instances behind it remain healthy. Additionally, backend-related metrics like HealthyHostCount, UnHealthyHostCount, Latency, and BackendConnectionErrors provide insights into the health and performance of backend instances. Monitoring these metrics is essential for identifying potential issues, such as high latency, overloaded backend hosts, or network issues, and taking corrective actions to ensure a smooth user experience. By understanding and monitoring these key ELB performance metrics, users can gain valuable insights into their load balancers' performance and backend servers' health, ultimately leading to improved application scalability and reliability.

To monitor Amazon Elastic Load Balancer (ELB) effectively, you can use CloudWatch, which provides key performance metrics that are available via AWS Management Console, command-line interface (CLI), and monitoring tools integrating the CloudWatch API. The AWS Management Console allows for setting up automated alerts and visualizing recent changes in individual metrics through graphs. Using the CLI enables quick spot checks and ad hoc investigations by querying for specific metrics. Monitoring tools integrated with CloudWatch offer extended functionality, such as correlating metrics from different infrastructure parts and accessing historical data. Additionally, ELB access logs can be used to investigate specific request issues, but may require dedicated log analytics tools for large traffic volumes.

The final installment of a three-part series on monitoring Amazon Elastic Load Balancers (ELB) explores the integration of Datadog for enhanced visualization and alerting of ELB metrics. It highlights the process of setting up AWS CloudWatch integration to access ELB data, allowing users to view comprehensive metrics through Datadog’s customizable dashboards. The article emphasizes the benefits of correlating ELB metrics with those of backend EC2 instances to diagnose performance issues effectively. Additionally, it describes the advantages of using the Datadog Agent for more granular monitoring of EC2 instances by collecting native metrics, which provides a higher-resolution view of system performance. By integrating these tools, users can gain critical insights into their infrastructure's health and performance, enabling improved visibility and cost savings.