December 2015 Summaries

iHeartRadio, a streaming music and digital radio platform, uses Docker to isolate dependencies and resource usage of applications from each other. However, monitoring Docker performance can be challenging due to the lack of visibility into container-level resource consumption. To overcome this issue, iHeartRadio uses Datadog for infrastructure monitoring, which collects CPU, memory, I/O, and network metrics from each Docker container and aggregates metrics by any tag or tags. This allows them to monitor service-level metrics by aggregating by Docker image. Additionally, they use Datadog to monitor non-Docker services such as HAProxy, MongoDB, and Elasticsearch, which helps in correlating Docker performance metrics with health and performance throughout their infrastructure.

iHeartRadio uses Docker to isolate dependencies and resource usage of applications from each other, leveraging its ability to control dependencies and resource usage. However, monitoring Docker performance was a challenge due to lack of visibility into container-level resource consumption. They use Datadog for infrastructure monitoring, which collects CPU, memory, I/O, and network metrics from each Docker container and can aggregate metrics by any tag or tags. This allows iHeartRadio to monitor service-level metrics by aggregating by Docker image, breaking down network traffic by image and container, and correlating Docker performance with non-Docker services. Datadog also enables rapid changes in internal network traffic to trigger alerts, aiding investigation of issues that arise.

OpenStack is an open-source cloud computing software platform primarily deployed as infrastructure-as-a-service. It was developed by Rackspace and NASA and has high-profile supporters such as Google, Hewlett-Packard, IBM, and Intel. The platform aims to provide an open alternative to cloud providers like AWS, Azure, DigitalOcean, and Joyent. OpenStack features a modular architecture with 16 services, including Nova (computing controller) and Neutron (network management module). Datadog's integration provides access to metrics from these components, enabling better coordination of load across the infrastructure and proper tuning of nodes. The integration also allows for visualization, aggregation, and correlation of metrics with other systems or software in a stack, as well as automatic detection of outliers and anomalies.

This article discusses how to monitor OpenStack with Datadog. It explains that a comprehensive monitoring implementation includes all the layers of your deployment, not just the metrics emitted by OpenStack itself. With Datadog, you can collect OpenStack metrics for visualization, alerting, full-infrastructure correlation, and more. The article provides step-by-step instructions on how to configure OpenStack with Datadog, including setting up roles and users, modifying configuration files, installing the Datadog Agent, configuring the Agent, integrating RabbitMQ with Datadog, verifying the configuration, enabling the integration, and viewing the metrics. It also touches upon diagnosing and alerting using Datadog.

The text discusses integrating OpenStack Nova and RabbitMQ with Datadog for comprehensive monitoring of the infrastructure. To achieve this integration, the user needs to configure the Datadog Agent to collect metrics from OpenStack, RabbitMQ, and other services in the deployment. The necessary steps include installing the Datadog Agent, configuring the OpenStack policy files, integrating RabbitMQ with Datadog, verifying the configuration, enabling the integration, and creating a custom dashboard to monitor OpenStack Nova and RabbitMQ metrics. With this setup, users can correlate what's happening in OpenStack with what's happening in their infrastructure, receive real-time alerts for potential issues, and gain high-resolution insights into their system performance.

OpenStack is a cloud-computing software platform that can be likened to Amazon Web Services but can be hosted anywhere. Developed by Rackspace and NASA, it has gained high-profile supporters like Google, Hewlett-Packard, IBM, and Intel. The platform features a modular architecture with 16 services, including OpenStack meta-services. Datadog's integration provides visibility into Nova, the computing controller, and Neutron, the network management module, offering metrics on workload, virtual machines, hypervisor metrics, project/tenant information, host-level performance statistics, and more. The integration also includes up/down information for Keystone, an identity management and access control layer, and allows users to visualize and aggregate metrics from other systems or software in their stack, alert on abnormal behavior, detect outliers, and more.

This summary delves into the various methods of collecting operational data for monitoring OpenStack's computation module, Nova. It discusses the importance of combining data from different channels to get a complete picture of the deployment. The post covers three main categories: collecting Nova metrics via API or SQL queries, using the CLI, and gathering metrics from RabbitMQ. Additionally, it explores collecting notifications, service checks, and API checks to gain valuable insights into system behavior. By leveraging these methods, users can effectively monitor their OpenStack deployments and identify potential issues.

The text is the first part of a four-part series focused on monitoring OpenStack Nova, an open-source cloud computing platform primarily used for infrastructure-as-a-service. This installment provides a comprehensive overview of Nova, the Compute module of OpenStack, and outlines various metrics crucial for monitoring its operations. It emphasizes the importance of tracking hypervisor metrics for understanding resource utilization and availability, tenant metrics for managing user resources, and Nova server metrics to ensure even distribution of load among compute nodes. The document also highlights the significance of integrating RabbitMQ metrics due to its role in OpenStack's message-passing pipeline and elaborates on the importance of listening to Nova notifications for insights into asynchronous operations. The text suggests that understanding these metrics and events is essential for maintaining the health and performance of an OpenStack deployment, while future installments promise to delve into data collection methods and specific monitoring tools like Datadog.

Amazon ElastiCache is a fully managed in-memory cache service offered by AWS that helps improve the performance of applications by retrieving data from memory instead of accessing it from slower disk storage. The key performance metrics for monitoring ElastiCache fall into four general categories: client, cache performance, memory, and host-level metrics. Monitoring these metrics can help identify latency issues, ensure efficient use of cache resources, and maintain optimal application performance. By understanding and tracking these metrics, users can make informed decisions about scaling up or down their cache clusters as needed.

AWS provides multiple ways to access ElastiCache metrics from CloudWatch, including using the online management console, the AWS Command Line Interface, and integrating with a dedicated monitoring tool. The online management console allows for basic automated alerts and visualizing recent changes in individual metrics, while the command line interface offers more flexibility and can be used for spot checks and ad hoc investigations. Dedicated monitoring tools, such as Datadog, offer extended functionality, including correlation of CloudWatch metrics with caching engine and other infrastructure metrics, dynamic slicing, aggregation, and filters on metrics, historical data access, and sophisticated alerting mechanisms. Collecting native Redis or Memcached metrics can provide a fuller picture with higher-resolution data, but may require additional tools and setup.

Amazon ElastiCache is a fully managed in-memory cache service offered by AWS, allowing applications to speed up requests and improve throughput. Key performance metrics include client metrics (e.g., number of connections, Get commands, Set commands), cache performance metrics (e.g., hits, misses, latency), memory metrics (e.g., memory usage, evictions, swap usage), and host-level metrics (e.g., CPU utilization). Monitoring these metrics helps ensure the health and performance of ElastiCache. It is essential to correlate metrics to see the full picture and investigate issues related to events sent by ElastiCache. By monitoring the right metrics, developers can gain insights into their cache's performance and optimize it for better user experience and application responsiveness.

Coursera, a leader in online education, leverages Amazon ElastiCache with Memcached as a read-through cache on top of Cassandra to support its extensive user base and course offerings. The platform employs Datadog to monitor ElastiCache's performance, ensuring the cache is effective in maintaining a high hit rate and preventing memory evictions that could impact application latency. Key metrics monitored include CPU utilization, memory usage, Get and Set commands, network throughput, and events to ensure that nodes remain balanced and efficient. Coursera's engineering team employs Datadog alerts to manage critical metrics such as eviction rates, available memory, and hit rates, using communication tools like PagerDuty and Slack for prompt notifications. By integrating Datadog, Coursera can visualize and correlate metrics across its infrastructure, enhancing the ability to quickly diagnose and address performance issues.

Apache Cassandra is a distributed database system known for its scalability and fault-tolerance. Monitoring Cassandra performance helps identify slowdowns or resource limitations and includes tracking throughput, latency, disk usage, garbage collection frequency and duration, and errors and overruns. Key metrics to monitor include read and write requests per second, write response time in microseconds, read response time in microseconds, key cache hit rate, load or total disk space used, completed compaction tasks, pending compaction tasks, ParNew count, ParNew time, ConcurrentMarkSweep count, ConcurrentMarkSweep time, exceptions, timeout exceptions, unavailable exceptions, pending tasks, and currently blocked tasks.

This article provides a step-by-step guide to setting up Cassandra monitoring in Datadog. It begins with installing the Datadog Agent and creating a configuration file for Cassandra. The Agent is then restarted, and verification of the configuration settings is performed using the Datadog info command. Finally, the Cassandra integration is installed within the Datadog account. Once properly configured, users can monitor dozens of Cassandra metrics in real-time, create comprehensive dashboards, and set up alerts for potential issues.

The text discusses setting up Cassandra monitoring in Datadog, a robust monitoring system that collects and visualizes metrics from various sources. To integrate Cassandra with Datadog, users need to install the Datadog Agent on their Linux hosts, create a simple configuration file for the Agent, and restart it after updating the configuration. The configuration file allows users to define which Cassandra metrics Datadog will collect and apply tags to instances for filtering and aggregating metrics. Once configured, users can verify the integration settings by running the Datadog `info` command and install the Cassandra integration in their Datadog account. With this setup, users can monitor dozens of Cassandra metrics, create comprehensive dashboards, and manipulate raw metrics into something more usable. Additionally, users can set up monitors to keep watch over their metrics and alert them when there are problems.

The second installment of a three-part series on monitoring Apache Cassandra explores various methods for collecting Cassandra metrics, essential for understanding the performance and resource utilization of the database. The post outlines options ranging from lightweight tools like nodetool and JConsole to more sophisticated JMX and Metrics integrations with external monitoring services. Nodetool, a command-line utility, provides quick snapshots of cluster health and node statistics, while JConsole offers a graphical interface for exploring a comprehensive set of metrics via JMX. For deeper insights into long-term trends and correlations, the article suggests employing advanced monitoring systems that integrate Cassandra metrics with other technologies in the stack. The text emphasizes the importance of using powerful monitoring solutions in production environments, highlighting Datadog’s capabilities for efficient Cassandra cluster monitoring.