December 2014 Summaries

Mike Perham, CEO of Contributed Systems, recounts his experience with a persistent memory issue at an eCommerce company where he served as Director of Engineering. The problem involved the company's Ruby on Rails app, where background worker processes gradually consumed excessive memory, necessitating manual restarts. Monitoring the system with Datadog provided alerts but not automated solutions, leading Perham to implement Inspeqtor, a Linux-based monitoring software, to automate these restarts. Inspeqtor allows users to define rules for monitoring critical application infrastructure, and in Perham's case, it successfully managed memory usage by automatically restarting processes when they exceeded specified limits. Inspeqtor, which is open source and free, also offers a Pro version with additional features like Statsd integration for enhanced data visualization through Datadog. Perham advocates for the combined use of Inspeqtor and Datadog, highlighting their complementary capabilities in monitoring and problem-solving.

The latest version of Datadog Agent, 5.1.0, introduces new options for tagging metrics collected by the JMX integration. Users can now assign custom tags to each metric, providing more descriptive and relevant information compared to default MBean developer-assigned tags. To use this feature, users need to upgrade to Datadog Agent 5.1.0 and follow the configuration instructions in the documentation. This enhancement was made possible by a pull request from customer Keen.io, highlighting the value of community contributions to software development.

Datadog Agent 5.1.0 introduces new options for tagging metrics collected by the JMX integration, allowing users to assign custom tags to each metric and break free from the limitations imposed by MBean developers. This feature was made possible through a pull request from Keen.io customer Cory Watson, who had a need to enumerate each column's family's metrics as separate JMX queries in his Cassandra environment. The updated jmxfetch code was deployed as part of the Datadog Agent 5.1, providing better support for JMX in Datadog and enabling users to create more descriptive tags, such as internal application names. To take advantage of this feature, users can upgrade to Datadog Agent 5.1 by following the instructions provided or sign up for a free trial to experience it firsthand.

Darron Froese, a Datadog Site Reliability Engineer, shared his experience of diagnosing and resolving memory issues on one of the servers hosting nonfiction's websites. The server was experiencing unresponsiveness due to large Apache processes consuming up to 1GB of RAM. To address this issue, they set memory limits for Apache + mod_php processes and monitored them using Datadog's DogStatsD. They also wrote a Ruby script to identify which sites were the memory hogs. Through these measures, they learned that keeping MinSpareServers and MaxSpareServers low can help reclaim idle server memory and improve overall system stability.

The Datadog community was recently shared with an article from Darron Froese, a Site Reliability Engineer. The author's team had been hosting their nonfiction sites on Rackspace Cloud instances, which have proven to be reliable for their workloads. However, one server had been experiencing issues and the team took a closer look to identify the problem. They discovered that some Apache processes were using excessive amounts of memory, leading to potential database issues if those pages were missing. To gain more insight into the issue, the team used Datadog's DogStatsD to monitor the memory usage of their Apache processes. By analyzing this data, they were able to identify and address the problem, including setting limits on process memory and implementing a script to send information to Datadog for further analysis. The team learned that keeping MinSpareServers and MaxSpareServers relatively low can help reduce overall memory usage, and that with the right visibility into system performance, even seemingly minor issues can have significant repercussions.

Datadog has released an extension called Availability Monitoring in its latest Agent 5.1.0 version. This new feature introduces five types of monitors - host, integration, network, process, and custom monitors - on top of the existing metric-based ones. These monitors are designed for simplicity and can be used to monitor when a host or service is up or down. They are particularly useful in large-scale deployments due to their use of tags, which allow monitoring based on environment, data center, or AWS AMI. An example of using Availability Monitoring is demonstrated with Elasticsearch, where both host and integration monitors can be used for comprehensive coverage along with metric monitors. The API and documentation have been updated to provide more details about Availability Monitoring.

Datadog has released a major extension to its monitors in the Datadog Agent 5.1.0, called Availability Monitoring, which introduces five new kinds of monitors on top of existing metric-based ones. These new monitors are designed for large-scale deployments and use tags to apply them to hosts that belong to the same environment or data center. The new monitors can be used in conjunction with existing metric-based monitors to provide comprehensive coverage. Datadog's Availability Monitoring allows users to monitor hosts, integration services, network services, process services, and custom services, providing a simpler way to know when a host or service is up or down. The new monitors are well-suited for large-scale deployments and can be easily reconfigured as the infrastructure changes in real-time. Datadog's example shows how to use these new monitors to monitor Elasticsearch, using both host monitors and integration monitors to get comprehensive coverage of the cluster's health and metrics.