January 2016 Summaries

Host Maps in Datadog provide a comprehensive view of an entire infrastructure, allowing users to drill down into specific segments with customizable aggregation and filtering. Since its release last year, numerous enhancements have been made including a dashboard widget, three new color schemes, custom fill and size by metrics, and saved Host Maps. These features enable users to better understand their infrastructure at any scale and share specific views with their team.

Host Maps is a powerful tool in Datadog that allows users to visualize their entire infrastructure at any scale, giving them a bird's-eye view of their hosts and the ability to drill down into specific segments for arbitrary aggregation and filtering. The latest enhancements include a dashboard widget, three new color schemes, custom fill and size by metrics, and saved Host Maps, which enable users to embed customizable views on screensboards or timeboards, choose colors that convey meaningful data representations, filter data by any metric, save customized views for later use, and slice and dice their data into any representation they like. These features make it easier for engineering teams to understand their infrastructure at scale and identify root causes of performance issues.

This article explains how to deploy a full OpenStack stack using DevStack, a development and testing environment for OpenStack. The author needed an easy way to deploy and redeploy their OpenStack environment daily, so they created scripts that automate the process. The script `stack_setup.sh` prepares the local environment for DevStack installation, clones the Kilo release of DevStack into `/usr/local/src/devstack`, creates a stack user with the bundled create-stack-user.sh script from the tools directory, and sets up OpenStack notifications. After creating the new user, it sets defaults for admin, database, RabbitMQ, and Horizon passwords, as well as the service token to bootstrap Keystone. The final step is running the `stack.sh` script as the stack user, which handles the actual deployment of DevStack. The author also provides instructions for deploying on AWS and using Tugboat to automate the process on DigitalOcean.

This article discusses how Lithium, a social software provider, uses OpenStack as a private datacenter and monitors it with the help of Datadog. Monitoring is crucial for Lithium's infrastructure since failure can lead to significant issues such as inability to create demo environments or increased response times. Key metrics tracked by Lithium include the number of instances running, instances per project, available memory, VCPU available, and metric deltas. Datadog alerts are used to inform the right people about problems at the right time across various platforms. By using Datadog, Lithium has improved its infrastructure monitoring capabilities, allowing engineers to react quickly to issues with minimal customer impact.

Lithium, a leading social software provider, uses OpenStack as its private datacenter to host over 400 communities and various infrastructure services. Monitoring OpenStack is critical for Lithium's service platform, with failure being unacceptable due to the large number of production communities and infrastructure services hosted on it. Lithium engineers use Datadog to monitor OpenStack around the clock, correlating key metrics such as instances running, available memory, VCPUs available, and metric deltas to spot issues at a glance. They also set up advanced alerts for mission-critical metrics and have flexibility in informing the right people through various communication channels. Lithium adopted Datadog to gain deeper insight into their OpenStack deployment, adjust internal workflows, and react quickly to problems with minimal customer impact, saving time, money, and reputation.

Datadog introduces tiered alerts to enhance the effectiveness of its alerting system. This feature allows users to configure different severity levels for each metric, triggering notifications to specific individuals or teams via various communication channels based on the alert's urgency. Tiered alerts eliminate the need for multiple monitors and help reduce alert fatigue by ensuring that only relevant people are notified about critical issues. To set up a tiered alert, users can define threshold values and wrap their notification message and recipient list in specific tags to indicate the severity level of each alert or warning. This feature is available for all Datadog integrations and custom metrics.

Datadog has introduced a new feature called tiered alerts, which allows users to configure different severity levels for their metric alerts. This feature enables the user to trigger alerts that go to different people or via different channels depending on the alert severity. With this feature, users can set up alerts to send email or chat notifications for relatively low-urgency issues and page the engineer on call if the situation gets worse. The new feature works with any integration that supports regular metric alerts and reduces alert fatigue by ensuring that the right people receive the right alerts over the right channels. Users can try out this feature by navigating to the monitors page, choosing Metric as the monitor type, defining threshold values, and configuring notification settings.

The Apache Software Foundation, a decentralized community of developers producing open source software, needed an efficient monitoring suite for its many servers facilitating various services across multiple hosting providers. They sought a solution that was easy to deploy and set up, collected useful information, and had extensible monitoring options. Datadog, a hosted service, met these criteria by being quick to deploy, offering default graphs and collected data, and having pre-built integrations for numerous technologies. Real-world examples demonstrate how Datadog has helped the Apache community identify resource abusers, backend capacity problems, and monitor RDMS leader/follower replication. Overall, Datadog has provided valuable and actionable information that has improved their work and prevented potential service disruptions.

The Apache Software Foundation, a decentralized community of developers producing over 100 open-source software projects, faced challenges in setting up monitoring for their servers. They needed a monitoring suite that was easy to deploy and set up, collected useful information, had extensible monitoring options, and did not require significant resources from the organization itself. After evaluating different options, they adopted Datadog, which satisfied all their criteria. The integration with various technologies, including httpd, LDAP, and Puppet, made it an ideal choice for their needs. With Datadog, the Apache Infrastructure Team was able to identify issues such as resource abuse, backend capacity problems, and replication delays, allowing them to take proactive measures to prevent service disruptions and improve overall performance.

Datadog was first started working on in late 2010 due to the expected significant changes in IT infrastructure by 2020, driven by Agile processes and cloud infrastructure. The company's founders believed that a new kind of monitoring and analytics platform would be needed to address this change. Since its launch, Datadog has seen its thesis validated as companies of all sizes have adopted cloud-based solutions, leading to an explosive growth in the number of infrastructure units, frequency of code changes, number of engineers interacting with the infrastructure, and diversity of platforms and tools used. As a result, the magnitude of the monitoring problem has drastically changed, and Datadog was built to scale across these dimensions, predicting 2016 to be a banner year for cloud adoption and the era of Monitoring at Scale.

A new integration with Ansible has been introduced to join other automation integrations like Chef and Puppet. This integration allows users to monitor their infrastructure automation tasks, such as provisioning, deployment, and configuration management. Real-time reports on server runs, tracking of key performance metrics, setting alerts for failed tasks, and correlating events with performance metrics from any part of the infrastructure are some of the features provided by this new integration. Additionally, Ansible can help automate monitoring by installing and configuring the Datadog Agent on each host, allowing monitoring to scale effortlessly along with the infrastructure.

Connectifier, a company that helps place candidates into new jobs, has been using Datadog to optimize its infrastructure and troubleshoot performance issues. When the company's application began slowing down due to an issue with MongoDB's WiredTiger database, they used Datadog to identify the problem and implement a fix. The open-source model of Datadog allowed Connectifier to contribute directly to its development by submitting pull requests and receiving prompt feedback from the Datadog team. Overall, the company found Datadog to be a valuable tool for monitoring and analyzing their infrastructure's performance.