July 2015 Summaries

John Sheehan, Co-founder and CEO of Runscope, discusses the importance of API performance monitoring in microservices architecture with 60+ APIs. He introduces a turn-key integration between Runscope's API performance monitoring and testing tools and Datadog's powerful interactive dashboards and real-time notifications. This integration allows users to easily create tests, monitor APIs for uptime, performance, and correctness, and correlate API performance with infrastructure data. The example provided demonstrates how measuring API performance against IT infrastructure data can help quickly identify the cause of API issues related to infrastructure. Keen IO uses this integration to monitor average query durations and integrate mission-critical API data into their existing Datadog dashboards for fine-grained control.

Runscope provides API performance monitoring and testing tools, offering a suite of features to help companies monitor their APIs for uptime, performance, and correctness. With its integration with Datadog, Runscope enables users to correlate API health with infrastructure data, providing targeted insights into API performance issues. By connecting Runscope tests to Datadog accounts, users can easily integrate API monitoring into their existing dashboards and alerts, gaining fine-grained control over their mission-critical API data. The integration allows for real-time notifications and easy correlation of API performance issues with CPU load or other infrastructure metrics, enabling teams to quickly identify and resolve problems before they impact the system.

Part 2 of a three-part series on Varnish monitoring delves into the tools Varnish Cache offers for precise monitoring, focusing primarily on varnishstat and varnishlog. Varnishstat provides real-time access to various performance metrics, such as cache hits, resource consumption, and thread information, and can be customized to display specific metrics. It is useful for quick health checks, but integrating its data with a dedicated monitoring service is recommended for comprehensive performance tracking and alerts. Varnishlog, on the other hand, is valuable for debugging and tuning configurations by offering detailed logs of individual requests. Specialized tools like varnishtop, varnishhist, and varnishsizes build on varnishlog to analyze frequently occurring entries, latency, and request sizes, respectively. The choice of metrics to monitor should align with specific use cases and the importance of the insights they provide. The article hints at an upcoming discussion on using Datadog for Varnish monitoring in the series' final installment.

This post, the final in a three-part series on monitoring Varnish, details using Datadog to integrate and visualize Varnish metrics, ensuring continuous and effective monitoring. It begins by verifying the proper functioning of Varnish and varnishstat, followed by installing and configuring the Datadog Agent to collect and report metrics. The setup process includes creating a configuration file, restarting the Agent, and verifying integration success. Once the Datadog Agent is operational, users can access a Varnish dashboard for key metrics and create custom dashboards by adding other relevant metrics. The post also guides setting up alerts for specific metrics, such as Varnish’s dropped client connections, using threshold-based alerts to notify teams via channels like Slack and PagerDuty whenever metrics exceed predefined values. The goal is to enhance visibility into web infrastructure and automate alerts tailored to specific organizational needs, with a call to action for those without a Datadog account to sign up for a free trial.

Varnish Cache is a web application accelerator designed to accelerate content-rich and dynamic websites by using caching HTTP reverse proxying. It stores often-used assets for faster retrieval, reducing the load on the backend server. Monitoring Varnish's key performance metrics, such as client connections, cache hits, thread metrics, and backend metrics, can help ensure its proper operation and performance. Key metrics to monitor include client requests per second, dropped client connections, cache hit rate, LRU Nuked objects, worker threads related metrics, and backend connection failures or unhealthy backends. These metrics provide visibility into the effectiveness of the cache, thread pools, and backend connections, helping identify areas for tuning to improve performance.

Joe McCourt discusses the introduction of Change Graphs, a new visualization tool designed to help users easily identify and measure differences between identical timeframes, such as comparing data from today with data from the same time yesterday. This tool is particularly useful for detecting changes related to seasonality and is implemented in visualizations on Datadog dashboards, which can be set up through a simple drag-and-drop or API process. Change Graphs focus on highlighting changes in web traffic, login rates, errors, latency, and other key system health indicators, offering an advantage over traditional graphs that may not clearly show significant changes. McCourt emphasizes the immediate visibility of changes, such as increased server-side errors across availability zones, and invites users to try this feature by signing up for a free trial of Datadog.

The enhanced Datadog monitor status page provides a faster and more comprehensive overview of automated monitors, allowing users to quickly assess the situation and take action. The redesigned page brings together a wealth of useful data, including timeseries graphs, detailed histories, evaluation graphs, alert notifications, and stream events. By accessing this page, users can jump-start investigations, explore monitor trends, identify issues, and isolate problematic hosts or groups. The updated status page also includes uptime statistics and provides a centralized location for all monitor-related data, enabling better understanding of monitors and the data they track.

Datadog's Process Check feature allows users to monitor the resources used by each process on their hosts, including CPU and memory usage, number of threads, and I/O. This helps identify processes that are hogging resources or leaking memory. Users can set up alerts for when specified thresholds are exceeded. The integration supports tagging and can be enabled with a list of processes to monitor. Existing Datadog customers can set up Process Check, while new users can try it out during a free trial.

Datadog's Process Check feature allows you to monitor the resource consumption of individual processes, enabling you to identify and take action on process-related issues. This feature provides detailed insights into CPU, memory, I/O, thread count, and other metrics for each process or group of processes. With Process Check, you can set up automated alerts when specified thresholds are exceeded, ensuring prompt identification and correction of resource-intensive processes. The integration is easily enabled by specifying the processes to monitor, and it leverages Datadog's existing tagging capabilities to provide a tailored view of relevant metrics.

In this post, Alexis Lê-Quôc discusses the importance of a structured approach to monitoring systems, emphasizing that effective monitoring extends beyond symptom detection to diagnosing root causes. Drawing from the experience of monitoring large-scale infrastructure and insights from experts like Brendan Gregg and Baron Schwartz, the article outlines a methodical process for diagnosing issues. It highlights the significance of three main types of monitoring data—work metrics, resource metrics, and events—to thoroughly understand system health and functionality. The post suggests starting investigations with work metrics of the highest-level systems to determine problems, then examining resource metrics if necessary, and considering any correlated events that might have caused the issue. It stresses the importance of pre-built dashboards for quick access to relevant data during outages and underscores a systematic framework for problem investigation, urging feedback from users on this approach.

The latest release (version 5.4) of the Datadog Agent includes profiling tools, known as "developer mode," to improve performance and efficiency. This feature is particularly useful for developers actively working on or contributing to the Datadog Agent code. Developer mode supports various metrics such as CPU usage, memory consumption, threads in use, network connections open, and total time to run configured checks. It can profile individual Agent Checks and enable full developer mode for profiling everything. The collected metrics are available in Datadog under the datadog.agent.* namespace and in collector.log files on Linux or Windows systems.

The Datadog Agent, a widely deployed open-source monitoring tool, has introduced a new feature called "developer mode" to improve its performance and efficiency. This feature provides profiling tools that allow developers to easily profile their code changes before submitting pull requests, reducing unnecessary GitHub conversations. The developer mode includes various metrics such as CPU usage, memory consumption, and network connections, which can be used to identify performance issues in the Agent's code or custom checks. Developers can enable this mode by adding a specific configuration option to the `datadog.conf` file or using the `--profile` command line flag, and the collected data will be available in Datadog under the `datadog.agent.*` namespace. Additionally, developers can contribute their new code or custom checks with profiled data, making it easier to track performance improvements.

Datadog allows you to monitor NGINX performance metrics alongside all your other systems on customizable dashboards. To implement comprehensive, ongoing NGINX monitoring, you need a robust monitoring system to store and visualize your metrics and logs, and to alert you when anomalies happen. You can easily create a comprehensive dashboard for monitoring your entire web stack by adding additional graphs with important metrics from outside NGINX. Datadog will process any log formatted as JSON automatically, turning key-value pairs into attributes that you can use to group and filter. To isolate this service's logs in the Log Explorer page, click my.nginx.service under the Service list in the sidebar. You can build alerts around key data collected in your logs, such as HTTP response codes and request processing time, that aren't available on the standard NGINX status page. Once Datadog is capturing and visualizing your metrics and logs, you will likely want to set up some alerts to automatically notify you when there are problems.

NGINX is a popular HTTP server and reverse proxy server that serves static content efficiently and reliably. It can also be used as a mail proxy, generic TCP proxy, or for caching and load balancing. Monitoring NGINX provides insights into resource issues within the server itself and problems in the web infrastructure. Key metrics to monitor include requests per second, server error rate, and request processing time. These metrics help identify spikes in incoming web traffic, errors, slow response times, and potential resource saturation. NGINX Plus offers additional metrics such as active connections by upstream server, 5xx codes by upstream server, and available servers per upstream group, which are relevant to a reverse proxy setup. By monitoring these metrics, users can gain visibility into the health and activity levels of their web infrastructure and identify potential issues before they become major problems.

Capturing NGINX metrics varies depending on whether you're using the open-source or commercial NGINX Plus version, and the specific metrics you need. Both versions can report metrics through status modules and logs, with NGINX Plus offering more comprehensive metrics via the ngx_http_api_module, including data in JSON format for better integration with external systems. Open-source NGINX provides basic metrics through a status page when the HTTP stub status module is enabled, while NGINX Plus offers additional metrics on bytes streamed and upstream systems. The logging module of NGINX allows for customizable access logs, capturing detailed data points like HTTP requests and response codes, which can be processed using tools like logstash and Fluentd. Monitoring tools such as Nagios and Datadog can integrate with these systems to provide continuous monitoring. Ultimately, the choice of metrics to monitor will depend on the specific needs and resources of an organization, balanced against the cost of implementation, with Datadog offering a unified integration to facilitate easy collection and monitoring of NGINX metrics.

Datadog has introduced a new integration to monitor PHP-FPM (FastCGI Process Manager) performance, which is widely used with web servers like Apache and NGINX. This integration provides a built-in screenboard to display metrics such as listen queue, active processes, and slow logs, allowing users to visualize and correlate these metrics with other application data. Users can also make comments, discuss specific metrics with team members, and set up custom service checks and monitors for their FPM instances. The integration supports tracking multiple FPM instances for redundancy and failover, recognizing tags associated with server instances to offer a comprehensive overview. Additionally, users can pinpoint when FPM is underpowered and adjust configurations accordingly. Datadog offers a customizable PHP-FPM service check, which can be easily set up to monitor the status of FPM instances, with the option to receive alerts based on specific criteria. A free 14-day trial is available for users to explore the benefits of this integration.

The text introduces a new integration that allows users to monitor Secure Shell (SSH) connections directly with Datadog. This integration enables monitoring of key SSH statuses and metrics, including SSH daemon availability, SFTP response times, and target's key fingerprint changes. It ensures immediate notification if any issues arise with SSH connections, even when exchanging with third-party services. The integration is available for Datadog customers and can be set up in minutes during a 14-day free trial for non-customers.

The company behind Datadog recently introduced a new integration that allows users to monitor Secure Shell (SSH) connections directly within the platform. This integration enables real-time monitoring of SSH stability, performance, and security, including alerts for issues such as unavailable SSH daemons, slow SFTP response times, and changes in target key fingerprints. The feature is designed to provide immediate notification when anything goes wrong with SSH connections, even if the endpoint is not under the user's control. Users can set up the integration for free or try a 14-day trial of Datadog to begin monitoring SSH in their environment.

Supervisor is a popular tool for managing long-running application processes. Its instances can become difficult to track, especially with multiple supervisord servers running in production. Datadog's supervisord integration provides real-time monitoring and alerting capabilities, allowing users to set up customizable checks and monitor process states, uptimes, and overall deployment status. This integration also enables the tracking of supervisord itself, ensuring that processes can be restarted when issues arise. With this integration, developers can gain a better understanding of their application's performance and identify potential problems more easily.