August 2018 Summaries

This post demonstrates how to forward Rails application logs to Datadog and utilize its features like faceted log search, analytics, custom processing pipelines, and log-based alerting. To set up Datadog for collecting application logs, install the Datadog Agent on your host using an API key from your Datadog account. Create a new directory and YAML file in the Agent's conf.d directory to store the configuration necessary for Rails log collection. Restart the Agent and check its status with appropriate commands. Datadog enables you to explore logs, create facets, and perform searches based on attributes from individual logs. It also provides integration pipelines that capture and parse standard Ruby logs as well as enhance logs in JSON format. Custom processing pipelines allow users to create their own processing rules or modify existing ones for more control over log customization. Log-based alerting can be set up by creating a new alert based on attributes from processed logs, allowing you to get notified when specific events occur within your application. Datadog supports 650+ other integrations and provides a comprehensive monitoring solution for Rails applications, including infrastructure metrics from hosts, databases, and web servers. To start using these features, sign up for a free Datadog trial or refer to the documentation if you're already using Datadog.

You can configure logging for your Rails application to forward logs to Datadog and track behavior with faceted search, custom processing pipelines, and log-based alerting. To start, install the Datadog Agent on your host and create a new directory and YAML file in the Agent's conf.d directory to store the configuration necessary to set up Rails log collection. Restart the Agent and check its status after adding the necessary configuration. Once set up, you can explore your logs in the Log Explorer, perform searches based on facets, create custom processing pipelines, and set up alerting for specific events. With Datadog's log management tools and built-in Ruby integration, you can collect, manage, and monitor your Rails application logs in one place, correlating data with logs and metrics from every part of your infrastructure.

Rails applications come with six different log levels: debug, info, warn, error, fatal, and unknown. Each level defines how much information your application will log, ranging from the most verbose (debug) to the least informative (unknown). By default, Rails applications generate logs at the debug level for all environments, including production. However, this can be adjusted by editing the applicable environment file. The info log level provides a good balance between diagnostic information and verbosity, making it suitable for development or test environments. Lograge is a library that transforms Rails logs into a format that is easier to parse and read, such as JSON, which can help manage the complexity of production applications generating logs for multiple processes at the same time. Custom logging can also be added using the ActiveSupport::Logger class to further enhance the usefulness of your logs.

Datadog introduces Auto Smoother, an automatic smoothing function that helps users identify trends in their metrics by removing noise while preserving the shape of timeseries data. The algorithm is inspired by Stanford's ASAP (Automatic Smoothing for Attention Prioritization) and uses a moving average to calculate the optimal window size based on two properties: roughness and kurtosis. Auto Smoother offers several advantages over traditional smoothing functions, including automatic adjustment of the smoothing window in real-time and consistent smoothing across multiple timeseries for easy comparison. The feature is now available in Datadog, allowing users to extract valuable insights from noisy data.

Datadog's Auto Smoother is a smoothing function that helps identify trends in noisy timeseries data, particularly in high-scale infrastructure and applications. It automatically chooses the optimal window size to smooth the timeseries, adapting to changes in noise levels as new data is collected. This algorithm uses a combination of roughness and kurtosis measures to ensure that the smoothed series preserves large-scale trends while preventing oversmoothing. Auto Smoother provides several advantages over traditional smoothing functions, including automatic parameter selection, real-time adaptation, and consistency across different time ranges and hosts. It can be used in conjunction with other algorithms to highlight important abnormalities in metrics, making it easier to extract valuable signals from noisy timeseries data.

The latest version of Datadog's Agent now supports Prometheus metrics through the OpenMetric exposition format. This integration allows users to monitor Prometheus metrics alongside other data collected by Datadog's built-in integrations and custom instrumentation libraries. Users can configure the Datadog Agent to scrape available endpoints for exposed metrics, visualize their data in comprehensive graphs and dashboards, and create custom checks for additional control over metric collection. This integration also works seamlessly with Kubernetes components that expose metrics via the OpenMetrics exposition format, such as CoreDNS and kube-dns, and supports Datadog's Autodiscovery feature to automate the collection of Prometheus metrics in a cluster.

The Datadog Service Map is a tool that visualizes an application's topology by decomposing it into its component services and drawing observed dependencies between these services in real time. This helps identify bottlenecks, understand data flow through the architecture, and isolate service affinity and dependencies at scale. By automatically capturing interdependencies of hundreds or thousands of services, Datadog Service Map enables users to focus on a single service, trace calls across distributed services and hosts, and investigate individual traces from that service. It also integrates data from APM, infrastructure monitoring, and log management for quick remediation of performance issues.

The Datadog Service Map is a visual tool that helps developers understand the topology of their application, identify bottlenecks, and troubleshoot issues. It decomposes the application into its component services, displays dependencies in real-time, and provides insights into data flows through the architecture. The map automatically clusters related services together, allowing users to focus on specific services and dependencies. By integrating with Datadog APM, the Service Map provides a single pane of glass for investigating performance issues, including tracing calls across distributed services, hosts, and infrastructure metrics. This enables teams to quickly isolate and remediate problems without relying on outdated whiteboard diagrams or complex documentation.

The text discusses how the company uses Apache Kafka as their messaging persistence layer for handling large amounts of data. They have developed a collection of tools called Kafka-Kit, which includes topicmappr and autothrottle, to handle partition to broker mappings, failed broker replacements, storage based partition rebalancing, and replication auto-throttling. The two primary tools are designed for data placement and replication auto-throttling. They also explain their capacity planning methods and how they use Kafka-Kit in this process. The long term goal is to continue refining the coordination between scaling resource pools and mapping capacity to the right place at the right time, making data movement and recovery hands-off.

Portworx, a leading provider of Kubernetes storage solutions, has announced an integration with Datadog to enhance monitoring capabilities for its customers. The integration allows users to correlate performance metrics from Portworx with data from infrastructure and application components, helping identify performance bottlenecks and enabling appropriate resource provisioning. With the new integration, Datadog collects cluster-level metrics from Portworx, allowing users to monitor health, capacity usage, and set alerts for indicators like quorum or capacity used. Additionally, machine learning features in Datadog enable automatic notifications if a single node is behaving differently than others. The integration also provides capacity planning tools, such as host maps and forecasting, to help track overall usage against available resources. Furthermore, developers can monitor per-volume I/O, throughput, and latency metrics in context with other infrastructure and application performance data. This comprehensive monitoring solution helps users troubleshoot performance issues more effectively and maintain optimal resource allocation for their Portworx clusters and nodes.

Kafka-Kit` is a collection of tools developed by Datadog for managing Kafka clusters. It includes `topicmappr`, which handles partition to broker mappings, failed broker replacements, storage based partition rebalancing, and replication auto-throttling. The tool's primary inputs are topics and brokers, and it provides deterministic output and minimizes movement broker replacements. It also exposes a selection of partition replica placement strategies, including the `count` strategy, which ensures leadership is maximized among brokers and partitions are evenly distributed. Another strategy, `storage`, prioritizes even storage utilization across brokers while satisfying locality constraints. The tool can be used for capacity planning, data mapping, and making data movement and recovery hands-off. It's designed to work with Kafka 0.10.1 and has features such as dynamic throttle adjustment and automated recovery.

Portworx provides solutions for Kubernetes storage, reducing costs and downtime for mission-critical applications. The company's CLI and UI tools manage and monitor cluster health, while a new integration with Datadog enables correlated performance metrics across infrastructure and application components. This allows users to pinpoint bottlenecks and provision resources accordingly. With the integration, customers can monitor cluster-level metrics, track capacity usage, and understand performance issues in context, enabling proactive planning and troubleshooting. The solution is trusted by major organizations such as Comcast and Verizon, and offers features like timeseries monitoring, log analytics, and machine learning capabilities to support scalability and reliability. To get started with the new integration, users can check out the documentation on Portworx's website.

Microsoft's Internet Information Services (IIS) is a web server that comes bundled with Windows and has numerous extensibility features. It allows for the use of various backend technologies like Flask, Node.js, and ASP.NET. IIS also supports modules to perform tasks such as URL rewriting and programmatic load balancing. Monitoring key metrics in IIS can help ensure its availability and performance. These include HTTP request metrics, HTTP response metrics, availability metrics, and resource metrics. By tracking these metrics, administrators can identify potential issues and optimize the server's configuration for better performance.

In this article, the author discusses how to use built-in IIS monitoring tools to access and graph performance counters, configure logging in IIS, and query your logs with Microsoft's Log Parser Studio. They also explain how to use a diagnostic tool to investigate memory leaks and high CPU utilization in your application pools and worker processes. The author covers the key points of using PowerShell scripts, Performance Monitor, and the IIS Administration API to access performance counters. Additionally, they discuss configuring web service logs and querying them with Log Parser Studio. Finally, they introduce DebugDiag as a tool for analyzing memory dumps and integrating with IIS to investigate performance issues.

In this tutorial, we learn how to use Datadog's IIS and WMI integrations to monitor an Internet Information Services (IIS) deployment. We start by setting up the Datadog Agent on our Windows hosts, which collects metrics and logs from our IIS environment. Then, we configure IIS monitoring by enabling the IIS integration in the Agent configuration file and specifying the host and sites we want to monitor. Next, we set up the WMI integration to pull data from any formatted performance counters, including those for IIS worker processes. We create a custom dashboard that displays metrics from three different sources: HTTP.sys request queue, Windows processes, and the Web Service performance counter. Finally, we configure IIS log collection in the Agent configuration file and use Datadog's Log Explorer to analyze our IIS logs. By using Datadog for IIS monitoring, we can visualize, compare, and correlate metrics from all components of our IIS environment in one place, set up automated alerts, and gain full visibility into our deployment.

Datadog provides a comprehensive approach to IIS monitoring by integrating with out-of-the-box dashboards, automated alerts, and log analytics. The Datadog Agent collects metrics automatically and parses IIS logs without manual querying, providing full visibility into the deployment. Users can configure the Agent to collect logs from IIS, parse them, and send them to Datadog for analysis. With Datadog, users can graph and analyze IIS log data, set up automated alerts for performance issues, and visualize key metrics in a centralized platform. The integration allows for monitoring of multiple sites and worker processes, as well as real-time alerting when health and performance issues arise. By using Datadog's IIS integration, users can gain full visibility into their IIS deployment and make data-driven decisions to optimize performance.

The text discusses various methods to monitor and analyze IIS performance counters, web service logs, and application pools. It explains how to access performance counters via PowerShell scripts, the Performance Monitor, and the IIS API, as well as configure logging in IIS and query log files with Log Parser Studio. Additionally, it introduces the Debug Diagnostic Tool (DebugDiag) for analyzing memory dumps and identifying issues such as memory leaks and high CPU utilization. The text concludes by highlighting the need for automation of IIS monitoring to gain insights into server performance and provides a mention of Datadog's IIS integration as a comprehensive solution.