October 2018 Summaries

The addition of Agent check status information to Datadog's Infrastructure page allows users to monitor their infrastructure more effectively by identifying configuration issues in their agents and applications. This feature highlights problematic apps with a yellow color, indicating that at least one of its checks is not working properly. Clicking on the problematic app opens an inspector panel providing detailed information about the configuration issue. With this feature, users can quickly pinpoint and resolve issues, ensuring all integrations are reporting correctly. This feature is available for hosts with Agents v4.3+ installed.

The Datadog Cluster Agent is designed to efficiently gather monitoring data from across an orchestrated cluster, particularly beneficial for large Kubernetes clusters with hundreds or even thousands of nodes. It acts as a proxy between the API server and the node-based Agents, reducing load on the API server while still allowing valuable insights into the state of infrastructure. The Cluster Agent also implements the External Metrics Provider interface to expose metrics from Datadog accounts to Kubernetes, enabling automated use of Horizontal Pod Autoscaling based on real-time health and performance data.

Log Patterns is a new feature developed by Datadog that helps users analyze large volumes of logs in real time, grouping them into clusters based on common patterns for easier interpretation. This tool assists in identifying unusual occurrences and steering investigations during system outages. The Log Patterns view can be accessed from the Log Explorer by filtering logs based on service name, status, or other attributes, then clicking the "Patterns" button to collapse the full list of logs into groups. Each log grouping displays common snippets and highlights varying snippets across members. Users can drill down for more details or pivot directly from logs to related sources of data like APM and host-level metrics. The tool also helps in identifying low-value logs, reducing the number of logs indexed while ensuring access to important data.

The Datadog Cluster Agent enables autoscaling of Kubernetes applications in response to real-time fluctuations in any metric collected by Datadog. With the release of version 1.10, support for external metrics was introduced, allowing users to autoscale off of any metric from outside the cluster. The Datadog Cluster Agent has made it possible to autoscale Kubernetes workloads based on custom-built metric queries, giving users increased flexibility for certain use cases. To enable high availability for Horizontal Pod Autoscaling (HPA), the Datadog Cluster Agent allows users to fetch metrics from multiple regions and automatically failover if one of the endpoints is degraded, based on availability and latency.

The Datadog team developed Log Patterns, a new view that automatically analyzes logs in real-time, grouping them into clusters based on common patterns. This helps users quickly interpret their logs, identify unusual occurrences, and accelerate their investigation. The Log Patterns view can be used to understand the scope of an issue, drill down into specific log entries for more details, and analyze normal and abnormal patterns to get a full picture of the environment's state. It also allows users to refine their log management setup by identifying low-value logs and reducing indexing and monitoring with Datadog, while ensuring access to all necessary data.

Meghan Jordan discusses the importance of service level objectives (SLOs) in site reliability engineering and how Datadog enables teams to track, manage, and monitor their SLOs in one place. SLOs provide a framework for defining clear targets around application performance, which ultimately help teams provide a consistent customer experience, balance feature development with platform stability, and improve communication with internal and external users. Datadog's features allow users to search, sort, and filter all their SLOs, easily visualize the status of individual SLOs on application dashboards, and track real-time performance comparisons to established objectives. Additionally, Datadog provides a comprehensive view of SLOs across multiple products and teams, allows users to save queries as views, displays error budgets and SLO status at a glance, breaks down SLO status by tags, and visualizes SLO status on dashboards.

The Datadog Cluster Agent is a purpose-built tool designed to efficiently gather monitoring data from across an orchestrated cluster. It was developed to address the need for dynamic ways of monitoring containers and workloads in environments where orchestration technologies like Kubernetes, DC/OS, and Swarm manage container workloads both at the node level and at the cluster level. The Cluster Agent acts as a proxy between the API server and the node-based Agents, relieving the direct load on the API server while enabling node-based Agents to focus on collecting node-level data. It also implements the External Metrics Provider interface to expose metrics from your Datadog account to Kubernetes, allowing for automated Horizontal Pod Autoscaling based on real-time health and performance data. The Cluster Agent has been successfully tested on large Kubernetes clusters with hundreds of nodes and thousands of pods, reducing the load on API servers while providing valuable insights into cluster-level monitoring data.

Pivotal Platform's Firehose provides a stream of monitoring data, including application logs and component metrics. To access these metrics, developers can use the cf CLI or third-party nozzles such as Pivotal Healthwatch, which surfaces key platform metrics and status checks in a web UI. Other tools like PCF Metrics offer visualizations of application performance and resource utilization, while Metrics Registrar allows developers to emit custom metrics from their applications. System logs, including component syslogs, can be accessed through the Ops Manager or forwarded to an external service via rsyslog.

Pivotal Platform is a multi-cloud platform that abstracts away the process of setting up and managing an application runtime environment, allowing developers to focus solely on their applications and associated data. It provides scalability capabilities, with hundreds of thousands of application instances able to run across over a thousand host VMs. Pivotal Platform offers monitoring capabilities through various components, including BOSH, Ops Manager, User Account and Authentication, Gorouter, Cloud Controller, Diego, Loggregator, and Firehose. These components work together to provide real-time metrics and logs for the platform, enabling operators to monitor performance and capacity indicators, as well as ensure high availability and resilience. The platform supports multiple runtime environments, including Pivotal Application Service (PAS) and Enterprise Pivotal Container Service (PKS), and offers additional features and services from third-party providers.

Maxim Brown has written an in-depth guide on how to use Datadog to monitor the health and performance of a Pivotal Platform (formerly known as Pivotal Cloud Foundry) deployment, both for operators and developers. The guide covers installing the Datadog Cluster Monitoring tile to collect metrics and logs from Pivotal Platform components, and using the Datadog Application Monitoring tile to track custom metrics, traces, and logs from applications running on the platform. It also explains how to collect system logs with Datadog, including configuring syslog forwarding to send logs to a server or endpoint for processing and analysis. The guide provides detailed instructions on setting up Datadog for both operators and developers, and offers tips and best practices for getting started with using Datadog to monitor Pivotal Platform deployments.

The text discusses monitoring the health and performance of a Pivotal Platform (formerly known as Pivotal Cloud Foundry) deployment, focusing on the Application Service runtime. Key metrics to monitor include system CPU utilization, memory usage, disk space, request latency, and application instance counts. Monitoring these metrics can help identify potential issues before they become major problems, such as scaling bottlenecks or application crashes. The text also touches on the use of Pivotal Healthwatch, a service that provides additional metrics to help operators gauge the health and utilization of their deployment. Overall, monitoring key metrics is essential for ensuring the smooth operation of a Pivotal Platform deployment and preventing performance issues.