March 2020 Summaries

Watchdog Insights is a feature developed by Datadog that helps monitor highly distributed applications by automatically detecting errors, high latency, and root causes of issues. It watches indexed spans returned by Trace Analytics queries in real-time, scans all available tags, and identifies tagged objects associated with higher-than-usual error rates or latency. This enables devops teams to quickly identify issues and dig deeper into relevant traces, services, infrastructure components, and code profiles to discover possible root causes. Watchdog Insights is embedded in Datadog's Trace Analytics UI, allowing users to investigate indexed spans and outlier data in a single view, and provides features such as drilling down into tags, identifying outliers in multi-tenant architecture, spotting feature flag performance degradation, and leveraging AIOps-driven features for log management, metric correlations, and log patterns.

Datadog, a company that provides monitoring and analytics for cloud-based applications, has been actively contributing to the fight against COVID-19. Employees have donated time and resources to organizations like CovidNearYou and Folding@home, while also sponsoring hackathons dedicated to tracking the virus's spread and researching new treatments. Datadog has also empowered individual employees to support their communities in various ways. In addition, some Datadog employees have built public dashboards using official COVID-19 data sources to help organize and parse information. These efforts aim to provide a more realistic picture of the virus's spread and assist scientists in developing new treatments, including research into COVID-19. Datadog is also encouraging others to get involved by sharing information about organizations that could use support in fighting the pandemic. They have set up an email address ([email protected]) for people to suggest organizations they believe need assistance.

Datadog, a company focused on software development and developer tools, has been actively contributing to efforts against COVID-19. The company is donating time and resources to organizations such as CovidNearYou, Folding@home, and sponsoring hackathons dedicated to tracking the spread of the virus and researching new treatments. Datadog employees have also built public dashboards to organize and parse official sources of COVID-19 data, including a dashboard using WHO and ECDC data, which provides day-to-day trends and insights into infection growth rates. Additionally, Datadog has empowered individual employees to contribute to their communities and is supporting the development of web applications such as CovidCommitment.org, which aims to flatten the curve of COVID-19 infections by engaging individuals in taking steps to stop the spread of the virus.

Apache Flink is an open source framework for real-time and batch data stream processing, written in Java and Scala. It offers robust libraries and layered APIs for building scalable, event-driven applications for data analytics and more. Datadog has introduced a new integration with Apache Flink to provide deep visibility into Flink deployments. The integration allows users to visualize metrics such as job uptime, buffer usage, and checkpoint count in an out-of-the-box dashboard. It also enables the correlation of logs with metrics for effective troubleshooting of performance issues. Flink processes dataflow programs using directed acyclic graphs (DAG) consisting of streams and transformations. Monitoring checkpoints, buffer pool usage, and JVM resource usage can help ensure high performance and availability of Flink applications. Datadog's integration provides a comprehensive view of these metrics alongside other components of the Apache ecosystem, allowing users to identify and diagnose performance bottlenecks effectively.

Apache Flink is an open-source framework for stateful processing of real-time and batch data streams. It offers robust libraries and layered APIs for building scalable, event-driven applications for data analytics, data processing, and more. The integration with Datadog provides visibility into Flink deployments, allowing users to visualize metrics such as job uptime, buffer usage, and checkpoint count in an out-of-the-box dashboard. Flink achieves fault tolerance by creating checkpoints to roll back to previous states and stream positions in the event of a failure. Monitoring the number of successful and failed checkpoints, along with the time taken to complete a checkpoint can help ensure that Flink applications are always available. The integration also helps users effectively handle backpressure to ensure high performance by identifying root causes such as insufficient resources or network channel oversubscription. Additionally, it provides an overview of JVM resource usage for JobManagers and TaskManagers to help diagnose performance bottlenecks. With Datadog's integration, users can get comprehensive visibility into their Flink deployments alongside other components of the Apache ecosystem, and more than 850 other technologies.

VMware vSphere is a server virtualization platform that enables organizations to manage and scale virtual machines effectively. Datadog's enhanced vSphere integration allows users to monitor real-time metrics and events from various components, including clusters, datastores, and datacenters. The out-of-the-box dashboard provides key resource and performance metrics for ESXi hosts, VMs, and datastores. Users can also customize the dashboard to include metrics from other supported technologies running on vSphere. Datadog's integration supports both real-time and historical vSphere performance metrics, allowing users to track memory ballooning, monitor datastore disk usage, and optimize CPU utilization across clusters. Additionally, fine-tuning metric collection helps reduce load on the vCenter Server while still providing visibility into essential data.

Scylla is an open-source database alternative to Apache Cassandra, designed to offer higher throughput, low latency, and high availability for real-time applications. Implemented in C++ rather than Java, Scylla provides better control over low-level operations and avoids latency issues associated with garbage collection. Its shared-nothing architecture assigns a single thread per core to enhance performance scalability. Recent integration improvements allow users to monitor Scylla clusters' health, visualize key metrics, and set alerts for performance issues. Scylla maintains fault tolerance using replication factors and consistency levels, and users can track resource utilization and optimize database operations to prevent query timeouts and manage disk space effectively. By leveraging Datadog integration, users can gain comprehensive insights into Scylla's performance alongside other technologies, ensuring efficient operation and resource use.

This article discusses troubleshooting NGINX 502 Bad Gateway errors when using Gunicorn as the application server for Python applications. The common causes of these errors include Gunicorn not running, NGINX being unable to communicate with Gunicorn, and Gunicorn timing out. To resolve these issues, the article provides guidance on checking if Gunicorn is running, ensuring that both Gunicorn and NGINX are configured to use the same socket, and adjusting timeouts as necessary. Additionally, it emphasizes the importance of collecting logs and application performance monitoring (APM) data for effective troubleshooting.

Datadog provides critical context for troubleshooting issues across any dimension of your environment by applying best practices for tagging your systems and monitors. By using monitor tags, you can efficiently organize and analyze all your monitoring data, set up automated multi-alerts to streamline alerting workflows, and quickly find specific monitors in your account. Monitor tags add dimensions to your monitors, allowing you to filter, aggregate, and visualize them just like any other kind of monitoring data. When used judiciously, monitor tags help you effectively organize your monitors and streamline the way you manage and utilize your monitors, making it easier to troubleshoot issues. By tagging your monitors with relevant metadata, you can quickly resolve issues, reduce mean time to resolution, and minimize potential impact on your customers.

This post is part one of a four-part series about monitoring the health and performance of Pivotal Platform (formerly known as Pivotal Cloud Foundry), focusing specifically on the Pivotal Application Service runtime. The Pivotal Platform is now part of VMware Tanzu following VMware’s acquisition of Pivotal in late 2019. The Pivotal Application Service (PAS)—a distribution of the open source Cloud Foundry Application Runtime—is one of two runtime environments available for Pivotal Platform, with the other being Enterprise Pivotal Container Service (PKS). The platform is aimed at enterprise users and offers additional features and services from Pivotal and third parties for installing and operating Cloud Foundry as well as to expand its capabilities and make it easier to use. The primary components of a typical Pivotal Platform cluster using PAS are: BOSH/the Ops Manager (deployment automation), the User Account and Authentication server (identity management), the Gorouter (application and system routing), the Cloud Controller (application staging and running), Diego (application execution and runtime), and Loggregator (logs and metric aggregation). Monitoring Pivotal Platform is vital for ensuring that applications are running optimally. Key metrics operators will want to collect and monitor in Pivotal Platform include system-level resource metrics, platform metrics from PAS components, application logs, and metrics coming from installed services.

In part two of this three-part series on monitoring your Pivotal Platform deployment, we will take a closer look at some key metrics that can provide insight into the performance and health of various components within the platform. We will cover metrics related to the Cloud Controller, Diego, BBS, Locket, Cells/Rep, Route-Emitter, and Loggregator. The following are some of the most important Pivotal Platform metrics for monitoring component availability, utilization, and performance: |Component |Metric Name |Description |Metric type |Cloud Controller |ccdb.indexingPassesCompleted |Total number of indexing passes completed by the Cloud Controller database |Work: Performance |ccdb.indexingPassDuration |Average time (in nanoseconds) for an indexing pass to complete in the Cloud Controller database |Work: Performance |Diego |diego.cellsTotal |Total number of Diego cells registered with the BBS |Other |diego.tasksRunningTotal |Total number of tasks currently running on all Diego cells |Resource: Utilization |diego.lrpInstancesRunningTotal |Total number of LRP instances currently running on all Diego cells |Resource: Utilization |Auctioneer |auctioneer.LRPAuctionsStarted |Number of times the Auctioneer has started an auction for placing LRPs on cells |Work: Performance |auctioneer.TaskAuctionsStarted |Number of times the Auctioneer has started an auction for placing tasks on cells |Work: Performance |BBS |bbs.convergenceLRPDuration |Total time (in nanoseconds) that the BBS takes to run a convergence pass |Work: Performance |Locket |locket.<component>.LockHeld |Indicates whether a component lock is held; returns 1 if held, 0 if not |Other |cell.Rep |rep.ActivePresencesTotal |Total number of active cell presences registered with the BBS |Resource: Availability |Loggregator |doppler.dropped |The lifetime number of messages dropped by the Doppler without being delivered to a downstream component |Resource: Saturation These metrics provide insight into the overall health and performance of your Pivotal Platform deployment, as well as the status of individual components within that deployment. By monitoring these metrics, you can gain valuable information about how effectively your platform is operating and whether it needs scaling up or other adjustments to ensure optimal performance for developers and end users. In part three of this series, we will demonstrate several methods of collecting Pivotal Platform metrics and logs. We will cover both how to tap the Firehose stream of deployment metrics and application logs, and how to utilize a syslog drain to view component system logs.

In this three-part series, we have explored how to monitor your Pivotal Platform (PAS) deployment using native tools as well as third-party services. We began by discussing the built-in monitoring capabilities of PAS and its various components, including Firehose, Loggregator, Healthwatch, Metrics Registrar, and syslog drains. In Part 2, we looked at how to use these native tools to collect logs and metrics from your deployment. We also examined the role that third-party nozzles can play in extending PAS’s monitoring capabilities by ingesting Firehose data into managed services for processing, visualization, and storage. In this final part of our series, we will explore how you can integrate Pivotal Platform with Datadog to aggregate the full range of deployment and application metrics, as well as application and system logs, so that you can get deep visibility into your entire deployment in a single platform. Datadog’s Pivotal Platform integration enables operators and developers to collect Pivotal Platform deployment metrics and logs for use with Datadog’s powerful visualization, analytics, and alerting features. In this final part of this series, we’ll show you how you can integrate Pivotal Platform with Datadog to aggregate the full range of deployment and application metrics, as well as application and system logs, so that you can get deep visibility into your entire deployment in a single platform. Acknowledgments We wish to thank Amber Alston, Katrina Bakas, Matt Cholick, Jared Ruckle, and the rest of the Pivotal team for their technical review and feedback for this series.

This post discusses how to use Datadog to monitor the health and performance of Pivotal Platform, formerly known as Pivotal Cloud Foundry. It covers using Datadog's Cluster Monitoring tile for operators to collect metrics from Pivotal Platform components and system-level resource metrics from underlying VMs. Additionally, it explains how developers can use Datadog's Application Monitoring tile to monitor the status and performance of their applications running in Pivotal Platform by collecting custom metrics, distributed traces, and logs. The post also details how to collect system logs with Datadog and forward them to a syslog server for further processing. Finally, it introduces Datadog's Logging without Limits™ approach, which allows users to ship all their logs without worrying about missing data or limitations on log volume.

Domain resolution is now generally available in Network Performance Monitoring to provide enhanced visibility into endpoints within and outside the network. This feature translates temporary IPs into stable domain names, allowing users to quickly pinpoint dependencies to cloud and external endpoints like Amazon S3, load balancers, and more. It also helps isolate connectivity and latency issues in traffic to external endpoints where an agent cannot be installed, such as third-party platforms and APIs. Domain resolution is automatically enabled for Network Performance Monitoring with the latest release of the Datadog Agent.

Yield is increasingly difficult as networks become more complex due to cloud migration and container deployment. To address this, Datadog has introduced domain resolution in Cloud Network Monitoring, which extracts meaning from network endpoints by translating temporary IPs into stable, easily recognizable domain names. This feature provides enhanced visibility into endpoints within and outside the network, enabling users to pinpoint dependencies to cloud and external services, isolate connectivity issues, debug traffic to external endpoints, and pinpoint security issues such as malicious domains and typosquatting attacks. With this feature, users can monitor cloud provider services, view traffic through load balancers, debug traffic to external endpoints, and more, ultimately gaining deeper insights into their network connections.