May 2021 Summaries

The adoption of Arm processors in cloud computing is growing, offering better performance, higher energy efficiency, and lower costs compared to x86-based predecessors. Oracle recently launched its new Oracle Cloud Infrastructure Ampere A1 Compute platform built on the Ampere Altra Arm processor. Datadog Agent for Arm enables comprehensive visibility into infrastructure by collecting metrics, traces, logs, and more from Oracle Arm-based Ampere A1 instances. This helps monitor migrations to Ampere A1 instances, rightsize instances to fit workloads, and identify and optimize resource-intensive parts of applications. Datadog's support for Oracle Arm-based Ampere A1 instances extends infrastructure coverage and allows monitoring all virtual machines, on-premise servers, containers, and more in one place.

Oracle has announced its new Oracle Cloud Infrastructure Ampere A1 Compute platform, built on the Ampere Altra Arm processor, and is now supporting it with Datadog's monitoring capabilities. This allows users to collect metrics, traces, logs, and more from their Oracle Arm-based Ampere A1 instances, providing comprehensive visibility into their entire infrastructure. By using Datadog, users can monitor migrations to Ampere A1 instances, rightsize their instances to fit their workloads, identify and optimize resource-intensive parts of their applications, and gain insights into how code modifications affect performance over time. With Datadog's support for Oracle Arm-based Ampere A1 instances, users can extend their infrastructure coverage and see how the performance of their infrastructure impacts the services running on it.

Amazon Elastic Container Service (ECS) is a managed compute platform for containers that enables users to run containerized workloads on either Amazon EC2 instances or the serverless Fargate platform without maintaining a control plane. AWS introduced ECS Anywhere, allowing customers with on-premise data centers to use ECS-managed control plane and integrate it with other AWS resources like Network Load Balancers. Datadog is a launch partner for ECS Anywhere, providing comprehensive visibility into containerized applications across various deployment environments. This integration helps monitor application performance, track health and resource utilization of workloads during migration to containers, and keep an eye on devices at the edge of networks.

Amazon Elastic Container Service (ECS) is a managed compute platform designed to be simple to configure with opinionated defaults, allowing users to quickly get started with containerized workloads on Amazon EC2 instances or the serverless Fargate platform. ECS Anywhere enables customers with on-premise data centers to run containerized environments using their existing infrastructure investments, ensuring data security and compliance while running containerized applications on edge devices. Datadog is a launch partner for ECS Anywhere, providing comprehensive visibility into containerized applications wherever they are deployed, allowing users to monitor on-premise applications as they migrate to containers, modernize on-prem workloads, track application performance, and ensure edge devices run as expected. With ECS Anywhere, users can get full visibility into their ECS clusters, no matter where they have deployed them, and leverage Datadog's 850+ integrations to monitor the entire on-premise or hybrid infrastructure.

Datadog's infrastructure list provides a centralized view of all hosts within an environment and integrates metadata and relevant metrics from across the platform to offer a comprehensive overview of each host. New features and visualizations have been added, making it easier for teams to navigate their infrastructure using tags, focus on specific hosts, and access more detailed information in other parts of Datadog. The updated list allows users to build powerful queries by filtering and grouping hosts based on tags, adjust the time range of the list, and view individual host's sidepanel containing relevant metadata, logs, infrastructure metrics, APM traces, network performance insights, security signals, and more. This feature is available for all customers without any additional installation required.

Datadog has improved its infrastructure list feature to provide a more comprehensive and customizable view of hosts across an environment, allowing users to easily filter, sort, and group hosts based on tags. The updated feature includes new visualizations, query optimizations, and sidepanel enhancements that enable users to quickly identify key data about their hosts, such as service status, agent version, and security signals. Additionally, the host's sidepanel provides a full picture of each host's health and performance by bringing together metadata, logs, infrastructure metrics, APM traces, network performance insights, and security signals from across Datadog. With these enhancements, users can now quickly identify issues and get more context around observed problems, making it easier to navigate their infrastructure and make data-driven decisions.

AWS Lambda extensions offer a streamlined integration of third-party tools, such as Datadog, with Lambda environments, facilitating the execution of custom code and monitoring tasks alongside functions. This collaboration allows users to collect and forward function logs, enhanced metrics, and traces directly to Datadog, complementing existing CloudWatch metrics and logs from other AWS services. The Datadog Lambda extension, available as a Lambda Layer or Docker dependency for Node.js and Python runtimes, simplifies monitoring by eliminating the need for additional infrastructure and data storage costs. Enhanced metrics, such as cost estimates and memory usage, are displayed in real-time on Datadog's platform, providing insights into function performance and identifying issues like high memory utilization. Tools like AWS SAM and Serverless Framework, along with Datadog's CloudFormation macro and Serverless plugin, further ease the installation process, making it simpler to collect and send telemetry data without added complexity.

The latest update from Datadog introduces a new dashboard layout that combines the best features of Screenboards and Timeboards, offering users a seamless editing experience. Built on a responsive grid, this new layout scales to any screen size and supports all Datadog widget types. It also offers intelligent positioning tools and editing workflows for easy customization. The update includes new interactions such as grouping widgets for faster edits, enabling users to compare data side-by-side and create dashboards more efficiently. Additionally, the dashboard experience now supports partial width groups, allowing users to place groups of different sizes next to each other. This latest feature aims to enhance visualization and correlation of telemetry data across various platforms in a single, user-friendly space.

Pending pods in Kubernetes are normal and indicate that the scheduler is working on assigning them to nodes. However, if a pod remains in this state for an unusually long period of time, it may be due to issues with node availability or scheduling predicates. In this post, we covered several reasons why the Kubernetes scheduler might encounter difficulties with placing Pending pods, including: 1. Taints and tolerations: If a node has taints that are not tolerated by any of its assigned pods, it will remain unschedulable until those taints are removed or tolerated by new pods. 2. Node selectors: If a pod’s node selector does not match the labels on any available nodes, it will remain Pending. 3. Resource requests and limits: If there are no nodes in the cluster that can satisfy a pod’s resource requests, it will remain Pending until resources become available or the pod is modified to request fewer resources. 4. PersistentVolumeClaims: If a pod’s PersistentVolumeClaim cannot be bound to a compatible PersistentVolume due to scheduling conflicts (e.g., because the associated node and volume are located in different zones), it will remain Pending. 5. Local volumes: If a pod’s PersistentVolumeClaim is still bound to local storage on an unavailable node, it will remain Pending until that node becomes available or the pod is deleted and recreated with a new PersistentVolumeClaim. 6. Inter-pod affinity and anti-affinity rules: These rules define where certain pods can (and cannot) get scheduled based on other pods that are already running on nodes. If these rules prevent a pod from getting scheduled, it will remain Pending until the scheduling conflict is resolved. 7. Rolling update deployment settings: During a rolling update, Kubernetes attempts to progressively update pods to reduce the likelihood of degrading the availability of the workload. However, if the number of desired pods is less than four and the default values for maxUnavailable and maxSurge are in effect, or if the maxUnavailable threshold is reached, a rolling update may need to pause as it hits that threshold. To troubleshoot Kubernetes Pending pods, you can use kubectl describe to get more details about the scheduling predicates for the affected pods and events from your clusters. You can also set up automated alerts to detect issues in your workloads and auto-detect stories generated by Datadog Watchdog to help identify abnormalities in your clusters.

The new Notebooks API by Datadog allows users to create and update notebooks programmatically as part of their existing workflows. This feature enables automatic generation of notebooks on new services and monitors, keeping notebooks up-to-date with service evolutions, and creating offline backups for future reference. The API can be integrated into command lines or setup scripts, and can also be chained to other Datadog APIs for seamless monitoring workflows.

A Pending pod in Kubernetes is unable to get scheduled on any node because the scheduler cannot find a suitable node that meets its scheduling requirements. This can happen due to various reasons such as node-based scheduling constraints, pods' requested resources exceeding allocatable capacity, PersistentVolume-related issues, or rolling update deployment settings. To debug Pending pods, it's essential to understand how Kubernetes makes scheduling decisions and evaluate predicates in a specific order. Node-based scheduling constraints, including readiness and taints, are checked first, followed by pods' requested resources exceeding allocatable capacity, PersistentVolume-related issues, pod affinity or anti-affinity rules, and rolling update deployment settings. By analyzing the "Events" section of the `kubectl describe` output and looking at the "Tolerations" and "Events" sections, it's possible to identify why a Pending pod is not getting scheduled. Additionally, monitoring Kubernetes with tools like Datadog can help detect abnormal scheduling issues and provide visibility into critical problems. To troubleshoot Kubernetes Pending pods, it's recommended to start by checking node conditions, node labels, and taints, as well as evaluating rolling update deployment settings.

The new dashboard layout introduced by Datadog offers a responsive grid-based design that scales to any screen size, allowing users to easily build dashboards for monitoring and troubleshooting purposes. The new layout features intelligent positioning tools and editing workflows, making it easy to place widgets of any type on the grid, streamline workflow with editing shortcuts, and perform bulk actions on widget groups. Additionally, the new layout supports partial width groups, which enables users to create dashboards with varying group sizes side-by-side. Users can also copy and paste widgets between dashboards using the Datadog clipboard, making it easier to work with multiple dashboards efficiently. The new dashboard experience is designed to combine the best of both Screenboards and Timeboards, offering a seamless editing experience for users to visualize and correlate system-wide monitoring data on screens of any size.

Datadog Notebooks have simplified the way teams find and share knowledge by combining live data and rich Markdown text, enabling powerful, data-driven documents from runbooks to incident postmortems. The new Notebooks API allows for programmatically creating and updating notebooks as part of existing workflows, automating tasks such as generating notebooks on new services and monitors, keeping notebooks up-to-date, and creating offline backups of these documents. This API integration enables teams to seamlessly integrate data-driven documents into their workflows, streamlining knowledge sharing and decision-making across organizations.

Datadog has announced the addition of cross-browser testing to its Synthetic Monitoring platform, allowing users to proactively test their applications on multiple browsers and identify issues before they affect their user base. The new feature supports testing on Firefox, Chrome, Microsoft Edge, and Internet Explorer 11, with the ability to run tests automatically in CI pipelines and across various devices and environments. This shift in cross-browser testing strategy enables teams to deploy changes more frequently while maintaining a consistent user experience, regardless of browser choice.

AWS has launched AWS App Runner, a service designed to simplify the deployment and management of containerized web applications, mobile backends, and API services by handling tasks such as building, deployment, load balancing, and TLS encryption. Backed by AWS Fargate, App Runner ensures automatic scaling and availability of applications without additional infrastructure provisioning. Datadog has integrated with AWS App Runner to provide comprehensive monitoring of applications, offering insights into errors, resource management, and security. Users can track request-handling errors, manage application resourcing, and monitor security through AWS CloudTrail logs. The integration allows for full visibility of containerized applications, helping to identify issues quickly and optimize performance and costs. Existing Datadog users can easily incorporate AWS App Runner metrics and logs, while new users can start with a free trial to explore these capabilities.

Apache JMeter is an open source tool for load testing Java applications to ensure they can handle sudden spikes in traffic without causing latency. However, parsing outcomes and reading patterns from such tests can be challenging, especially for large organizations that frequently test and deploy new code. Datadog's JMeter integration collects test result metrics in aggregate via a plugin, providing real-time overviews of how new features affect an application's performance under stress. It also ingests individual test results as log events to offer granular insights into each request. The integration allows users to visualize key JMeter test metrics and dive into individual test results with logs, enabling them to identify patterns and trends that occur when their applications are running under stress.

Datadog's integration with Apache JMeter allows users to collect and visualize key test metrics in real-time, providing a quick overview of how new features affect an application's performance under stress. The integration collects test result metrics via a plugin and ingests individual test results as log events, enabling granular insight into what occurred during each request. Users can correlate test metrics with monitoring data from elsewhere in their stack, identifying patterns and trends that may indicate performance regression due to increased traffic. Additionally, the integration provides logs alongside metrics, allowing users to view detailed information about individual test runs and zero in on specific patterns that occur when an application is running under stress.

Retool, a low-code platform, has integrated with Datadog to allow users to create internal web applications using a drag-and-drop interface. This integration enables teams to build custom centralized management tools for various employee-facing use cases by integrating with backend databases and APIs. By combining the full spectrum of Datadog API operations, users can manage complex workflows from a single UI without worrying about logic or integration maintenance. Retool's Datadog integration also allows users to embed their apps into Datadog dashboards, supporting management workflows and informing team decisions from a single pane of glass.

Datadog Live Processes now monitors all third-party workloads in one place, allowing users to quickly find and share saved views of processes running third-party software anywhere in their infrastructure. The tool also enables the viewing of performance metrics alongside resource usage of third-party software. Additionally, it extends visibility by enabling auto-detected integrations. Users can customize saved views, correlate performance with resource usage, and discover new integrations to expand their monitoring capabilities.

Retool is a low-code platform that allows teams to build internal web applications using a drag-and-drop interface, reducing friction and lost productivity caused by multiple tools requiring different CLIs or APIs. The platform integrates with various backend databases and APIs, including Datadog, enabling users to create custom, centralized management tools for employee-facing use cases. By integrating with the Datadog API, Retool can access monitoring data, allowing users to visualize key metrics, spot problems, and implement solutions from a single window. Users can build apps that manipulate Elasticsearch resources and pull in key metrics from Datadog within the same window, driving informed cluster management and scaling decisions. The integration enables users to create custom UIs with drag-and-drop components, read/write operations, and components such as Wizards, to manage workflows and make data-driven decisions without switching between multiple services.

Datadog has introduced Live Processes, which provides visibility into third-party software running throughout your infrastructure. With Live Processes, you can quickly find and share saved views to explore processes running third-party software anywhere in your infrastructure, view performance metrics alongside resource usage of your third-party software, and extend your visibility by enabling auto-detected integrations. The feature allows you to correlate performance with resource usage, discover new integrations, and track third-party software health and resource usage with out-of-the-box saved views and auto-detection capabilities.

A dark launch is a deployment strategy for testing new versions of a service in production without affecting users. Monitoring is essential during a dark launch to ensure the success of the new version and prevent any issues in infrastructure. Key practices include tracking SLIs, identifying unexpected response payloads, ensuring proper capacity, and spotting unintended service interactions. Datadog provides a comprehensive platform for monitoring dark launches with features like Unified Service Tagging, Deployment Tracking, Synthetic API tests, browser tests, SLO management, collaboration tools, and integration with various services and technologies.

Cloudflare is a content delivery network (CDN) that helps organizations secure their websites, applications, and APIs. Datadog's integration with Cloudflare allows for the collection of key metrics related to DNS, security, and CDN performance. The integration also enables the ingestion of HTTP request logs and events through Cloudflare's Logpush service, providing deeper insight into the health and performance of Cloudflare Workers and load balancing utilities. This helps teams respond more quickly to issues and reduce service downtime for customers. Datadog can analyze full volumes of Cloudflare logs in real-time, allowing users to troubleshoot problems faster with additional context from logs. The integration also collects metrics related to Cloudflare Workers and load balancers, providing deeper visibility into the performance of Cloudflare-powered applications.

A dark launch is a deployment strategy where a new version of a service is deployed in production and traffic is routed to it without returning responses to users. This allows testing of the new version's performance, error handling, and compatibility with production load. Monitoring is crucial during a dark launch to ensure its success, prevent infrastructure issues, and catch any unexpected behavior or errors. To monitor a dark launch, you should track service level indicators (SLIs), identify unexpected responses, analyze logs, run automated tests, and set up dashboards to visualize performance metrics alongside other key metrics. Additionally, you need to ensure that your infrastructure has ample capacity to handle the new version, spot unintended interactions with persistence layers, and get comprehensive visibility into your dark launches. Tools like Datadog can help simplify this process by providing a unified visualization, alerting, and analysis platform for monitoring all data from your dark launches.

Template variables in Datadog have been enhanced with the $tempvar.value syntax, allowing users more control over tag filtering and data visualization. This dynamic syntax enables users to select multiple sets of tag values using a single key, unify data from different tagging conventions, and simplify network traffic queries. By attributing template variable values as partial strings, users can consolidate their template variables and reduce the number needed for dashboard filtering. The new feature is available to all Datadog customers without additional configuration.

You can now use the `$tempvar.value` syntax when defining template variables for your dashboard widgets in Datadog, giving you more control over what tags your template variables point to and allowing you to visualize all of the data you need across any tags. This new dynamic syntax enables you to select multiple sets of tag values using a single key, unify data from different tagging conventions, simplify network traffic queries, and more. You can use this syntax to reduce the number of template variables needed by attributing values as partial strings, such as specifying a service to graph metrics from in a staging environment. Additionally, you can create "OR" statements in your graph queries using the `$tempvar.value` syntax to include both metric data tagged with different keys, and leverage this feature to build more flexible network traffic queries. With the new dynamic template variable syntax, you can easily create adaptive template variables, reducing the number of templates needed to manage and helping ensure no gaps in the data your dashboards visualize.

Single-page applications (SPAs) offer benefits such as flexibility in moving application logic to the frontend and a smooth user experience with fewer page loads. However, monitoring end-user experiences is crucial for ensuring that SPAs run smoothly on all devices. Traditional methods of frontend performance monitoring are not sufficient for SPAs, which require tracking dynamic page elements like animations, API calls, and rendering lifecycles. A user-centric approach to frontend monitoring involves listening to browser events and tracking user interactions over time. Three aspects of single-page apps that can help optimize users' experience include route changes, user interactions, and errors. Monitoring these aspects requires a combination of techniques such as tracking Largest Contentful Paint (LCP), using User Timing API, listening to route changes, and implementing error tracking.

The latest version of kube-state-metrics (v2.0) brings updates and performance improvements to its predecessor. Datadog's Kubernetes integration now supports full, real-time visibility into your Kubernetes environment from a single pane of glass. To upgrade your Datadog Cluster Agent deployment to enable the new kube-state-metrics v2.0 integration, use Helm and add the value "kubeStateMetricsCore: enabled: true" to your values.yaml file. Once you've upgraded your Agents and enabled kube-state-metrics v2.0 functionality, you can continue analyzing your kube-state-metrics data in Datadog’s out-of-the-box Kubernetes dashboard and set up alerts to stay on top of any cluster-level problems that may arise.

The text discusses the importance of having deep visibility into Kubernetes resources for modern devops teams. It highlights the benefits of using kube-state-metrics, an open source service that generates key metrics about Kubernetes objects, such as pods, nodes, and deployments. The release of kube-state-metrics version 2.0 brings several updates and performance improvements, including a new integration with Datadog's Cluster Agent. To enable this integration, users need to upgrade their Datadog Cluster Agent using Helm and make some updates to metric names in the `values.yaml` file. Once enabled, users can monitor kube-state-metrics data in Datadog's out-of-the-box Kubernetes dashboard and set up alerts to notify teams of any cluster-level problems that may arise. The integration allows for full, real-time visibility into the Kubernetes environment from a single pane of glass, making it easier to track large or unexpected changes in the availability or status of Kubernetes objects.

Mary Jac Heuman discusses the benefits and challenges of Single-Page Applications (SPAs) in a web development context. SPAs offer flexibility, smooth user experience, and fewer page loads compared to traditional multiple-page apps. However, they also introduce trade-offs such as increased complexity and potential issues with older or less powerful devices. To effectively monitor SPA performance, developers need to track dynamic page elements like animations, API calls, and rendering lifecycles that can interrupt the user experience if there are slowdowns. A user-centric approach by listening to browser events and tracking user interactions over time is recommended. Key aspects of SPA performance include route changes, user interactions, and error tracking, which require monitoring tools to optimize the users' experience.

The text discusses how to incorporate monitoring as code into existing Kubernetes infrastructure using the Datadog Operator and custom resource definitions (CRDs). It explains how to get started with the DatadogMonitor CRD, create monitors for specific applications, track updates to cluster pods, and notify teams on the state of cluster nodes. The text also highlights that Datadog offers deeper insights into Kubernetes resources and can automatically connect monitors to instrumented services in Datadog APM using service tags. It concludes by encouraging readers to learn more about using the Datadog Operator and CRDs in their Kubernetes environment or sign up for a free trial if they don't already use Datadog for monitoring.

Application abuse and fraud involve adversaries misusing an application's functionality or features to gain personal benefits or access critical systems. The MITRE ATT&CK® framework outlines various tactics, such as reconnaissance, resource development, credential access, persistence, and impact, that can be used in these attacks. To detect and mitigate signs of application abuse and fraud, organizations should monitor their applications for unusual activity, ask specific questions about potential vulnerabilities, and use tools like Datadog to proactively identify threats and enhance security controls.

Google Cloud Platform has expanded its suite of serverless products since launching Google App Engine in 2008. To provide comprehensive visibility into serverless applications running on Google Cloud, new enhancements have been made to Google Cloud Functions, Google Cloud Run, and Google App Engine integrations. These include out-of-the-box dashboards and enhanced latency and resource utilization metrics (p95 and p99) for troubleshooting performance issues. The text also covers the features of Google Cloud Functions, Google Cloud Run, and Google App Engine, as well as how Datadog's new Google Cloud Functions dashboard provides a high-level overview of key performance metrics from functions. It also explains how to debug Google Cloud Run errors across revisions and monitor Google App Engine standard and flex environments. Additionally, the text discusses enhanced metrics for Google serverless applications and how users can start monitoring their Google serverless apps today.

Google Cloud Platform is expanding its serverless offerings with enhanced integrations and new features. The company has introduced out-of-the-box dashboards for Google Cloud Functions, Google Cloud Run, and Google App Engine to provide comprehensive visibility into serverless applications running on the platform. These dashboards offer high-level overviews of key performance metrics, such as latency, error rate, and resource utilization, allowing users to troubleshoot performance issues before they impact their users. Additionally, Datadog has enhanced its integrations with these platforms by adding percentile aggregations for latency and resource utilization metrics, enabling users to make more informed decisions about serverless application resources. With these new features, users can gain deeper insights into the performance and health of their serverless applications, reducing the risk of errors and improving overall user experience.

The Datadog Operator enables teams to easily incorporate monitoring as code into their existing Kubernetes infrastructure. By implementing infrastructure as code, teams can optimize the process for provisioning and updating production-ready resources, reducing the challenges of tracking services with varying configurations and resources. The operator allows users to create customized monitors for Kubernetes resources using a custom resource definition (CRD), enabling automatic creation and management of monitors via deployment manifests and tools like kubectl. This approach provides visibility into all Kubernetes resources, enables teams to deploy repeatable processes for monitoring, and notifies them on critical issues with their pods, nodes, and more, ensuring they can quickly resolve problems and improve the overall health of their environment.

Mallory Mooney and Justin Massey discuss the importance of protecting applications from abuse of functionality, highlighting key tactics used by adversaries to exploit vulnerabilities in authentication controls, inventory management, payment services, and customer rewards programs. To identify signs of application abuse and fraud, it's essential to monitor logs, filter out noise, and ask questions about your application, such as exposing information that could be harvested or targeting specific users. Datadog can help proactively monitor abuse of functionality by collecting logs, providing threat intelligence, and offering full visibility into application activity. To mitigate these threats, consider implementing measures like two-factor authentication, rate limiting, web application firewalls, and least privilege policies for internal services. By leveraging Datadog's security offerings, you can gain greater confidence in identifying risks to your applications and take proactive steps to secure them.