September 2021 Summaries

Performance bottlenecks are often unexpected and difficult to locate without proper tools. In this case, a Java application using the Akka framework experienced high CPU usage due to an issue with the ForkJoinPool. The problem was discovered and fixed by utilizing Datadog Continuous Profiler, which measures software resource usage. By analyzing flame graphs and stack frames, it was determined that the LatencyReportActor was responsible for increased time spent inside the ForkJoinPool.scan()/Unsafe.park() methods. Moving this actor to a different dispatcher resolved the issue, reducing CPU usage by 30%. The experience highlights the importance of continuous profiling and not relying solely on intuition when addressing performance issues in production environments.

We discovered a significant performance bottleneck in our Java application using the Akka framework, which was attributed to an irregular flow of incoming tasks and high maximum number of threads in the underlying thread pool. This led to frequent spikes in CPU usage due to expensive native calls made by the `ForkJoinPool.scan()` and `Unsafe.park()` methods. By analyzing flame graphs and profiling output, we identified the root cause as the default Akka dispatcher using a `ForkJoinPool` with 32 threads, which was not stable under our workload. We fixed the issue by moving the problematic actor to a more suitable "work" dispatcher with a stable flow of tasks, resulting in a 30% reduction in overall CPU usage and confirming our hypothesis through configuration changes.

AWS Graviton2 processors use the Arm architecture to provide high-efficiency and low-cost computing. Datadog has partnered with AWS to support new Graviton2 compute resources for Lambda functions, allowing users to monitor function performance alongside telemetry data from the rest of their stack. This enables users to determine whether Graviton2 is a good fit for their serverless workloads. Datadog's Lambda extension fully supports Arm-based architectures and automatically tags functions with key metadata, including architecture, which helps visualize and compare performance across different platforms. AWS reports that Graviton2-powered functions can experience up to 34% reduced duration and faster cold start times, making it an ideal choice for functions susceptible to frequent or long cold starts. Datadog will also generate real-time enhanced Lambda metrics, providing users with granular insights into function performance.

Datadog is partnering with AWS to provide visibility into Lambda functions running on Graviton2 processors, which offer high-efficiency and low-cost computing. Datadog's Lambda extension automatically tags functions with metadata, including architecture, enabling comparison of performance across different platforms. This allows users to assess whether migrating to Graviton2 makes sense for their serverless workloads. With Datadog, users can monitor function performance alongside telemetry data from the rest of their stack, and gain insights into cold start times, memory usage, and estimated costs. The partnership enables users to visualize performance and make informed decisions about serverless application migration.

In this tutorial, we will show you how to monitor the newer, monolithic versions of Istio using Prometheus and Grafana. We will also demonstrate how to use Datadog for comprehensive visibility into your Istio cluster. First, let's introduce some key concepts related to monitoring Istio: - Metrics are numerical values that represent a specific aspect of the system being monitored. In the context of Istio, metrics can include request rates, error counts, and latency distributions for service-to-service traffic within your mesh. - Logs provide detailed information about individual events or transactions within your system. For example, you might use logs to investigate a specific spike in request latencies or an unexpected increase in the number of 500 response codes from one of your services. - Traces are sequences of related requests that pass through multiple services and components within your mesh. By collecting traces from Envoy proxies, you can visualize the architecture of your service mesh and understand how traffic flows between different parts of your system. Istio generates telemetry data within istiod and your mesh of Envoy proxies via Prometheus, and you can access this data by enabling several popular monitoring tools that Istio includes as a pre-configured bundle. To get greater detail for ad hoc investigations, you can access istiod and Envoy logs via kubectl, and troubleshoot your Istio configuration with istioctl analyze. Installing Istio add-ons Each Envoy proxy and istiod container in your Istio cluster will run a Prometheus instance that emits the metrics we introduced earlier. You can access istiod metrics at <ISTIOD_CONTAINER_IP>:15014/metrics, and Envoy metrics at <ENVOY_CONTAINER_IP>:15000/stats. You can quickly set up monitoring for your cluster by enabling Istio's out-of-the-box add-ons. Istio's Prometheus add-on uses Kubernetes’ built-in service discovery to fetch the DNS addresses of istiod pods and Envoy proxy containers. You can then open Istio's Grafana dashboards (which we introduced in Part 2) to visualize metrics for istiod and your service mesh. And if you enable Zipkin or Kiali, you can visualize traces collected from Envoy. This helps you understand your mesh's architecture and visualize the performance of service-to-service traffic. Beginning with version 1.4.0, Istio has deprecated Helm as an installation method, and you can install Istio's monitoring add-ons by using the istioctl CLI. To install the add-ons, run the following command: istioctl install --set <KEY1>=<VALUE1> --set <KEY2>=<VALUE2> The second column in the table below shows the value of the --set flags you should add to enable specific add-ons. Once you’ve enabled an add-on, you can open it by running the command in the third column. |Add-on | How to enable | How to view | |Prometheus | --set values.prometheus.enabled=true | istioctl dashboard prometheus | |Grafana | --set values.grafana.enabled=true | istioctl dashboard grafana | |Kiali | --set values.tracing.enabled=true --set values.kiali.enabled=true | istioctl dashboard kiali | |Zipkin | --set values.tracing.enabled=true --set values.tracing.provider=zipkin | istioctl dashboard zipkin | Istio and Envoy logging Both istiod and Envoy log error messages and debugging information that you can use to get more context for troubleshooting (we addressed this in more detail in Part 2). istiod publishes logs to stdout and stderr by default. You can access istiod logs with the kubectl logs command, using the -l app=istiod option to collect logs from all istiod instances. The -f flag prints new logs to stdout as they arrive: kubectl logs -f -l app=istiod -n istio-system Envoy access logs are disabled by default. You can run the following command to configure Envoy to print its access logs to stdout. istioctl install --set meshConfig.accessLogFile="/dev/stdout" To print logs for your Envoy proxies, run the following command: kubectl logs -f <SERVICE_POD_NAME> -c istio-proxy If you want to change the format of your Envoy logs and the type of information they include, you can use the --set flag in istioctl install to configure two options. First, you can set global.proxy.accessLogEncoding to JSON (the default is TEXT) to enable structured logging in this format. Second, the accessLogFormat option lets you customize the fields that Envoy prints within its access logs, as we discussed in more detail in Part 2. istioctl analyze If your Istio metrics are showing unexpected traffic patterns, anomalously low sidecar injections, or other issues, you may have misconfigured your Istio deployment. You can use the istioctl analyze command to see if this is the case. To check for configuration issues in all Kubernetes namespaces, run the following command: istioctl analyze --all-namespaces The output will be similar to the following: Warn [IST0102] (Namespace app) The namespace is not enabled for Istio injection. Run 'kubectl label namespace app istio-injection=enabled' to enable it, or 'kubectl label namespace app istio-injection=disabled' to explicitly mark it as not needing injection Info [IST0120] (Policy grafana-ports-mtls-disabled.istio-system) The Policy resource is deprecated and will be removed in a future Istio release. Migrate to the PeerAuthentication resource. Error: Analyzers found issues when analyzing all namespaces. See https://istio.io/docs/reference/config/analysis for more information about causes and resolutions. In this case, the first warning explains why mesh metrics are missing in the app namespace: because we have not yet enabled sidecar injection. After enabling automatic sidecar injection for the app namespace, we can watch our sidecar injection metrics to ensure our configuration is working. Monitoring istiod with Datadog Datadog gives you comprehensive visibility into the health and performance of your mesh by enabling you to visualize and alert on all the data that Istio generates within a single platform. This makes it easy to navigate between metrics, traces, and logs, and to set intelligent alerts. In this section, we’ll show you how to monitor istiod with Datadog. Datadog monitors your Istio deployment through a collection of Datadog Agents, which are designed to maximize visibility with minimal overhead. One Agent runs on each node in your cluster, and gathers metrics, traces, and logs from local Envoy and istiod containers. The Cluster Agent passes cluster-level metadata from the Kubernetes API server to the node-based Agents along with any configurations needed to collect data from Istio. This enables node-based Agents to get comprehensive visibility into your Istio cluster, and to enrich metrics with cluster-level tags. Set up Datadog’s Istio integration We recommend that you install the Datadog Operator within your Istio cluster using the Helm package manager. The Operator will track the states of Datadog resources, compare them to user configurations, and apply changes accordingly. In this section, we will show you how to: - Deploy the Datadog Operator - Annotate your Istio services so the Datadog Agents can discover them Deploy the Datadog Operator Before you deploy the Datadog Operator, you should create a Kubernetes namespace for all of your Datadog-related resources. This way, you can manage them separately from your Istio components and mesh services, and exempt your Datadog resources from Istio's automatic sidecar injection. Run the following command to create a namespace for your Datadog resources: kubectl apply -f - <<EOF { "apiVersion": "v1", "kind": "Namespace", "metadata": { "name": "datadog", "labels": { "name": "datadog" } } } EOF Next, create a manifest called dd_agent.yaml that the Datadog Operator will use to install the Datadog Agents. Note that you’ll only need to provide the keys below the spec section of a typical Kubernetes manifest (the Operator installation process will fill in the rest): dd_agent.yaml credentials: apiKey: "<DATADOG_API_KEY>" # Fill this in appKey: "<DATADOG_APP_KEY>" # Fill this in # Node Agent configuration agent: image: name: "datadog/agent:latest" config: # The Node Agent will tolerate all taints, meaning it can be deployed to # any node in your cluster. # https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ tolerations: - operator: Exists # Cluster Agent configuration clusterAgent: image: name: "datadog/cluster-agent:latest" config: metricsProviderEnabled: true clusterChecksEnabled: true # We recommend two replicas for high availability replicas: 2 You’ll need to include your Datadog API key within the manifest. You won't have to provide an application key to collect data from Istio, but one is required if you want the Datadog Operator to send data to Datadog for troubleshooting Datadog Agent deployments. You can find your API and application keys within Datadog. Once you’ve created your manifest, use the following Helm (version 3.0.0 and above) and kubectl commands to install and configure the Datadog Operator: helm repo add datadog https://helm.datadoghq.com helm install -n datadog my-datadog-operator datadog/datadog-operator kubectl -n datadog apply -f "dd_agent.yaml" # Configure the Operator The helm install command uses the -n datadog flag to deploy the Datadog Operator and the resources it manages into the datadog namespace we created earlier. After you’ve installed the Datadog Operator, it will deploy the node-based Agents and Cluster Agent automatically. $ kubectl get pods -n datadog NAME READY STATUS RESTARTS AGE datadog-datadog-operator-5656964cc6-76mdt 1/1 Running 0 1m datadog-operator-agent-hp8xs 1/1 Running 0 1m datadog-operator-agent-ns6ws 1/1 Running 0 1m datadog-operator-agent-rqhmk 1/1 Running 0 1m datadog-operator-agent-wkq64 1/1 Running 0 1m datadog-operator-cluster-agent-68f8cf5f9b-7qkgr 1/1 Running 0 1m datadog-operator-cluster-agent-68f8cf5f9b-v8cp6 1/1 Running 0 1m Automatically track Envoy proxies within your cluster The node-based Datadog Agents are pre-configured to track Envoy containers running on their local hosts. This means that Datadog will track mesh metrics from your services as soon as you have deployed the node-based Agents. Configure the Datadog Agents to track your Istio deployment Since Kubernetes can schedule istiod and service pods on any host in your cluster, the Datadog Agent needs to track the containers running the relevant metrics endpoints—no matter which pods run them. We’ll show you how to configure Datadog to use endpoints checks, which collect metrics from the pods backing the istiod Kubernetes service. With endpoints checks enabled, the Cluster Agent ensures that each node-based Agent is querying the istiod pods on its local host. The Cluster Agent then populates an Istio Autodiscovery template for each node-based Agent. The Datadog Cluster Agent determines which Kubernetes services to query by extracting configurations from service annotations. It then fills in the configurations with up-to-date data from the pods backing these services. You can configure the Cluster Agent to look for the istiod service by running the following script (note that the service you’ll patch is called istio-pilot in version 1.5.x and istiod in version 1.6.x.): kubectl -n istio-system patch service <istio-pilot|istiod> --patch "$(cat<<EOF metadata: annotations: ad.datadoghq.com/endpoints.check_names: '["istio"]' ad.datadoghq.com/endpoints.init_configs: '[{}]' ad.datadoghq.com/endpoints.instances: | [ { "istiod_endpoint": "http://%%host%%:15014/metrics", "send_histograms_buckets": true } ] EOF )" The ad.datadoghq.com/endpoints.instances annotation includes the configuration for the Istio check. Once the istiod service is annotated, the Cluster Agent will dynamically fill in the %%host%% variable with the IP address of each istiod pod, then send the resulting configuration to the Agent on the node where that pod is running. After applying this change, you’ll see istiod metrics appear within Datadog. Get the visibility you need into istiod's performance Datadog helps you monitor metrics for istiod, Envoy, and 650+ integrations, giving you insights into every component of your Istio cluster. For example, you can create a dashboard that visualizes request rates from your mesh alongside Kubernetes resource utilization (below), as well as metrics from common Kubernetes infrastructure components like Harbor and CoreDNS. Istio's most recent benchmark shows that Envoy proxies consume 0.6 vCPU for every 1,000 requests per second—to ensure that all services remain available, you should keep an eye on per-node resource utilization as you deploy new services or scale existing ones. You can also use Datadog to get a clearer picture into whether to scale your istiod deployment. By tracking istiod's work and resource utilization metrics, you can understand when istiod pods are under heavy load and scale them as needed. The dashboard below shows key throughput metrics for each of istiod’s core functions—handling sidecar injection requests, pushing xDS requests, creating certificates, and validating configurations—along with high-level resource utilization metrics (CPU and memory utilization) for the istiod pods in your cluster. Cut through complexity with automated alerts You can reduce the time it takes to identify issues within a complex Istio deployment by letting Datadog notify you if something is amiss. Datadog enables you to create automated alerts based on your istiod and mesh metrics, plus APM data, logs, and other information that Datadog collects from your cluster. In this section, we’ll show you two kinds of alerts: metric monitors and forecast monitors. Metric monitors can notify your team if a particular metric's value crosses a threshold for a specific period of time. This is particularly useful when monitoring automated sidecar injections, which are critical to Istio's functionality. You’ll want to know as soon as possible if your cluster has added new pods without injecting sidecars so you can check for misconfigurations or errors within the injection webhook. You can do this by setting a metric monitor on the difference between sidecar_injection_success_total and sidecar_injection_requests_total, which tells you how many sidecar requests were skipped or resulted in an error). If you get alerted that this value is unusually low, you can immediately investigate.

Postmortems are crucial for preventing future incidents by recording root causes and providing insights into system resilience. However, they can be difficult to produce due to deeper analysis requirements and coordination among busy team members. To streamline the process, workflows that centralize data collection, facilitate discussions, and generate interactive postmortem documents automatically are recommended. Key practices include gathering data in a shared view, automating postmortem generation, using the postmortem as a thinking tool, and making it easy to find later. Datadog's comprehensive platform incorporates these best practices to make writing postmortems seamless and efficient.

Stephanie Niu and Paul Gottschling discuss the importance of writing thorough postmortems after an incident to prevent similar incidents in the future. They provide best practices for writing postmortems, including centralizing data collection, automating generation of postmortem documents, using the postmortem as a thinking tool, making it easy to find later, and leveraging Datadog's comprehensive platform to streamline the process. By following these guidelines, teams can focus on building more resilient systems rather than compiling data and coordinating with teammates.

Datadog's Continuous Profiler provides detailed insights into application code performance by measuring execution time and resource utilization. The profiler allows continuous capture of code profiles for all production instances, enabling users to compare profiles in the profile comparison view to identify and optimize inefficient code. This helps manage compute costs and resolve performance bottlenecks that affect user experience. The profile comparison view also quantifies changes made to fix a performance issue, allowing users to calculate estimated savings on production infrastructure costs. It efficiently troubleshoots root causes behind performance blind spots at the method level, detecting regressions such as increased heap memory size or service latency. Additionally, it helps identify the cause of increased latency for service endpoints and compares code performance between releases. Overall, Datadog's Continuous Profiler enhances visibility into code performance, enabling users to diagnose issues, optimize costs, and deploy fixes more effectively.

Datadog's Continuous Profiler provides detailed insight into the efficiency of application code by measuring execution time and resource utilization, enabling users to manage compute costs and optimize inefficient code. The profiler allows for continuous capture of code profiles for all production instances, providing a view of how performance changes over time, and helping users identify and fix performance bottlenecks. This feature is particularly useful in detecting code performance regressions, identifying the cause of increased latency in service endpoints, and comparing code performance between releases. By providing real-time monitoring and comparison capabilities, Datadog's Continuous Profiler enables developers to diagnose performance issues, optimize costly lines of code, and deploy fixes faster, ultimately improving their users' experience.

ActiveMQ Classic and Artemis are two versions of the open-source messaging server from Apache Software Foundation. They support both point-to-point (queue) and publish/subscribe (topic) communication styles, and they can be used to integrate different systems within an organization or across multiple organizations. Monitoring ActiveMQ metrics is essential for maintaining a healthy messaging infrastructure. In this series of posts, we’ll explore the key metrics you should monitor in both versions of ActiveMQ, as well as some tools that can help you collect and visualize these metrics. In Part 1 of this series, we discussed: - The importance of monitoring your messaging system - Key metrics to watch for in both versions of ActiveMQ - How to interpret those metrics and what actions you might take based on their values In Part 2 of this series, we’ll look at some tools that use Java Management Extensions (JMX) to monitor ActiveMQ Classic and Artemis brokers. We’ll also explore how you can use these tools to collect and visualize the metrics discussed in Part 1. Monitoring your messaging system is crucial for maintaining its performance, availability, and security. By monitoring key metrics from both versions of ActiveMQ, you can detect potential issues before they affect your services, allowing you to take proactive measures to prevent or mitigate problems. In the next post in this series, we’ll dive deeper into the tools available for monitoring ActiveMQ Classic and Artemis brokers. Stay tuned!

This article discusses various tools that can be used to collect metrics from ActiveMQ, a popular open-source messaging server. These include command line tools, web consoles, JMX monitoring tools like JConsole and Hawtio, as well as the statistics plugin and management message API for more programmatic access to metrics. The article also mentions that while these tools provide valuable insights into ActiveMQ's performance, a comprehensive monitoring strategy should also include monitoring of producers, consumers, and infrastructure.

Comprehensive monitoring of ActiveMQ, both Classic and Artemis, can be achieved by integrating it with Datadog to track performance metrics, logs, and alerts for potential issues. This involves configuring ActiveMQ to send metrics to Datadog via JMX and the Web Console, installing the Datadog Agent, and customizing dashboards to visualize metrics. By enabling log collection and using custom tags, users can correlate logs with metrics to analyze trends and troubleshoot issues effectively. Datadog's alert system can be tailored using tags to maintain awareness of critical issues, and integration with various notification services ensures timely communication with teams. The article provides detailed instructions for setting up these monitoring capabilities, emphasizing the importance of visibility into the entire messaging infrastructure.

Datadog Network Performance Monitoring (NPM) is a tool that helps identify and troubleshoot problems in modern services and applications by providing comprehensive visibility into network traffic flows. It collects key TCP/UDP and DNS data related to traffic between endpoints, including VMs, containers, services, cloud regions or datacenters, and more. The NPM Overview dashboard provides a unified view of key network health and performance metrics across different facets of your distributed network, making it easier to troubleshoot problems within different layers of network performance and correlate that data with other telemetry from the environment. Datadog NPM also enables users to quickly monitor critical dependencies with Saved Views and correlates network data with telemetry from each layer of their stack for a deeper understanding of the health of their environment.

Datadog's Cloud Network Monitoring (CNM) feature provides comprehensive visibility into network and DNS performance, enabling users to quickly spot issues that manifest as traffic spikes, drops, or latency between different endpoints in their environment. CNM automatically collects key transport-layer (TCP/UDP) and DNS data related to traffic between each endpoint, including VMs, containers, services, cloud regions or datacenters, and much more. The feature provides an out-of-the-box CNM Overview dashboard that organizes network metrics into essential categories, such as Network load, TCP metrics, DNS metrics, Application overview, and Cross-regional traffic metrics. These metrics provide a unified view of key network health and performance across different facets of the distributed network. Additionally, CNM enables users to quickly monitor critical dependencies with Saved Views, which provide quick access to relevant network flow data. The feature also correlates network data with telemetry from each layer of the stack, enabling users to pinpoint root causes of issues. Overall, Datadog's Cloud Network Monitoring helps make troubleshooting problems with your network easier by providing a comprehensive and unified view of key network performance metrics.

Amazon Elastic Kubernetes Service (EKS) Anywhere is an automation tool that simplifies cluster operations by launching EKS Distro clusters on vSphere virtual machines. This makes it easier for organizations to manage both cloud-based and on-premise EKS clusters, making it a good fit for those planning to migrate their on-premise Kubernetes deployments to the cloud or handle bursts in traffic. Datadog is a launch partner for EKS Anywhere, providing full visibility into the health and performance of EKS Anywhere and cloud-based EKS workloads, as well as their underlying infrastructure. With Datadog's Kubernetes resources view, users can monitor resource capacity and usage of both on-premise and cloud-based EKS clusters in one platform. Additionally, Datadog enables custom dashboards to monitor Kubernetes metrics and vSphere metrics for deeper insight into the health of their on-premise infrastructure.

Netlify is a web development platform that enables users to build and deploy dynamic, high-performance web applications using popular JavaScript frameworks, developer tools, and APIs. It supports various use cases such as e-commerce stores, SaaS applications, and corporate sites with integrated CI/CD tools, global multi-cloud edge networks, and serverless backends. Datadog can now be used to capture Netlify web traffic and serverless function logs for long-term retention and analysis. By forwarding Netlify logs to Datadog using the Log Drains feature, users can monitor and visualize key web traffic and function performance data. This integration also allows users to generate metrics from their logs to visualize and alert on various aspects of their applications' health and performance. Additionally, Datadog Synthetic Monitoring provides comprehensive visibility into the frontend performance of Netlify sites by creating multistep browser tests that measure response times and detect errors in user flows.

Amazon Elastic Kubernetes Service (EKS) Anywhere is a cloud-based compute platform that simplifies cluster operations by providing a fully managed Kubernetes control plane. EKS Anywhere allows teams to operate and manage both cloud-based and on-premise EKS clusters using consistent tooling, making it suitable for organizations planning to migrate their on-premise deployments or deploying workloads in the cloud to handle bursts in traffic. Datadog is a launch partner for EKS Anywhere, enabling full visibility into the health and performance of EKS Anywhere and cloud-based EKS workloads, as well as underlying infrastructure, whether running on premises or in the cloud.

Netlify is a Jamstack web development platform that enables rapid deployment and shipping of dynamic, highly performant web apps. It integrates popular JavaScript frameworks, developer tools, and APIs into streamlined workflows to support common use cases such as e-commerce stores, SaaS applications, and corporate sites. Netlify supports these deployments with an integrated CI/CD tool, global multi-cloud edge network, and serverless backend. By ingesting Netlify logs into Datadog, developers can monitor and visualize key web traffic and function performance data, capture and analyze serverless function logs to identify error-prone or slow functions, use traffic logs to understand user behavior and trends, and monitor frontend performance with Datadog Synthetic Monitoring. This integration allows for full visibility into serverless functions and site traffic, enabling developers to make data-driven decisions and stay ahead of user-facing problems.

Datadog Network Performance Monitoring (NPM) has introduced automatic detection and tagging of AWS- and GCP-managed endpoints in your network. This feature provides an unprecedented view into applications' communication with managed cloud services, such as Amazon S3 and RDS, Google Cloud DNS, and more. It enables users to monitor network traffic to their AWS services and API calls made to key GCP dependencies, making it easier to identify the source of communication issues. The NPM also allows visualization of network throughput, latency, and other key metrics about the traffic between any tagged objects in your environment. This helps users spot where latency and connectivity issues are concentrated and identify which client services and managed endpoints are causing or affected by these performance bottlenecks.