April 2022 Summaries

Davison Bare, a Technical Enablement Manager (TEM) at Datadog, shares his experience in this role. Starting as a Solutions Engineer in 2017 and being promoted to Senior Solutions Engineer in 2019, he transitioned into the TEM position within two months. As a TEM, Davison spends most of his time on customer calls teaching them how to effectively use Datadog's platform. He distinguishes between TEM and Technical Account Manager (TAM) roles, noting that TEMs are not tied to specific organizations but rather help any customers understand the product. The Technical Post Sales team at Datadog values knowledge sharing and collaboration. Davison finds pride in helping customers understand the platform and seeing their lightbulb moments.

Sara Verdi introduces Davison Bare, a Technical Enablement Manager at Datadog, who shares his journey to the role and insights into what it's like to be a TEM. As a TEM, Davison spends most of his time on calls with customers, teaching them how to effectively use Datadog's platform. He highlights the differences between the TEM and Technical Account Manager (TAM) roles, noting that TEMs are not tied to a particular organization and prioritize helping all customers. The Technical Post Sales team is known for its collaborative culture, where knowledge-sharing is encouraged to overcome the challenges of supporting a rapidly growing platform. Davison finds pride in immediate results from his work, feeling satisfied when he helps customers understand how to use Datadog's platform.

Knative for Anthos is a project developed by Google in collaboration with other companies that simplifies building, deploying, and scaling serverless containers across Kubernetes infrastructure. Datadog's integration allows seamless monitoring of all serverless workloads in one place, regardless of their location. The integration provides key visibility into hybrid workloads, autoscaling activity, and revision management for Knative for Anthos. By using Datadog, users can monitor the health and performance of their applications and services across hybrid environments, ensuring optimal functionality and resource allocation.

The Knative project is a serverless container platform developed by Google with contributions from various companies. It aims to simplify the deployment, scaling, and management of serverless applications across existing Kubernetes infrastructure. The Knative for Anthos integration with Datadog provides seamless monitoring of all serverless workloads in one place, regardless of their location. This allows developers to focus on application logic and code while leveraging a unified control plane to manage hybrid environments. Key use cases include monitoring hybrid workloads, autoscaling, revision management, and visualizing latency distribution and request counts. The integration provides real-time insights into key health and performance metrics, enabling effective issue management and ensuring the smooth operation of serverless applications across multiple cloud and on-prem environments.

Cryptocurrency mining has become an attractive target for cyber threat actors due to its potential profitability and resource-intensive nature. As a result, these attackers are increasingly targeting organizations' cloud resources to mine cryptocurrencies. Datadog Cloud SIEM offers a built-in detection rule that helps monitor cloud-based systems for unwanted crypto mining activity by scanning log data from all your cloud resources for suspicious IP or domain addresses associated with mining servers or pools. Once enabled, the rule generates security signals when flagged IP or domains are detected in logs, providing key insights about affected hosts and processes. This helps organizations investigate further signs of mining activity, identify compromised hosts, mitigate the threat by killing unauthorized processes and adding suspicious IP addresses to firewall deny lists, and fine-tune rules with suppression lists to reduce false positives. Overall, Datadog Cloud SIEM enables quick detection of unwanted crypto mining activity in cloud environments, protecting resources, maintaining performance, and preventing unexpected costs.

Datadog Cloud SIEM has introduced a built-in detection rule to monitor cloud-based systems for unwanted cryptocurrency mining, which can quickly strain servers and cause unexpected computing costs. The rule scans log data from all cloud resources for activity from known IP or domain addresses associated with mining servers or pools. Once enabled, Datadog generates security signals when it detects suspicious activity, providing insights into affected hosts such as performance metrics and running processes. These signals help investigators identify potential threats like cryptominers, allowing them to pivot to related processes, search for unexpected activities, and mitigate attacks by killing unauthorized processes, blocking malicious IP addresses, or adding them to firewall deny lists. Additionally, Datadog's suppression lists enable users to reduce false positives by controlling when security signals are generated, preventing legitimate cryptocurrency applications from being flagged as malicious activity.

In a competitive mobile app market, ensuring continuous availability and a seamless user experience is crucial to minimize user churn. Mobile apps face unique challenges due to the variety of devices and the time-consuming nature of rolling out updates, unlike web apps. Key performance metrics such as app start time, view rendering time, network performance, and resource utilization are vital for monitoring app health. Understanding user actions and collecting crash data are essential for diagnosing errors and enhancing user experience. Tracking these metrics helps developers identify performance issues and prioritize optimizations. Datadog offers tools for mobile real user monitoring, end-to-end tracing, and error tracking to help developers maintain high app performance and availability.

Organizations must secure and store sensitive data logged by application services to protect customers and remain compliant with various regulations such as HIPAA and GDPR. This process involves identifying confidential information, categorizing flagged data based on severity and sensitivity, and taking appropriate action based on classifications. Datadog Sensitive Data Scanner can help implement these steps at scale for complete visibility and control over sensitive data logged by services.

The text discusses the enhancement of Real User Monitoring (RUM) with the use of formulas and functions to create more informative and actionable alerts. It highlights how traditional RUM monitors rely on static thresholds for metrics like page load times, which may not adequately capture user experience or application performance. By employing formulas and functions, users can generate alerts based on ratios, percentages, and comparisons to past data, providing deeper insights into user behavior and application functionality. Examples include tracking error rates as a percentage of total views and comparing current performance metrics with historical data to detect anomalies. This method reduces alert fatigue and provides a clearer understanding of significant changes impacting application performance. The text also mentions that further details on implementing these techniques are available in the documentation, and encourages trying Datadog's services with a free trial.

Amazon Elastic Kubernetes Service (EKS) has released EKS Blueprints to help customers create internal development platforms on EKS. Datadog is an official launch partner for EKS Blueprints, allowing users to deploy the Datadog Agent and gain deep visibility into their infrastructure's health and performance. EKS Blueprints simplifies workload deployment while still supporting existing frameworks and adding new capabilities. Datadog provides tools to monitor clusters, pods, containers, and other Kubernetes resources, offering out-of-the-box Kubernetes dashboards for a high-level overview of cluster performance. The Live Container view offers real-time resource metrics that can be filtered using dynamic tags. Integrating EKS Blueprints with Datadog provides comprehensive insights into workload health and performance.

Shoreline has released its Datadog App, allowing users to leverage debug and repair features within the Datadog UI. The app provides an out-of-the-box Shoreline Remediation Dashboard that displays relevant metrics and tags for monitoring alerts. Users can run debug statements and configure automated remediations directly from the dashboard using Shoreline's Op language. Additionally, users can create and manage Shoreline Actions to automate remediation scripts across all relevant hosts simultaneously. The app helps reduce DevOps overhead, lower mean time to resolution (MTTR), and enable more engineers and developers to take on site reliability engineering work.

The APM Service Page has been updated to provide a comprehensive overview of your service's performance issues and enable quick diagnosis and investigation. It now features highlights on deployments, errors, SLOs, and incidents, as well as integrated Error Tracking, traces, log patterns, and code profiles. With the new Service Page, users can easily monitor faulty deployments, new issues, SLO breaches, and ongoing incidents. The service summary introduces a dependency map for clear visibility of upstream and downstream service dependencies. Automatic anomaly detection with Watchdog Insights has been added to aid investigations. Additionally, end-to-end visibility is provided through distributed traces, log patterns, and code profiles.

Amazon Elastic Kubernetes Service (EKS) is a managed container service designed to deploy and scale cloud-based or on-premise Kubernetes applications. AWS released EKS Blueprints, a framework for creating internal development platforms on EKS, which enables enterprises to reduce operational complexity by packaging tools into a cohesive platform that allows seamless deployment of EKS workloads at scale. Datadog is an official launch partner with EKS Blueprints, providing customers with the ability to deploy the Datadog Agent and gain deep visibility into the health and performance of their dynamic infrastructure. The integration supports extensibility and scalability, allowing developers to create well-architected EKS clusters across multiple accounts and regions from a single Git repository. With this integration, users can monitor clusters, pods, containers, and other Kubernetes resources in real-time, providing critical insights into metrics such as pod availability and resource utilization.

Session Replay now includes Browser Dev Tools, allowing engineers to identify and debug issues more efficiently by exposing key information about a playback session. The Console tab in Dev Tools shows all browser logs collected during a session, enabling efficient troubleshooting of user-facing issues. Additionally, the Performance tab provides a timeline of resources loaded and Google's Core Web Vitals, allowing for better understanding and addressing customer needs by identifying long requests tied to performance issues. Session Replay Dev Tools offers valuable insights into user experience and helps identify performance bottlenecks.

Nicholas Thomson and Miranda Kapin from Datadog discuss Session Replay now, a feature that enables engineers to identify and debug the root causes of issues even faster by exposing key information about a playback session. This includes network performance bottlenecks and any console log errors. By leveraging Dev Tools console logs, developers can resolve user-facing issues with ease, gain more insight into the cause of problems, and trace issues from the frontend to the backend. Session Replay's video-like reproduction of real user journeys on your site complements this by making it easy to address any pain points. The feature also provides a waterfall of events and their network performance during any given page view, allowing developers to better understand and address customer needs. Additionally, Datadog offers a 14-day free trial for those new to the platform.

Microsoft's Cloud Adoption Framework (CAF) for Azure helps organizations plan and execute successful cloud migrations by providing detailed guidelines, best practices, and tools. Datadog has partnered with Microsoft to integrate its observability and security platform into the CAF, enabling continuous monitoring of each migration step. This partnership allows users to better monitor their workload shifts to Azure, ensuring faster scaling and reduced downtime or disruptions. Datadog's features include Service Map for visualizing service dependencies, Host Map for tracking infrastructure metrics across environments, integration with Azure services, Synthetic Monitoring for testing endpoints, Real User Monitoring (RUM) for identifying latency issues, and App Service extension for monitoring app performance on Azure App Service.

Microsoft's Cloud Adoption Framework for Azure is designed to help organizations migrate their applications to the cloud with confidence. Datadog partners with Microsoft to provide critical visibility into the health, performance, and makeup of an organization's environment during the migration process. By integrating Datadog with the CAF, organizations can better monitor each step of their migration, enabling them to shift and scale their workloads faster and with more confidence. This is achieved through features such as Datadog's Service Map, Host Map, and APM instrumentation, which provide real-time visibility into application architecture, infrastructure metrics, and end-to-end performance. With Datadog's support, organizations can limit potential downtime or disruptions, ensure service continuity for users, and make adjustments to the new environment as needed.

The new Redis Enterprise integration provides vital KPIs for Enterprise-specific features, allowing users to monitor the health and efficiency of their databases. Key points include optimizing database performance across clusters by tracking capacity metrics such as RAM utilization, latency, and cache-hit ratios; visualizing the health of Redis Enterprise clusters through dashboards that display shard counts, node counts, memory usage, and other internal tiering metrics; ensuring high availability and consistency with Active-Active databases using Conflict-free Replicated Data Types (CRDT); and monitoring significant cluster events for rapid troubleshooting.

The text discusses the integration of Redis Enterprise with Datadog, a monitoring and analytics platform. The integration provides vital KPIs for Enterprise-specific features, enabling users to monitor the health and efficiency of their databases. It offers out-of-the-box dashboards, including the Cluster Top View dashboard, which provides a quick summary of the state of the system, as well as the Database Overview dashboard, which displays capacity metrics, such as RAM utilization, to help users gain insight into their workload. The integration also supports multi-tier storage through Redis on Flash (RoF), reducing storage costs without impacting overall performance, and allows users to create monitors that alert them when KPIs fail to meet performance standards. Additionally, it provides deep visibility into the entire Redis Enterprise cluster, with metrics such as shard counts and capacity data, helping users evaluate whether their cluster has sufficient resources to handle their workload. The integration is designed for simplicity, with automated discovery of multi-tenant databases for easy setup, and a 14-day free trial available for new users.

Modern DevOps teams often struggle with the increasing volume of logs in dynamic environments, making it difficult to troubleshoot incidents effectively. Datadog's Log Anomaly Detection feature helps automate the process of finding relevant logs during incident investigations by surfacing anomalous patterns at the top of the Log Explorer. This tool breaks down silos between DevOps teams and allows anyone in an organization to efficiently investigate incidents without complex query languages or additional configuration. By prioritizing anomalies based on factors such as status, history, and current state, Log Anomaly Detection helps users identify potential root causes of issues more quickly and reduce mean time to resolution (MTTR).

The text describes a series of network issues that occurred when updates were made to a critical service. Initially, DNS errors were suspected as the cause, but further investigation revealed more complex problems involving dropped packets, connection tracking, and gRPC client reconnect algorithms. Through extensive analysis and testing, the team discovered that the root cause was an aggressive gRPC reconnect parameter that led to a SYN flood during rollouts. By addressing this issue, they were able to resolve the incident and gain valuable insights into their network infrastructure.

Watchdog, an AI-based anomaly detection tool, has introduced its Root Cause Analysis (RCA) feature to automatically identify causal relationships between symptoms across applications and infrastructure, pinpointing the root cause of issues. This hands-free approach significantly reduces mean time to resolution (MTTR). When Watchdog detects an anomaly, it creates a "story" with contextual information such as impacted services and users. The RCA feature maps applications and infrastructure, understanding how components typically interact, and identifies the probable root cause of issues. It also surfaces resulting critical failures. In addition to problematic code changes, Watchdog can detect other root causes like increased traffic from a client, AWS instance failure, or disk reaching its maximum capacity. The tool also provides impact analysis to help prioritize troubleshooting efforts and uses real user monitoring metrics for better visibility into end-user experience.

The investigation into the error began when a routine update to one of their critical services caused an increase in errors. The logs initially pointed to DNS issues, but further analysis revealed that NodeLocal DNSCache was reaching its concurrency limit, causing OOM errors. Increasing the memory allocation for the pods didn't resolve the issue, and it was unclear why the cache was hitting its limit so frequently. Further investigation led to the discovery of a saturated VPC conntrack, which was preventing network connections and leading to DNS errors. Analyzing VPC Flow Logs revealed that clients were sending SYN requests to old IP addresses after pod deletion, causing a high rate of dropped packets. The issue was eventually resolved by changing the gRPC load balancing policy from `pick_first` to `round_robin`, which caused clients to reconnect automatically and reduced the number of SYN requests sent to old IP addresses. The incident highlighted the importance of understanding edge cases within complex systems and the need for careful analysis and testing before making changes that can have unintended effects.

Datadog has introduced Log Anomaly Detection, an out-of-the-box Watchdog Insights feature that automates the process of finding relevant logs during incident investigations. This feature surfaces anomalous patterns at the top of the Log Explorer, reducing mean time to resolve (MTTR) and saving teams time and effort. By leveraging augmented insights, distributed teams can navigate growing volumes of complex data to efficiently investigate incidents without requiring prior knowledge of impacted applications. The feature fits seamlessly into Datadog's broader constellation of Log Management tools, breaking down silos between DevOps teams. With Log Anomaly Detection, teams can streamline incident investigations, reduce MTTR, and document lessons learned by enriching postmortems with historical anomaly data.

Datadog has introduced an update to its Events feature, enhancing its capabilities for managing and analyzing infrastructure and application data. The new Events Explorer replaces the older event stream, offering robust querying and analytics tools that allow users to filter and analyze event data more effectively. Enhancements include the ability to create custom metrics for predictive monitoring, improved dashboards with updated event widgets, and more powerful alerting options using advanced query syntax. Additionally, processing pipelines can now enrich events with important data attributes. These updates aim to streamline monitoring and troubleshooting processes, and the rollout will be completed for all Datadog customers by May 5, with new users immediately benefiting from these enhancements.

Datadog Synthetic Monitoring now supports gRPC testing, allowing users to test the health and response time of their gRPC endpoints, ensuring APIs respond correctly. The framework enables customers to easily implement SSL authentication, load balancing, and tracing via plug-in libraries, making it a popular choice for building complex microservice meshes. gRPC-based APIs can perform requests up to seven times faster than REST APIs. Datadog's Synthetic Monitoring platform provides a range of configuration options, enabling users to customize tests to fit their team's needs. The platform also integrates with services such as Jenkins and GitHub Actions, allowing users to run tests on demand or schedule them for regular intervals. Users can monitor gRPC test results in the Synthetic Monitoring view, including uptime graphs, network timings, and error information, making it easier to detect and resolve issues quickly. Additionally, Datadog's robust alerting capabilities enable teams to notify themselves when a gRPC test is failing, providing valuable context to investigate and troubleshoot issues effectively.

CI Visibility and Real User Monitoring (RUM) can be used together to improve the performance of your CI/CD pipelines and tests. With CI Visibility, you can monitor test performance within default branches and investigate flaky or slow tests by drilling down into individual test runs. RUM provides a waterfall view of events in a test run, helping identify bottlenecks and potential causes of flakiness. Additionally, Session Replay offers a video-like playback to provide context for errors. By optimizing slow tests, you can maintain your team's velocity and improve pipeline performance. Starting from RUM, users can also pivot to CI Visibility data to investigate errors experienced by real users.

Datadog CI Visibility provides insights into the performance of CI/CD pipelines and tests. It allows users to track the performance of their Cypress end-to-end tests using Real User Monitoring (RUM), enabling them to investigate flaky and failing tests, optimize slow tests, and troubleshoot performance gaps. With Datadog's tools, developers can gain deeper visibility into their test suites, identify bottlenecks in their application, and improve the quality and performance of their tests. The integration with RUM allows users to see how real users interact with their application during test runs, providing valuable context for debugging and optimization. By using CI Visibility and RUM together, developers can ensure that their tests are running efficiently and effectively, without sacrificing reliability or user experience.

AWS Function URLs allow users to create serverless applications that can be accessed and triggered by HTTP/S requests. They provide a simple way to connect Lambda functions to the web, making it easier to build and deploy serverless services quickly. Function URLs can also be used to invoke Lambda functions with Response Streaming enabled, which enables streaming response payloads back to clients in parts rather than waiting for entire payloads to fully generate and buffer. Datadog provides end-to-end visibility into requests to functions triggered by URLs, as well as key Function URL and Response Streaming metrics. With Datadog, users can get deep insight into the health and performance of their Function URLs, including real-time visibility into Function URL spans in Lambda Function traces, monitoring of key Function URL metrics, and support for monitoring Lambda functions with Response Streaming enabled.

Recent research indicates that approximately 20% of iOS app uninstalls are due to crashes or other code errors. To manage this potential churn, developers can use Datadog's RUM iOS SDK to collect crash data and correlate it with user metadata and app interactions. This helps determine the scope of application errors, allowing for effective triaging and debugging. The SDK enables collection of logs, traces, and RUM data from mobile applications, as well as crash reports. Datadog's CLI can be used to symbolicate iOS crash data by replacing memory addresses with meaningful symbols in stacktraces. Error Tracking monitors iOS crash reports, groups similar errors together into issues, and enriches each error with key metadata for root-cause analysis. Alerts can be set up using Datadog's event monitors to inform developers of new issues in specific environments or view paths.

The Spring4Shell vulnerability is a critical flaw in the Spring Java framework that allows attackers to bypass protections and execute remote code on vulnerable systems. It was initially confused with another vulnerability but has been identified as a separate issue, now tracked as CVE-2022-22965. The vulnerability affects Spring Core and can be exploited by sending specially crafted HTTP requests, which can lead to unauthenticated remote code execution. Several proofs of concept have been published, and the vulnerability is already being actively exploited in the wild. To remediate the issue, users need to ensure their deployments of the Spring framework are running a version equal to or greater than 5.3.18 or 5.2.20. Additionally, Datadog's Security Platform can provide automatic detection of Spring4Shell exploit attempts and post-exploitation activity within systems.