June 2023 Summaries

The Frontend Developer Experience team at Datadog sought to improve the lives of 300 frontend engineers by addressing pain points such as difficult-to-maintain acceptance tests. They migrated from Puppeteer (Chromium Headless Browser) to Synthetic tests, which are more robust and maintainable. The team identified pain points through surveys and addressed the flakiness of acceptance tests with a new solution using their own product, Synthetic Monitoring. They developed a CLI runner called synthetics-ci for executing Synthetic tests from the CI and later generalized it to datadog-ci. The migration process took one year, during which they gained engineers' trust in the system by providing documentation, presentations, and tooling support. They also tracked the progress of each test migration in Jira. Through this experience, the team learned valuable lessons about changing ingrained processes, working together to improve products, and building trust with users.

The Frontend Developer Experience team at Datadog aimed to improve the lives of 300 frontend engineers by addressing pain points such as difficult-to-maintain acceptance tests. They identified flaky and hard-to-implement acceptance tests using Puppeteer, which were a significant challenge due to their complexity and maintenance requirements. The team decided to use Synthetic Monitoring, a product that allows recording page interactions without manual scripting, to replace the old approach. To implement this change, they created a CLI runner called `synthetics-ci` and generalized it into `datadog-ci`, which enables users to execute commands with Datadog from within CI/CD scripts. The team drove adoption through trust, information, and tooling by writing documentation, holding presentations, and working with specific teams to implement new tests. They also implemented a non-blocking job in their CI/CD pipeline to build trust among engineers without impacting the entire workflow. Through this migration process, which took over a year, the team improved one of their products, learned how to work together better, and created tools that serve the frontend community's needs.

Content delivery networks (CDNs) store copies of content in geographically distributed servers to extend the reach of applications without sacrificing performance. CDNs reduce demand on individual web hosts by increasing the number and regional spread of servers that can respond to incoming requests for cached content, enabling faster delivery of web content and a better experience for end users. They are ideal for delivering content requiring large amounts of data transmission, such as music or video streaming. Additionally, CDNs help reduce the impact of network attacks by decentralizing web processing and enabling quick re-routing of traffic. Monitoring CDN logs can provide critical insights into system health, including issues in resource utilization and network security. CDN logs catalog information about incoming and outgoing web requests, helping determine who is visiting an app, how often they visit it, and what they access when they are there. They also include information about request performance, such as processing time and response size. Datadog provides features to aggregate, visualize, and correlate CDN logs alongside metrics and traces from the entire system, enabling users to see all their data in one view.

Kubernetes generates events to document changes in its components like nodes, pods or containers. These events provide crucial information about the health and status of clusters, helping troubleshoot issues affecting infrastructure. However, as the scale of a Kubernetes environment increases, so does the volume of generated events. Monitoring these events can help identify problems related to pod scheduling, resource constraints, access to external volumes, and other aspects of the Kubernetes environment. Datadog provides comprehensive support for monitoring Kubernetes events, including real-time event collection, retention, filterable event feeds, custom metrics creation from events, monitors for critical events, and integration with timeseries and dashboards.

CDN logs provide valuable insights into network performance and security by cataloging information about incoming and outgoing web requests, including request performance metrics such as latency and response size. The layout of raw CDN log data can vary based on the specific provider, but it typically includes basic information like IP address, username, timestamp, request action, HTTP version, and status. To effectively utilize this information for troubleshooting and analysis, it's essential to monitor CDN logs regularly and use tools like Datadog to collect, process, and visualize log data. Datadog offers features such as Observability Pipelines, Log Management, Cloud SIEM, and Network Performance Monitoring (NPM) that enable users to extract metrics from their CDN logs, analyze historical trends, and identify potential security threats in real-time. By leveraging these tools, organizations can gain a better understanding of their network's health and performance, making it easier to detect and respond to issues before they impact the user experience.

Kubernetes generates events when components like nodes, pods, or containers change state. These events provide key information about the health and status of a cluster, including container creation failures, eviction events, volume events, scheduling events, and unready node events. Monitoring these events can help troubleshoot issues affecting infrastructure. To effectively monitor Kubernetes events, tools like `kubectl` and Datadog can be used to filter, view, and analyze events in real-time, providing rich contextual clues for troubleshooting problems related to pod scheduling, resource constraints, access to external volumes, and other aspects of a Kubernetes environment. By identifying the types of events that help troubleshoot issues more efficiently, teams can reduce mean time to resolution (MTTR) when issues occur.

Mobile Vitals is a set of metrics that track fundamental health and performance indicators from mobile applications, including Android, iOS, React Native, and Flutter. These metrics help ensure a positive experience for mobile users by monitoring aspects such as slow renders, JavaScript slow renders, frozen frames, application not responding, crash-free sessions, CPU utilization, and memory utilization. By tracking these vitals, developers can gain comprehensive visibility into their mobile app performance and address any issues that may arise, ultimately improving user experience and engagement.

Twingate is a network access platform that allows customers to deploy zero trust authentication layers with their infrastructure as code (IAC) provider of choice. By integrating Twingate with Datadog, users can visualize key network events using the Twingate dashboard and configure automatic alerts on suspicious access activity. The integration enables monitoring of network activity alongside other resources in a unified platform. Users can purchase a subscription to Twingate in the Datadog Marketplace and sign up for a free 14-day trial of Datadog.

Google Cloud Platform (GCP) integration with Datadog has been updated to provide faster and more secure setup experience. The new features include automatic discovery of projects, an improved integration tile layout for better visibility, and service account impersonation for secure access. These updates enable seamless monitoring across large numbers of projects within the Google Cloud environment. To get started using the new setup, users can install the GCP integration and follow the step-by-step instructions provided in the documentation.

Twingate is a network access platform that enables customers to deploy a zero-trust authentication layer with their infrastructure as code (IAC) provider of choice, allowing for strict access control rules and real-time monitoring. After installing the Twingate integration, Datadog populates an out-of-the-box dashboard with key network events, such as recent Twingate events, resources accessed by clients, and data volume traveling across the network. This dashboard can be used to monitor security audit events in near-real time, identify suspicious clients accessing high-risk countries or under imposed sanctions, and track entity activity across cloud platforms. Additionally, log-based monitors can be configured to automatically alert on suspicious activity, such as spikes in data transfers, access attempts from public IP addresses, and more, enabling immediate response within Datadog Incident Management. By integrating Twingate with Datadog, customers can monitor their network's access activity alongside the rest of their environment in a unified platform, promoting security and compliance.

The Google Cloud Platform (GCP) integration with Datadog has been improved with a faster, streamlined, and more secure setup experience. This new setup allows users to quickly monitor their GCP environment by automatically discovering new projects and assigning access across multiple projects using a single service account. The integration tile has also been updated to provide better visibility at a glance, making it easier for enterprises to manage large numbers of projects seamlessly. Additionally, the integration now uses service account impersonation for secure remote access, removing the need for static credentials and improving security. With these changes, users can enable visibility across their entire organization in minutes, leveraging automatic discovery, intuitive tile layout, and secure remote access features.

NetFlow is a network protocol that provides key traffic flow data, enabling administrators to understand bandwidth utilization and identify network bottlenecks, unlike SNMP, which offers broader network-device telemetry. Datadog has introduced NetFlow monitoring through its Network Device Monitoring (NDM) view, allowing users to visualize NetFlow data and variants like IPFIX and sFlow within the platform. This feature helps organizations identify top network traffic contributors, optimize resource utilization, and plan for capacity upgrades. By visualizing critical data on a customizable dashboard, administrators can monitor the state of their network, address congestion, and prioritize essential applications to ensure smooth operations. With the integration of NetFlow monitoring, Datadog users can diagnose network issues alongside SNMP Trap data, aiding in troubleshooting and network health management.

On March 8, 2023, Datadog experienced a global outage that affected all services across multiple regions due to an unexpected system patch. The company's teams worked for approximately 13 hours to restore the majority of their compute capacity across all regions. They faced several challenges during this process, including different responses required for each region, limitations on the maximum number of VM instances in a peering group, and reaching subnet capacity limits. Despite these obstacles, Datadog managed to recover its platform-level capabilities and continue working towards complete recovery.

The Datadog platform experienced a global outage that affected all services across multiple regions, resulting in a loss of 60 percent of compute capacity. The teams quickly realized they needed to restore the platform and worked through decisions whose outcomes and downstream effects were not immediately obvious. They recovered the EU1 region by rebooting impacted nodes, progressively recovering clusters, and ultimately restoring 100 percent compute capacity. However, they faced challenges in other regions, including US1, where instances remained in an "Unregistered" state due to Vault instability, and scaling up to process a backlog of data caused issues with node registration and IP allocation. The teams learned several lessons, including the importance of strong relationships with partners, testing for global failure scenarios, and preparing for sudden capacity increases. They successfully restored their platform-level capabilities and are now working on restoring the Datadog application.

GitLab is a DevSecOps platform that automates software delivery, and its integration with Datadog enhances monitoring capabilities by collecting metrics, logs, and service checks. This integration allows users to visualize the health of their GitLab environment through a comprehensive dashboard that covers various components such as Rails, Redis, and Gitaly, providing insights into potential roadblocks and bottlenecks in the CI/CD workflow. Datadog's GitLab integration not only helps identify errors and performance issues, such as network traffic discrepancies and high latency, but also offers solutions to optimize resource allocation and ensure smooth pipeline execution. By leveraging Gitaly metrics and other data, users can proactively address bottlenecks, ultimately improving productivity and reducing downtime. Additionally, Datadog's CI Visibility feature provides further insights into the continuous integration process, helping to identify issues such as flaky tests that may compromise builds.

The text discusses the integration of Fiddler, an AI observability platform, into the Datadog Marketplace, enhancing the monitoring and performance analysis of machine learning (ML) models. This integration allows users to centralize their ML model monitoring, providing real-time alerts for performance issues and offering insights into root causes of these issues. Fiddler's dashboards enable tracking of model performance metrics, data drift, data integrity, and AI fairness, among others, to ensure accurate and reliable model predictions. The platform aims to increase transparency in ML models, fostering trust among business leaders and regulators, and supporting compliance through model diagnostics and explainability methods. The integration offers a 14-day trial for users to explore its capabilities in troubleshooting and optimizing ML model performance before issues impact business applications.

Classless Inter-Domain Routing (CIDR) is the primary IP addressing scheme used in modern networks. It enables network engineers to create subnets that encapsulate a range of IP addresses, facilitating efficient allocation of IPs in virtual private clouds (VPCs) and other networks. Datadog Log Management offers an intuitive syntax for CIDR queries, allowing users to search logs within a specific subnet by filtering on its corresponding CIDR block. This feature is useful in various scenarios such as troubleshooting network issues, monitoring security threats, and analyzing network performance. By using CIDR notation queries, users can identify traffic from unexpected sources, analyze network failures, and investigate network communication within a VPC.

Datadog Log Management offers an intuitive syntax for CIDR queries, allowing users to search for logs within a subnet by filtering on its corresponding CIDR block. This enables network engineers and analysts to efficiently allocate IPs in virtual private clouds (VPCs) and other networks. By using CIDR notation, users can identify traffic from unexpected sources, analyze network performance, and spot failures. The syntax also makes it easy to create supernetting within a VPC, improving network traffic efficiency by allowing request data to reach the destination address without taking unnecessary paths.

As a new Datadog customer, the platform offers built-in integrations and Recommended Monitors to help users quickly maximize its potential and deliver value to their organization. The latest addition is Azure Recommended Monitors, which provide preconfigured alert queries and thresholds for various Azure services. These monitors are based on expertise from internal teams, technology partners, and thousands of customers, ensuring that any issues within an environment will be promptly addressed. Azure Recommended Monitors can be viewed or added to out-of-the-box dashboards in the Datadog platform, allowing users to obtain relevant information from a single location. Additionally, these monitors include machine learning-powered anomaly detection and service health checks. By leveraging Azure Recommended Monitors, users can easily incorporate actionable alerts into their monitoring workflow and quickly start investigations with detailed next steps.

Testing is crucial to ensure an optimized user experience (UX) in applications. Combining observability data from real-user monitoring (RUM) and synthetic monitoring provides full visibility into end-user experiences, enabling users to analyze UX from actual and optimal conditions. Datadog allows users to view key performance metrics for RUM and Synthetic Monitoring sessions side-by-side, providing insights into frontend experience data from both test and real-user sessions. This helps identify the source of issues quickly and deliver a reliable end-user experience. By analyzing real-user session data from RUM, users can gain better insight into which user journeys to examine and create useful, relevant tests. Combining RUM and Synthetic Monitoring provides complete visibility into user journeys, enabling users to optimize their app and delight customers.

With Datadog's Azure Recommended Monitors, users can easily maximize the platform's potential and deliver value to their organization quickly and seamlessly. These preconfigured monitors provide alert queries and thresholds for various services, allowing users to get started with confidence in minutes. The monitors are based on firsthand expertise and feedback from Datadog's internal teams, technology partners, and thousands of customers, ensuring that issues within the environment will be brought to attention at the right time. Users can start monitoring their Azure resources at scale in minutes, receive alerts on performance, and obtain contextual data for quick investigations. The monitors also include machine learning-powered anomaly detection and service health checks, enabling users to safeguard and optimize their Azure environment. With Datadog's out-of-the-box dashboards and recommendations, users can troubleshoot with the right metrics from one convenient location, making it easy to resolve issues, prevent future disruptions, and incorporate actionable alerts into their monitoring workflow.

Datadog's RUM and Synthetic Monitoring tools provide a comprehensive view of an application's frontend experience, enabling users to identify and resolve issues before they break critical functionality. By combining data from both real-user and synthetic sessions, users can analyze their UX from two different angles, assess the impacts of issues, and identify root causes faster. Datadog enables users to view key performance metrics for RUM and Synthetic Monitoring sessions side-by-side, including core web vitals, API test response times, and browser performance metrics. This allows users to streamline test design with real-user data, provide a reliable UX with insights from RUM and synthetic sessions, and troubleshoot issues more efficiently. Additionally, Datadog's Test Coverage feature helps users build effective synthetic tests by leveraging RUM data to reveal discrepancies between their test design and actual user workflows.

Datadog streamlines the onboarding process for its monitoring and security platform by offering over 850 built-in integrations, out-of-the-box dashboards, and preconfigured alerts, known as Recommended Monitors, which are based on expertise from internal teams, technology partners, and customer feedback. These monitors provide alert queries and thresholds for various services, including specific AWS services like EC2, RDS, Lambda, and SQS, adhering to AWS service monitoring guidelines and best practices. Users can start monitoring AWS resources quickly by enabling these monitors, which offer features such as machine learning-powered anomaly detection and customizable alert settings to maintain optimal AWS environment health. The system is designed to be user-friendly, reducing configuration complexities, and allowing users to integrate actionable alerts into their workflows efficiently. For further guidance and to explore additional features, users can refer to Datadog's documentation and may also try a 14-day free trial.

Datadog's Cloud Cost Management provides deep visibility into cloud spend, enabling teams to understand and optimize their costs effectively. Key features include Cost Monitors for proactive detection of unexpected cost changes, customizable notifications based on significant cost increases or decreases, and enforcement of standardized tagging with Tag Pipelines. Additionally, Workflow Automation allows automatic mitigation of cost inefficiencies by triggering actions when monitors are triggered. These features help organizations minimize cloud cost overruns and maintain complete visibility into their cloud spend.

Datadog has introduced support for EKS, ECS, and Kubernetes in AWS through its Cloud Cost Management feature. This enables organizations to allocate costs and provision shared resources more accurately by leveraging existing observability data from Datadog. The new feature provides granular precision at the pod or task level and clearly maps Kubernetes and ECS costs to services, products, and teams. It helps users understand the true costs of their containerized applications, optimize container usage by monitoring idle spend, and centralize cloud monitoring and cost management.

Datadog Cloud Cost Management provides deep visibility into cloud costs, enabling teams to understand their spend and ensure cost efficiency. FinOps teams can use Cost Monitors to proactively detect unexpected cost changes, while DevOps teams can quickly mitigate these changes to minimize overruns. The monitors provide a complete view of cloud costs, including allocation by team, service, and product, and can be configured to alert on specific dollar amounts or percentage-based changes. Additionally, Tag Pipelines help enforce standardized tagging, ensuring that cloud resources are properly allocated and cost data is not missed. Workflows can also be triggered automatically when monitors detect unnecessary costs, allowing teams to quickly remediate these issues and keep costs in check. By leveraging Cost Monitors, teams can minimize cloud cost overruns and ensure complete visibility into their organization's cloud spend.

Datadog Cloud Cost Management now supports Kubernetes, EKS, and ECS in AWS, enabling precise allocation of costs and resources with granular precision at the pod or task level. This allows users to understand the true costs of their containerized applications, optimize usage by monitoring idle spend, and cultivate a culture of cost awareness among engineering teams. With Cloud Cost Management, organizations can break down costs into specific teams, services, and products, making it critical for determining business margins, enact chargeback and showback policies, and assess the overall health of applications. The platform provides a unified platform for monitoring and managing container infrastructure and its costs, allowing users to precisely attribute costs and allocate resources with confidence.

Datadog provides a comprehensive and customizable platform for monitoring application performance and security. The company has introduced Remote Configuration, which enables users to remotely configure the behavior of Datadog components deployed in their infrastructure. This feature allows users to strengthen their overall security posture, decrease management costs, and shorten resolution times. With Remote Configuration, users can receive real-time protection from attacks targeting web applications and APIs, update security detection rules for Cloud Workload Security, instrument tracing libraries, configure Observability Pipelines Workers, capture data from live applications, and more. To use this feature, users need to enable it in their organization's settings page, add a small line of code to their Datadog components' configuration files, and restart the Agent to apply changes to their environment. Remote Configuration is available for supported features such as Application Security Management (ASM) and Cloud Workload Security, allowing users to streamline security monitoring and enhance their overall security posture.

Datadog Workflow Automation is a solution that combines monitoring and remediation into a single streamlined process. It automates and orchestrates entire end-to-end processes across infrastructure and tools, allowing teams to quickly resolve issues. The platform enables the automatic execution of tasks in response to specific alerts, events, and threats, reducing mean time to resolution (MTTR). Datadog Workflow Automation can be used with various integrations such as AWS, Cloudflare, Jira, Github, and more. It also helps teams proactively respond to security threats by enabling workflows to trigger in response to security signals.

Continuous integration (CI) practices often involve large test suites that can slow down the development process. Datadog's Intelligent Test Runner (ITR) addresses this issue by automatically selecting and running only relevant tests for each commit, reducing testing time and resources required. ITR provides test impact analysis to determine which tests are affected by a code change and runs them accordingly. This helps minimize the risk of broken pipelines caused by irrelevant flaky tests. Additionally, the ITR dashboard allows developers to visualize resource savings from reduced testing times. By enabling ITR for their test services, users can accelerate their software delivery process while maintaining test coverage.

Datadog's version history feature for dashboards and notebooks offers users a robust tool for managing and tracking changes, enhancing troubleshooting processes while providing peace of mind. This feature allows users to view a detailed record of alterations, including changes made via API or Terraform, for up to 30 days, or 90 days for Audit Trail customers, enabling them to identify and reverse unwanted modifications swiftly. By facilitating the restoration of previous versions, version history empowers users to experiment and investigate without the fear of permanently impacting key dashboards, ensuring smooth collaboration across teams. This functionality is crucial in maintaining the integrity of dashboards used for generating reports, managing service level objectives, and conducting postmortems, as it allows users to track alterations and contact contributors if necessary. With version history, Datadog enhances the reliability and efficiency of its monitoring tools, encouraging confident and agile problem-solving within organizations.

Datadog experienced a global outage on March 8th, which was the first of its kind for the company. The incident involved several hundred engineers working in shifts and using various communication channels to resolve the issue. This post describes Datadog's incident response process, including monitoring systems, high-severity incident management, training, and a blameless culture. The outage provided valuable lessons on improving internal response, customer communications, and overall preparedness for future incidents.

The incident response process at Datadog involved multiple teams and a large number of engineers working together to resolve the global outage. The team used a "you build it, you own it" model, with all systems instrumented to provide telemetry data to teams. They had a rotation of senior engineers who were on call for high-severity incidents, and a system in place for rapid escalation to executives and customer support. The response was scaled by design, with workstreams and automation used to manage the incident. Despite some challenges with communication, the team was able to recover from the outage within 48 hours. Lessons learned included the importance of autonomy, ownership, and blamelessness, as well as the need for improved training and practice drills.