August 2021 Summaries

Statsig is a modern experimentation platform that helps businesses make informed product decisions by automatically running A/B tests on new features and measuring their impact on key business metrics. It integrates with Datadog, allowing users to monitor feature experiments alongside application telemetry data. The platform includes out-of-the-box tools for conducting experiments, such as Feature Gates (feature flags) that control access to new features. Statsig also automatically runs A/B tests and analyzes events related to feature usage. Additionally, the integration with Datadog allows users to monitor configuration change events alongside application telemetry data, helping them detect and resolve any unexpected issues before launching a feature. The dev tier of Statsig is free for up to 5M events per month, and it can be accessed through the Datadog Marketplace.

Statsig is a modern experimentation platform that provides crucial insight into how new features are received by users, enabling informed product decisions and deployment with confidence. It automatically runs A/B tests on features as they're rolled out, measuring their impact on key business metrics, such as user growth and engagement. The platform allows for the control of which users get access to new features through feature flags, and it automatically runs A/B tests for each feature gate, showing how features impact core business by ingesting, aggregating, and analyzing events that occur in the application. Statsig is now available in the Datadog Marketplace, allowing users to monitor configuration change events alongside application telemetry data, helping ensure reliability and performance are not compromised. The platform offers a range of tools for conducting experiments, including feature gates, which can be used to target specific user groups or define rules and conditions for feature access. By integrating with Datadog, users can visualize configuration change events alongside telemetry data, identifying issues before they affect the wider user base. Statsig is now available in the Datadog Marketplace, offering a free trial for up to 5M events per month, allowing users to experience the benefits of automated A/B testing and analytics.

The ASP.NET Core framework allows for cross-platform web development, providing developers with extensive control over the construction and deployment of .NET applications. The text details how to use Datadog to monitor a .NET Core application deployed on Azure App Service, a cloud-based platform for web and mobile applications. It guides readers through creating a sample .NET Core application, deploying it to Azure App Service, and utilizing Datadog’s Azure integration and extension for monitoring performance. This integration captures metrics and logs from Azure resources, while the Datadog extension offers automatic instrumentation and trace collection, enabling developers to track application requests, correlate data with code-level performance, and identify bottlenecks. The document emphasizes the importance of configuring the application with specific environment variables and explains how to install the Datadog extension via deployment scripts or Azure’s UI interfaces, ultimately allowing for enhanced visibility and troubleshooting of application performance via Datadog’s Azure integration tools.

The article provides a comprehensive guide on using Datadog for monitoring PostgreSQL databases, detailing the process of installing the Datadog Agent on PostgreSQL servers to visualize and optimize database performance. It explains how Datadog's integration allows for the automatic aggregation of PostgreSQL metrics through a customizable dashboard, enabling users to track key metrics such as locks, index usage, and replication delay. The text also covers setting up Datadog Database Monitoring for in-depth query-level insights, including query performance tracking and identifying bottlenecks through detailed explain plans. Additionally, the article describes how to use Datadog's Application Performance Monitoring (APM) for tracing PostgreSQL queries and services across distributed systems, offering tools for creating custom dashboards and setting alerts to monitor and improve database and application performance.

This post discusses how to set up comprehensive MySQL monitoring by installing the Datadog Agent on database servers. It covers integrating Datadog with MySQL, configuring the Agent to collect MySQL metrics, and enabling additional checks for more advanced metrics. The post also explains how to use Datadog Database Monitoring (DBM) to view historical query performance metrics, explain plans, and other query-level information. Additionally, it explores tracing MySQL queries with APM and setting up log collection from MySQL to Datadog. By integrating MySQL with Datadog, users can access all their database metrics in one place and easily create automated alerts on any metric.

Conviva is a platform that provides real-time insights into the performance and playback quality of streaming video content. With Datadog's integration, businesses can monitor end viewer experience alongside their infrastructure telemetry for an end-to-end view of their video supply chain. Key Conviva metrics are displayed on an out-of-the-box dashboard, allowing users to quickly identify issues that need troubleshooting. MetricLenses enable users to scope monitoring and alerting to specific views of data, such as content delivery networks serving traffic to a particular region. Datadog collects playback activity metrics like concurrent plays and attempted plays, which can be correlated with telemetry from the rest of the infrastructure. Monitoring video start-up times, rebuffering ratios, and failure percentages helps ensure good end-user experience. With Datadog's integration, businesses gain full end-to-end visibility into their entire video supply chain on a single platform.

Datadog has integrated its platform with Conviva to provide businesses with real-time insights into the performance and playback quality of their streaming video content. With this integration, users can view key Quality of Experience (QoE) metrics alongside infrastructure telemetry, offering an end-to-end view of their video supply chain. This allows for quicker identification of issues that need troubleshooting and enables companies to monitor playback activity, slow start-up times, rebuffering ratios, and other business-critical data to enhance the viewer experience. Datadog's MetricLenses also enable users to scope their monitoring and alerting to specific views of their data, such as video streams or regions, making it easier to identify issues and optimize performance. The integration provides unified end-to-end visibility into the entire video supply chain on a single platform, enabling businesses to monitor Conviva metrics alongside telemetry from over 850 other services and technologies.

Every autumn, Datadog recruits aspiring software developers, product managers, and designers for its Engineering Internship Program. The program provides interns with the opportunity to work on meaningful projects, learn from industry leaders, and build professional relationships. Despite being fully remote due to the pandemic, Datadog is now offering the option for interns to join them at offices worldwide. The primary goal of the internship program is to create a supportive learning environment where interns can hone their skills. Interns participate in weekly one-on-one sessions with managers, mentors, and buddies, and they have access to colleagues throughout the week for guidance or casual conversation. The high-growth nature of Datadog enables interns to make a significant impact on the product while learning from experienced professionals at the forefront of innovation. Interns are treated like full-fledged employees and contribute to features used by thousands of customers daily. They work alongside seasoned engineers, learn how to handle pressure, escalate incidents when necessary, and investigate root causes with confidence. Interns also have the opportunity to work on custom solutions due to Datadog's scale, which is unique in the industry. Interns participate in various social events, such as lunches, hangout sessions, outings, and fireside chats with leaders, allowing them to meet new people and build lasting relationships. These events also provide opportunities for interns to interact with senior leadership. Datadog offers highly competitive salaries and benefits packages to its interns, providing the equipment and guidance they need to be successful. Many graduates from the program receive full-time job offers, with some even growing into leadership roles within the company. Datadog is currently seeking talented interns to join their Engineering Internship Program for the next phase of their journey.

Datadog's Engineering Internship Program offers aspiring software developers, product managers, and designers the opportunity to work on meaningful projects, forge professional relationships, and gain valuable experience in a supportive learning environment. The program provides interns with the chance to contribute to features used by thousands of customers daily and learn from best-in-class leaders who drive innovation. With the option now available for interns to join Datadog offices around the world, this program offers a unique blend of remote and in-office work experiences. Interns can participate in weekly one-on-one sessions with managers, mentors, and buddies, as well as company-wide social events and activities that foster lasting relationships and personal growth. Upon completion of the internship, interns are offered highly competitive salaries and benefits packages, and many graduates go on to receive full-time job offers from Datadog or grow into leadership roles within the company.

In serverless environments, logging is crucial for monitoring and troubleshooting applications, as developers lack direct infrastructure access and must rely on logs from AWS services like Lambda, API Gateway, DynamoDB, and Step Functions. The text emphasizes the importance of standardizing log formats, setting appropriate log levels, and including useful context to maximize log value. Centralizing logs with tools like Datadog facilitates sophisticated analysis and correlation with other monitoring data, enabling efficient issue identification. Additionally, controlling logging costs is vital as logs can be retained indefinitely in CloudWatch Logs, and Datadog's Logging without Limits™ offers cost-effective solutions by allowing dynamic indexing decisions and turning logs into metrics. The article encourages using these best practices to achieve deep visibility into serverless applications and suggests Datadog for integrating and analyzing logs comprehensively.

Understanding the Linux process tree is crucial for detecting security threats, as it's difficult for attackers to fake or change. Monitoring launched shells and utilities can help identify malicious activities such as web shell attacks or unauthorized access attempts. Process data like environment variables and command-line arguments can provide insights into the scope of an attack. Datadog Cloud Workload Security helps detect threats in Linux processes by analyzing the process tree across all hosts and containers, automatically flagging suspicious behavior and providing full context around detected processes for effective threat response planning.

Datadog APM now provides Automatic Faulty Deployment Detection to help modern software development teams deploy features quickly while mitigating the risk of faulty deployments. This feature uses machine learning algorithms to spot faulty deployments within minutes, reducing mean time to detection (MTTD). Once detected, it provides detailed information about the affected service, including error types, error rates, request rates, and latency metrics. Automatic Faulty Deployment Detection also enables users to troubleshoot faulty deployments quickly by exploring service traces, create alerts to notify their team of a faulty deployment, and proactively set alerts to monitor future deployments. By using this feature, teams can deploy with confidence, reduce the risk of user-facing bugs, and maintain both velocity and quality in their development process.

In Linux systems, monitoring processes is crucial to detecting potential security threats, such as the creation of unexpected web shells or other malicious utilities. Understanding the process tree can help identify security threats and determine the scope of a breach. Key information includes environment variables, command-line arguments, and metadata that can reveal sensitive data or activity data used by attackers. Datadog Cloud Workload Security can help monitor processes across an entire environment to surface security threats in real-time, with out-of-the-box workload threat detection rules and custom rule writing capabilities.

Datadog Cloud SIEM has introduced anomaly detection rules to enhance the security of cloud environments by identifying and alerting on unusual activity. This feature allows for the analysis of logs to establish baseline behavior for specific entities such as hosts, IP addresses, and users, and generates Security Signals when deviations occur. Unlike threshold-based detection, which requires predefined limits, anomaly detection dynamically adjusts to historical behavior, helping to monitor activities like API calls or access requests that could indicate compromised accounts. For instance, it can detect unusual API activity from service accounts or anomalous spikes in Salesforce user queries, which might suggest unauthorized access attempts. Security Signals provide comprehensive data, including event times and associated user information, enabling quick investigation and response. These signals remain active as long as the anomaly persists, helping to determine its duration and impact. Datadog’s new feature is available to current customers and new users can explore it through a 14-day free trial.

Black Hat USA 2021 was a hybrid event that brought together in-person and virtual attendees in Las Vegas. The conference saw nearly 14,600 attendees participating virtually, making it the largest hybrid conference in cybersecurity since the shift to virtual events. Datadog participated as both an exhibitor and speaker at this year's event, showcasing its Cloud Security Posture Management product and announcing the general availability of Datadog Cloud Workload Security. The event highlighted the importance of collaboration among development, security, and operations teams in advancing cybersecurity efforts. Two notable briefings were "Cloudy with a Chance of APT: Novel Microsoft 365 Attacks in the Wild" and "I’m a Hacker Get Me Out of Here! Breaking Network Segregation Using Esoteric Command & Control Channels."

Database Monitoring is a new feature that provides deep visibility into databases across all hosts. It enables developers and database administrators to easily understand the health and performance of their databases, as well as quickly troubleshoot any issues that arise. The tool allows users to see the performance of normalized queries at a glance, analyze historical trends in query performance, explore and visualize sampled queries, and detect infrastructure-level issues impacting the database. Database Monitoring currently supports MySQL 5.6+ and PostgreSQL 9.6+ databases, regardless of whether they're self-hosted or fully managed.

The Black Hat USA conference, one of the industry's oldest and most well-established security events, took place this year as a hybrid event, combining in-person attendance with virtual participation. The event drew nearly 14,600 attendees who logged into the virtual platform, making it the largest hybrid conference in cybersecurity since the shift to virtual events. Datadog participated both as an exhibitor and speaker, showcasing its Cloud Security Posture Management product and announcing the general availability of Datadog Cloud Workload Security. The conference highlighted the growing demand for threat detection solutions, particularly cloud-native and managed SIEM solutions. Keynotes emphasized the importance of collaboration in moving security efforts forward, with a focus on DevOps and external government partnerships. The Black Hat Briefings featured a diverse range of topics, including cloud-targeted attacks, privilege escalation, and full-stack security with Datadog's Cloud Security Platform.

Database Monitoring is a tool that provides deep visibility into databases across all hosts, enabling developers and database administrators to easily understand the health and performance of their databases and quickly troubleshoot any issues that arise. It delivers historical query performance metrics, explain plans, and host-level metrics in one place, allowing users to see the performance of normalized queries at a glance, troubleshoot slow queries with detailed explain plans, analyze historical trends in query performance, explore and visualize sampled queries, and detect infrastructure-level issues impacting their database. The tool currently supports MySQL 5.6+ and PostgreSQL 9.6+ databases, regardless of whether they're self-hosted or fully managed.

The Serverless view has been fully redesigned by Datadog to provide a more seamless debugging experience for serverless applications. This new feature unifies telemetry data from Lambda functions and other AWS resources, giving users a full overview of their entire serverless stack. By default, the Serverless view groups resources by service, but users can also group them by CloudFormation stack name or any other tags they've configured. The Serverless view enables users to correlate high-level metrics from AWS resources with those of Lambda functions, allowing for quick detection and debugging of performance issues across their stack. Currently, only Python and Node.js functions are tied to related resources, but support for more runtimes is planned in the future.

The newly redesigned Serverless view in AWS provides a unified telemetry data dashboard that unifies data from Lambda functions and other AWS resources, enabling developers to monitor, debug, and optimize their serverless applications more efficiently. The view groups resources by service or custom tags, allowing for easy visualization of application performance. It also enables correlation of high-level metrics with those of Lambda functions, facilitating quick detection and debugging of performance issues across the entire stack. With this new feature, developers can start monitoring their serverless applications directly within the Serverless view, using tools like Datadog APM for tracing and AWS X-Ray to track application performance.

Containers are portable and scalable but introduce monitoring challenges due to complex network communication in distributed environments. Datadog Network Performance Monitoring (NPM) helps visualize network traffic between objects within containerized environments, enabling users to monitor dependencies across containers, services, and deployments. NPM also supports service mesh analysis for identifying misconfigurations and ensuring efficient communication between services. The tool provides real-time visibility into network topology, allowing users to spot architectural inefficiencies and troubleshoot performance issues. Additionally, Datadog APM offers insights into application layer issues, while DNS monitoring helps identify service discovery problems. With support for Istio integration and Envoy monitoring, NPM provides full visibility into containerized applications and their communication.

Amazon Elastic File System (EFS) provides shared, persistent, and elastic storage in the AWS cloud. It enables you to monitor EFS latency, I/O, throughput, and connections to ensure the performance of services and applications that access your file systems. EFS is based on the Network File System (NFS) protocol and automatically handles data consistency and manages file locking to safely allow for parallel access from multiple clients. It supports various use cases such as big data workloads, machine learning, and serving web content. Monitoring EFS can help you understand costs and ensure optimal performance of your applications. Key Amazon EFS metrics to monitor include storage, latency, I/O, throughput, and connection metrics.

Microsoft Azure and Datadog have partnered to streamline the process of purchasing, configuring, and managing Datadog directly within the Azure portal. This integration simplifies the onboarding experience for users, allowing them to visualize real-time Azure metrics in their Datadog account quickly. The partnership also enables single sign-on with Azure Active Directory during Datadog account creation. Additionally, users can manage the integration between Azure and Datadog using a native resource blade within the Azure portal. This consolidated billing approach simplifies purchasing and invoicing processes for new Datadog customers, allowing their usage to appear directly on their Azure invoice.

Datadog Cloud Network Monitoring provides visibility into network traffic between objects within containerized environments, making it easy to monitor network dependencies across all containers, services, and deployments. This feature enables users to visualize network communication with directional arrows, analyze service mesh traffic, and troubleshoot performance issues in containerized applications. It also supports DNS monitoring and provides full visibility into each layer of the containerized application, including the application layer and the network layer. By using Datadog Cloud Network Monitoring, users can identify architectural and performance issues quickly, spot misconfigurations, and take mitigating steps to reduce latency.

Amazon Elastic File System (EFS) provides shared, persistent, and elastic storage in the AWS cloud. It offers features such as simultaneous access from multiple clients, AWS Lambda integration, and supports big data workloads, machine learning, and serving web content. EFS operates in two performance modes: General Purpose mode, which provides low latency for most use cases, and Max I/O mode, which provides higher IOPS at the cost of additional latency. The storage class determines the availability and costs associated with storing data, with Standard being the default and One Zone providing reduced costs but lower availability. Monitoring EFS metrics such as file size, latency, I/O utilization, throughput, and connection count is crucial to ensure the performance and health of applications accessing shared storage.

David M. Lentz discusses the importance of comprehensive monitoring of Amazon Elastic File System (EFS) using Datadog to ensure full visibility into application health and performance. By integrating EFS with Datadog's AWS monitoring tools, users can visualize EFS metrics, receive alerts on activity and performance, and collect logs to understand the broader context of their AWS environment. The platform allows for the correlation of EFS data with metrics from other AWS services and technologies, thereby providing a more complete picture of system performance. Datadog's capabilities include automatic tagging, customizable dashboards, and anomaly-based alerts, which help to identify and address potential issues proactively. Additionally, the integration of EFS logs into Datadog via Kinesis Data Firehose further enhances the ability to explore and analyze data across the technology stack.

David M. Lentz's article explores methods for collecting and analyzing Amazon Elastic File System (EFS) metrics and logs to better understand their impact on application performance. The article discusses using the EFS console for basic monitoring and CloudWatch for more advanced metric visualization and alerting, including setting up custom dashboards and alarms. It also covers programmatic access to metrics via the CloudWatch API, with examples of using AWS SDKs and the CLI for integration into processes or applications. Additionally, the article highlights Linux tools for collecting client-specific metrics and logging EFS file system activity, detailing how these tools can identify changes that may indicate performance issues or security concerns. The use of AWS logging services, such as CloudWatch Logs and VPC Flow Logs, is explained for gathering and analyzing log data from EFS clients, while non-AWS tools like auditd and rpcdebug are recommended for monitoring file system changes at the host level. The piece concludes by mentioning that the upcoming part of the series will discuss integrating this data with Datadog for comprehensive stack monitoring.

The text discusses best practices for monitoring cloud migration and how Datadog can provide visibility throughout the process. It covers planning resources needed, building visibility into the cloud environment, and monitoring newly migrated applications. The article also provides tips on preparing, creating, and testing new environments, as well as validating, testing, and protecting cloud architectures. Finally, it explains how to track SLOs and migration progress using dashboards, monitor end-to-end with RUM and APM, and confirm the completion of a successful migration.

The text discusses best practices for monitoring and managing cloud migrations. It emphasizes the importance of planning, preparing, and testing a cloud environment before migrating an application. Datadog is highlighted as a tool that provides visibility throughout every phase of the migration process, including planning, setup, validation, and ongoing management. The article covers various aspects of cloud migration, such as monitoring network traffic, storage usage, and resource utilization, as well as security and performance metrics. It also discusses the use of Synthetic Monitoring to test API endpoints and key user workflows before and after a migration. Ultimately, the text aims to help developers and organizations successfully migrate their applications to the cloud while maintaining visibility and control over their infrastructure and application performance.

This guide demonstrates how to use Datadog to provide visibility into ASP.NET Core applications running on AWS Fargate. It covers the process of instrumenting and packaging a sample .NET application, publishing it to Docker Hub, deploying the instrumented .NET application using AWS Fargate, and monitoring application performance with Datadog APM. The guide also explains how to create an Amazon ECS cluster, an ECS task definition, an application load balancer, and an ECS service for launching the application and its resources. Finally, it discusses how to monitor application performance using Datadog's built-in integration dashboard and visualize traces with Datadog APM.

The guide demonstrates how to instrument a .NET Core application with Datadog's .NET tracer and deploy it as a container on AWS Fargate. It shows how to package the application, publish it to Docker Hub, create an ECS cluster, task definition, load balancer, and service for the Fargate deployment. The guide also explains how to monitor application performance with Datadog APM, visualize traces, and connect .NET logs to traces. Additionally, it provides information on how to use Datadog's .NET tracer to instrument a .NET Core application and deploy it as a container on AWS Fargate.