July 2020 Summaries

The increasing support for Windows containers by cloud providers and infrastructure technologies is enabling developers in the Windows ecosystem to benefit from containerization. Datadog offers deep visibility into Windows containers, allowing users to monitor application activity and make informed decisions on scaling. Its real-time monitoring capabilities provide insights into resource usage, node state, and other metrics across various environments like Kubernetes, Docker, Google Kubernetes Engine, Azure Kubernetes Service, or Amazon Elastic Kubernetes Service. Datadog's Autodiscovery feature helps track containers created by container orchestration technologies, while its Agent collects logs and distributed request traces for deeper visibility into the environment. With over 650 integrations, users can monitor their entire stack as they migrate to a modern, cloud-native environment.

Organizations adopting serverless architecture may find themselves managing numerous components for a single application, which can lead to performance issues. To effectively troubleshoot these problems, access to detailed observability data from serverless functions is crucial. Deployment tools like AWS Serverless Application Model (SAM) and AWS Cloud Development Kit (CDK) help developers build applications in a clean and coherent way using infrastructure as code. Datadog has built integrations with both of these tools to automatically collect metrics, traces, and logs from Lambda functions without any code instrumentation. By streamlining how you monitor your applications, Datadog reduces time to value and enables wider adoption of serverless across teams.

Datadog provides deep visibility into Windows containers, allowing developers to monitor application activity and make informed decisions on how to scale. The Datadog Agent automatically pulls real-time data from each host and container, populating customizable dashboards with metrics such as resource usage and node state. Container orchestration technologies like Kubernetes or Amazon ECS can be managed using the Datadog Agent's Autodiscovery feature, which automatically detects services running on containers as they are created. Additionally, Datadog collects logs and distributed request traces from Windows containers, enabling developers to find the source of errors or resource bottlenecks and determine the impact on their containers' vital signs. With over 850 integrations, Datadog provides a comprehensive view of the entire infrastructure and stack, allowing developers to migrate to a modern, cloud-native environment with ease.

The Datadog CloudFormation macro is an integration that helps organizations manage serverless observability data from their Lambda functions, without writing boilerplate code. It enables engineering teams to grow with Datadog by automatically collecting metrics, traces, and logs from Lambda functions, streamlining how they monitor their applications. The macro can be used with AWS SAM and AWS CDK tools to build serverless applications in a clean and coherent way, reducing friction in developing and monitoring applications. With the macro, developers can explicitly manage trace, log, and enhanced metric collection using infrastructure as code, and configure alerts that automatically notify them of issues degrading their user experience. The integration provides a comprehensive view of all serverless functions running in an environment, allowing users to drill down into specific functions and correlate traces with related logs and metrics for greater context around errors or slowdowns.

The OpenTelemetry project combines the OpenTracing and OpenCensus projects to provide standardized APIs, libraries, and tools for capturing distributed request traces and metrics from applications. Datadog supports this project and offers out-of-the-box instrumentation using its tracing libraries. A new Python exporter has been introduced for sending traces from instrumented Python applications to Datadog, with support for exporting metrics coming soon. OpenTelemetry exporters are libraries that transform and send data to one or more destinations. The Datadog exporter enables seamless integration of the OpenTelemetry tracing library into an application and connection to other applications already instrumented with either OpenTelemetry or Datadog libraries.

The OpenTelemetry project merges the OpenTracing and OpenCensus projects to provide a standard set of APIs, libraries, and tools for capturing distributed request traces and metrics from applications and exporting them to third-party monitoring platforms. Datadog is supporting this project and building on it to provide out-of-the-box instrumentation for Python applications using OpenTelemetry's suite of tools and its existing tracing libraries. The new Python exporter for sending traces from instrumented Python applications to Datadog enables seamless integration with other applications already instrumented with either OpenTelemetry or Datadog libraries, allowing developers to easily plug in any exporter without changing their instrumentation. This guide shows how to instrument a basic Python application with OpenTelemetry and how to plug in the new Python exporter to start collecting data, as well as how to instrument a Flask application with Datadog and OpenTelemetry, visualizing distributed traces in Datadog, and starting to instrument with OpenTelemetry and Datadog today.

The Datadog mobile app is now available for both Android and iOS devices, offering on-call engineers the ability to easily triage issues wherever they are. With full access to Datadog dashboards and alerts, seamless integration with on-call notification services like Pagerduty, OpsGenie, and Slack, users can quickly evaluate alert conditions, determine urgency, and decide their next course of action. The app provides in-app access to all Datadog dashboards, allowing for immediate context and correlation with other critical metrics. This portable solution enables cloud monitoring on the go, making it easier for engineers to manage alerts and collaborate with teammates.

Amazon Kinesis Data Firehose is a fully managed service that enables users to ingest, process, and load data from large distributed sources such as clickstreams into multiple consumers for storage and real-time analytics. AWS recently launched a new feature allowing users to stream logs directly from CloudWatch to third-party services like Datadog for further analysis. This integration provides an easy-to-configure process for streaming all AWS service logs to Datadog, offering greater visibility into applications. With Kinesis Data Firehose, users can capture logs from various AWS services and route them to multiple consumers simultaneously without managing additional infrastructure or forwarding configurations. By setting up a delivery stream in the AWS Management Console, users can send logs directly to Datadog or forward logs to multiple destinations by routing them through a Kinesis data stream. Once the new delivery stream is created, users need to create a CloudWatch subscription to route logs to the new stream. This integration automatically includes metadata such as source and AWS service logs are parsed for key attributes, allowing users to create facets and measures for deeper analysis. Datadog's Logging without Limits™ feature enables users to analyze all their logs while storing only the ones they need, reducing costs and enabling efficient monitoring of applications.

Amazon Data Firehose is a service that enables the ingestion, processing, and loading of data from large, distributed sources into multiple consumers for storage and real-time analytics. AWS recently launched a new feature allowing users to ingest AWS service logs from CloudWatch and stream them directly to a third-party service, such as Datadog, for further analysis. This integration allows users to easily capture logs from services like Amazon API Gateway and AWS Lambda in one place and route them to other consumers simultaneously without managing additional infrastructure or forwarding configurations. Users can set up a delivery stream in the AWS Management Console to automatically forward AWS service logs directly to Datadog, eliminating the need for creating separate forwarders. With this integration, users can gain deeper insights into their applications and AWS infrastructure by analyzing logs from various sources, including CloudWatch logs, and using features like Logging without Limits and Log Patterns to control and analyze streaming logs.

The Datadog mobile app for Android and iOS devices is now available, providing on-call engineers with full access to their Datadog dashboards and alerts. The app allows users to triage issues from anywhere, using features such as interactive graphs, native mobile gestures, and in-app access to all Datadog dashboards. With the app, users can instantly correlate alerts with critical metrics, pivot to other service dashboards, and collaborate with teammates. The app is designed for on-call engineers who need to respond quickly to pager notifications, and provides a portable solution that lets them take their cloud monitoring toolkit anywhere.

Datadog has developed and open-sourced two new client libraries for Java and Go, in addition to their existing Ruby and Python libraries. These libraries enable developers to programmatically interact with the Datadog API, allowing them to customize dashboards, search metrics, create alerts, and perform other tasks. The client libraries adhere to the OpenAPI Specification (formerly known as the Swagger Specification), making it easy for developers to contribute to them. The text provides detailed instructions on how to import these libraries into your application code and includes usage examples such as retrieving and updating dashboards, listing metrics, and retrieving logs.

Datadog has released new client libraries for Java and Go, in addition to existing libraries for Ruby and Python. These libraries make it easier for developers to programmatically interact with Datadog's API to customize dashboards, search metrics, create alerts, and perform other tasks. The libraries were generated with the OpenAPI Generator and adhere to the OpenAPI Specification, allowing developers to easily contribute to them. With these new libraries, development teams can build applications that execute Datadog-specific tasks, including retrieving dashboard data, listing metrics, and managing accounts.

Matthew Fornaciari, cofounder and CTO of Gremlin, shares how they use Datadog to monitor their systems and chaos experiments. They leverage template variables for dynamic dashboards, synthetic monitoring for key user flows, and chaos monitoring with the Gremlin integration in Datadog. This enables them to gain insights into system responses under stress, identify failures before they impact customers, and improve overall system reliability.

Reliable systems are vital to meeting customer expectations and downtime not only hurts a company's bottom line but can be detrimental to reputation. The goal at Gremlin is to help enterprises build more reliable systems using Chaos Engineering, which involves proactively testing how a system responds under stress in order to identify and fix failures before they cascade into customer-facing issues or system downtime. To achieve this, the team uses Datadog for monitoring their own systems, creating dynamic dashboards with template variables to filter key health metrics across multiple environments and apps, and using synthetic monitoring to keep an eye on outgoing changes and how they affect key user flows. Chaos experiments are used to intentionally provoke problems in a controlled manner, monitor the system's response, and use the collected insights to learn how to best mitigate the problem and prevent it from having a future customer impact. The Gremlin integration with Datadog enables users to get more context around their chaos experiments, allowing them to understand how their experiments play out in real time.

The Internet of Things (IoT) is rapidly growing and presents operational challenges due to the large amount of data collected by IoT devices distributed across various environments. Datadog has released an IoT Agent designed for IoT devices and embedded applications, which is a lightweight version of the standard Datadog Agent that collects over 100 health metrics and application logs while consuming fewer resources. The IoT Agent allows users to monitor all their IoT devices within a single unified platform, improving performance optimization, issue resolution, and customer satisfaction. It supports popular IoT operating systems and hardware architectures and enables custom dashboards for monitoring essential metrics across the entire device fleet. Datadog also integrates with technologies at every level of an IoT stack, allowing users to monitor devices, management services, and backend services simultaneously. With over 650 technology integrations supported, it's easier than ever to gain visibility into your entire IoT system.

The Datadog Agent designed for IoT devices and embedded applications is a lightweight version of the standard Datadog Agent that provides full visibility into devices by automatically collecting over 100 health metrics and application logs. The agent can be installed on popular IoT operating systems and hardware architectures with a single command, allowing users to monitor their fleet within a unified platform. With the IoT Agent, users can create custom dashboards to display essential metrics, collect custom metrics, and define alerts for key data that is most important to their IoT fleet. The agent integrates with various technologies at every level of the stack, enabling users to quickly determine if an issue is caused by malfunctioning devices, connection issues, or performance problems. Additionally, Datadog's machine learning-based Metrics Correlations and Watchdog tool provide automatic anomaly detection and alerts to prevent macro-level failures and ensure seamless operations.

OOM errors on Linux systems occur when the kernel can't provide enough memory to run all user-space processes, causing at least one process to exit without warning. Without a comprehensive monitoring solution, OOM errors can be tricky to diagnose. Datadog provides a way to diagnose and analyze OOM errors by collecting and parsing OOM logs, as well as providing automated alerts and notifications when low-memory conditions are detected. The platform allows users to track key memory metrics, identify potential causes of high memory utilization, and set up alerts to notify teams before OOM errors occur. By using Datadog's OOM kill check, users can get direct insights into kernel OOM errors, including the number of errors that have taken place in a particular interval, as well as detailed information on how much memory different processes were using at the time of the error. This enables users to identify which parts of their system are running low on memory and why OOM errors may be occurring, allowing them to take proactive steps to prevent application downtime.

Datadog has introduced private locations in its Synthetic Monitoring service, allowing users to run tests on applications within their private networks. This feature enables organizations to monitor internal applications that employees depend on and launch Synthetic tests in a secured CI environment. Users can also add significant business locations such as factories or call centers for testing purposes. With this addition, teams can compare performance as experienced by users inside and outside the internal network. Setting up private locations is easy with Datadog's built-in integrations and management tools.

Datadog Synthetic Monitoring has introduced private locations, allowing users to run tests on internal-facing services within their private network. This feature enables teams to monitor the performance of internal applications alongside external-facing applications and expand their test coverage by simulating traffic from a broad range of locations. Private locations complement Datadog's managed locations, providing more visibility into application performance and user experience. Users can set up private Synthetic Monitoring locations in seconds and manage them through the Datadog app or using the Datadog Agent. This feature is particularly useful for teams that need to test internal applications before they're deployed to production, ensuring a better user experience and fulfilling service level objectives (SLOs).

Apache Ignite is an in-memory computing platform that enables fast processing and storage of large datasets. Datadog's new integration with Apache Ignite allows users to monitor the health and performance of their Ignite clusters and nodes, collecting node and cluster-wide memory, cache, and storage metrics and logs. The integration also helps monitor the state of Ignite clusters and individual nodes, as well as the performance of all running JVMs. Datadog's built-in Ignite dashboard enables users to track memory usage across their Ignite nodes and create alerts for significant increases in used heap memory. This integration supports various environments, allowing users to always track the performance of their Ignite infrastructure regardless of deployment location.

Apache Ignite is a computing platform that enables storing and processing large datasets in memory, allowing for faster speeds than traditional databases. To monitor the performance of Apache Ignite clusters and nodes, Datadog has announced an integration with version 7+ of the Datadog Agent, which collects node and cluster-wide metrics and logs. This integration provides a comprehensive view of Ignite's performance and allows users to collect only the most important metrics, including cache, memory, thread, page, and job activity. The integration also monitors JVMs used by Ignite nodes, giving users visibility into the state of their nodes and allowing for alerts when a node goes offline or is experiencing issues with memory usage. With this integration, users can track memory usage across every node, respond to OutOfMemoryError exceptions, and gain deeper insights into the health of their in-memory cache, data grid, or database.

Datadog's HIPAA-compliant observability and security solutions provide healthcare organizations with the ability to capture and store monitoring data, including audit logs, distributed traces, RUM events, and more, on a long-term basis. This enables them to verify their level of compliance with other HIPAA provisions and automatically detect security threats and misconfigurations in real time. Datadog's unified platform provides end-to-end visibility into the health, performance, and security of cloud-based healthcare applications, helping ensure they remain reliable and secure for patients. The solution also includes features such as HIPAA-enabled log management, security monitoring, and sensitive data protection to help organizations meet their HIPAA compliance requirements. As a business associate, Datadog is committed to ensuring the security and privacy of its customers' data, with risk-based security controls and compliance programs in place.

Hazelcast is an in-memory computing platform that enables fast data processing with low latency and high availability. It offers clients for popular languages like Java, .NET, and Python. Datadog's integration with Hazelcast IMDG allows users to monitor the health of their data grid and ensure optimal performance. The integration includes built-in service checks, visualization of key metrics, and log correlation. Users can also track map query throughput, optimize cluster performance, and troubleshoot slowdowns using Datadog's features. With Hazelcast IMDG monitoring integrated into Datadog, users gain comprehensive visibility into their data grid health and performance alongside other technologies in their environment.

Hazelcast is a distributed in-memory computing platform that provides low-latency processing of large data sets. Its in-memory data grid (IMDG) stores data in partitions and distributes them among cluster members, ensuring high availability and scalability. The integration with Datadog enables real-time monitoring of Hazelcast cluster health, including key metrics such as cluster size, memory usage, and map operations. This allows users to visualize performance issues and optimize their deployment for optimal performance. By leveraging Datadog's features, users can gain comprehensive visibility into the health and performance of their Hazelcast IMDG environment, alongside other technologies in their ecosystem.