February 2020 Summaries

Apache Airflow is an open source system used to create, schedule, and monitor complex workflows including data processing pipelines. Originally developed by Airbnb in 2014, it is now part of the Apache Software Foundation with a community of contributing developers. Workflows are represented as Directed Acyclic Graphs (DAGs) made up of tasks written in Python. Datadog has introduced an integration with Apache Airflow that collects metrics using Airflow's StatsD plugin and provides an out-of-the-box dashboard for immediate insight into workloads. The integration allows users to monitor key metrics like DAG duration and task status, ensuring efficient execution of tasks within the DAGs.

Apache Airflow is an open source system for programmatically creating, scheduling, and monitoring complex workflows including data processing pipelines. It represents workflows as Directed Acyclic Graphs (DAGs), which are made up of tasks written in Python, allowing users to programmatically build and modify their workflows. The integration with Datadog provides key metrics like DAG duration and task status, enabling immediate insight into Airflow-managed workloads. To ensure optimal performance, users can track metrics such as average task duration, schedule delay, and concurrency limits, and use features like DogStatsD Mapper to surface slow tasks and DAGs. Additionally, the integration supports distributed tracing with Celery, providing visibility into workflow performance and helping identify latency issues. With this integration, Datadog offers comprehensive visibility into managed workflows, making it easier to meet service level agreements (SLAs) and optimize workflow execution.

Since its inception by Etsy in 2011, StatsD has been a crucial tool for infrastructure monitoring, but its lack of built-in support for tagging metrics has posed challenges for modern dev and ops teams working with containerized microservices. To address this limitation, Datadog introduced DogStatsD Mapper, which automatically converts parts of StatsD metric names into tags based on user-configured rules, without the need to re-instrument code or use separate servers. The DogStatsD Mapper is integrated into the Datadog Agent, enabling it to process untagged StatsD metrics by mapping segments of the metric's namespace to tags, thus facilitating better organization and filtering of metrics. This feature is especially useful for tools like Apache Airflow, which emit metrics only in StatsD format, as it allows for the extraction of meaningful tags from existing infrastructure. Datadog customers can immediately start using DogStatsD Mapper by defining mapping rules for their StatsD metrics, enhancing the visibility and insight into various aspects of their environment.

Datadog has found that audit logs are extremely helpful for tracking user interactions with the API server, debugging issues, and getting clarity into their workloads in Kubernetes clusters. They provide deep insight into cluster operations by recording requests to the Kubernetes API server. These logs can be used to monitor API authentication issues, slow API requests, and anonymous requests to the API server. By persisting audit logs to longer term storage, it's possible to go back in time and answer questions such as, "Why was this pod evicted?" and "What lifecycle operations occur when we update a deployment?". Datadog also provides guidance on how to configure Kubernetes audit logs for optimal results.

The Kubernetes audit log is an essential tool for tracking user interactions with the API server, debugging issues, and gaining insight into workload behavior. It captures detailed metadata about each request, including HTTP method, URL path, user information, and authorization decisions. By leveraging audit logs, organizations can detect anomalies in API traffic, identify potential security threats, and troubleshoot cluster operations. The Kubernetes audit log policy specifies which stages of each request should generate logs, as well as the desired level of verbosity for each type of resource. Organizations can configure their audit policies to capture specific details, filter out noise, and optimize storage requirements. Datadog's log management features can help organizations cost-effectively monitor their audit logs and set up automated alerts to detect unusual activity. By analyzing and visualizing audit logs, organizations can gain a deeper understanding of their cluster operations and make data-driven decisions to improve security, performance, and reliability.

Synthetic browser tests are useful tools for ensuring that key user workflows function consistently across devices and locations. Monitoring email workflows is an important aspect of these tests, as email plays a crucial role in onboarding users and providing account information. By incorporating email verification into browser tests, developers can monitor the entire user experience from signup to confirmation emails. This allows for automatic verification of email content and testing links within those emails, ensuring a seamless user experience. Datadog's Synthetic Monitoring provides visibility into application and email workflows alongside infrastructure metrics, distributed traces, and logs, offering a comprehensive picture of end-to-end performance.

Synthetic browser tests enable monitoring of critical user workflows that generate email, including new signups, password resets, account changes, and purchase confirmations. These tests can automatically verify the content of emails sent by applications and test links in those emails to ensure a seamless user experience. Browser tests can be run on a schedule and monitored across multiple locations and devices, and they provide visibility into application performance alongside infrastructure metrics and logs. The tests also include built-in private inboxes for secure verification of email contents without the need for third-party services. By adding email workflows to browser tests, developers can ensure that their applications reliably generate emails with the expected information, proactively identifying potential issues before they impact users.

Datadog's Simple Network Management Protocol (SNMP) integration allows users to monitor and alert on the health and performance of their on-premise network infrastructure alongside the rest of their stack from one centralized platform. SNMP enables administrators to remotely modify settings and view information about network devices, such as routers, switches, or servers, across local and wide-area networks. Datadog's Agent collects SNMP data by polling object identifiers (OIDs), which act as addresses pointing to device data, and submits the responses as metrics for visualization, correlation, and alerting. Monitoring network devices alongside the rest of your infrastructure can help break down organization-wide silos that make it difficult to troubleshoot hardware-to-application layer issues. Datadog's SNMP integration also provides automatic device discovery, device-specific profiles, common metrics from any device type, and machine learning-powered forecasting for bandwidth saturation detection.

Datadog's SNMP integration enables administrators to monitor and alert on the health and performance of their on-premise network infrastructure alongside the rest of their stack from a centralized platform. Simple Network Management Protocol (SNMP) is used to collect data about network devices, which can be accessed through object identifiers (OID). The Datadog Agent collects SNMP data by polling OIDs and submitting responses as metrics, making it possible to visualize, correlate, and alert on these metrics across the Datadog platform. This integration helps break down organization-wide silos that make it difficult to troubleshoot hardware-to-application layer issues, allowing for quicker resolution of problems. The system automatically discovers all SNMP-enabled network devices in a provided subnet after configuring the integration, and provides device-specific profiles and common metrics from any device type independent of the manufacturer. By using Datadog's machine learning-powered forecasting feature, administrators can preemptively catch bandwidth saturation on their switches and prevent potential issues. Additionally, tags on SNMP metrics enable granular data comparison across devices, providing a comprehensive end-to-end overview of the network environment.

ClickHouse is an open-source database management system designed for large-scale data analysis jobs. It integrates with Datadog to provide full visibility into big data analytics jobs and optimize queries. The integration allows users to track read and write performance, compare query throughput with resource usage, and monitor ZooKeeper connections. Additionally, it enables shipping logs from all ClickHouse instances for troubleshooting and identifying errors in queries. With Datadog's integration, users can gain comprehensive visibility into their distributed analytics jobs alongside other technologies like Apache Kafka.

ClickHouse is an open source column-oriented database management system originally developed as a backend for Yandex's Metrica analytics platform. It achieves optimal performance with a maximum of 100 queries per second on a single instance and a minimum of 1,000 rows for each INSERT query. Datadog integrates with ClickHouse to provide full visibility into big data analytics jobs, allowing users to track read and write performance over time, compare query throughput with query resource usage, and monitor ZooKeeper connections. The integration also enables users to ship logs from all ClickHouse instances to Datadog, enriching the logs with metadata for easier searching and filtering, and plotting trends over time. With this integration, users can gain comprehensive visibility into their distributed analytics jobs alongside other technologies, including Apache Kafka, and sign up for a free trial if needed.

AWS Lambda decouples the need to provision and maintain a runtime environment from running code, allowing developers to focus on applications rather than infrastructure. However, serverless architectures introduce new challenges into monitoring and observability. Datadog addresses these issues with its enhanced AWS Lambda integration, which provides deeper insights into serverless functions by tracking key metrics like execution times, number of invocations, errors, cold starts, estimated costs, and memory usage across all Lambda functions in real-time and at higher granularity than standard CloudWatch metrics. This enables developers to visualize and alert on performance issues as they occur, monitor the frequency of cold starts, track function memory usage, and optimize cost efficiency by identifying expensive functions and refactoring them for improved efficiency and reduced costs.

AWS Lambda is a compute service that enables developers to build applications without managing infrastructure resources. It is event-driven, meaning it triggers in response to events from other services such as API calls or changes to a DynamoDB table. Monitoring the efficiency of your Lambda functions can help you optimize their performance and manage costs. Key metrics include function utilization and performance, invocations, concurrency, and provisioned concurrency usage. AWS Lambda provides these metrics through CloudWatch, which enables users to monitor and troubleshoot issues in real-time.

In this article, the author discusses using Amazon's native tools like CloudWatch and X-Ray to monitor AWS Lambda functions. They explain how these tools collect logs, metrics, and events from Lambda and other AWS services, providing dashboards and alarms for your data. The author also demonstrates how to use the AWS Command Line Interface (CLI) to pull metric data. Additionally, they provide insights on analyzing metrics and logs from Lambda functions using CloudWatch, creating alarms for Lambda metrics, viewing and analyzing traces with X-Ray, and retrieving information about functions with the AWS CLI. The author concludes by emphasizing the importance of having a monitoring service that provides visibility into AWS and all other application components in one place, such as Datadog's AWS Lambda integration.

AWS Lambda is a serverless compute service that enables developers to build event-driven applications without provisioning or managing infrastructure resources. It automatically scales instances based on the number of incoming requests, and its pricing model is based on the execution time, memory size, and number of requests for each function. To effectively monitor Lambda, it's essential to understand how your functions are used, the type of resources they need to run efficiently, and how other services interact with them. Monitoring key metrics such as duration, billed duration, memory size, max memory used, errors, invocations, concurrency, provisioned concurrency utilization, and spillover invocations can help you optimize your Lambda code, manage costs, and troubleshoot issues. Additionally, using distributed tracing and APM tools like AWS X-Ray and Datadog can provide valuable insights into performance bottlenecks in your serverless architecture.

Amazon provides built-in monitoring functionality through CloudWatch and X-Ray, which enable users to query metrics, logs, and traces for their AWS Lambda functions. CloudWatch collects logs, metrics, and events from Lambda and other AWS services, while X-Ray allows users to trace requests as they invoke Lambda functions, enabling the pinpointing of bottlenecks in application workflows. The AWS Command Line Interface (CLI) can be used to pull metric data, and users can configure CLI settings for their account. CloudWatch divides metrics by namespace and dimension, allowing users to filter their metrics, while logs can be analyzed using query syntax to gain deeper insights into function activity. Alarms can be created in CloudWatch to notify users of issues such as Lambda errors and overprovisioned functions, and X-Ray enables the tracing of requests to identify performance bottlenecks. The AWS CLI provides access to Lambda functions' information, including logs and metrics, while Amazon's native tooling allows for monitoring platforms like Datadog to provide visibility into all components of an application in one place.

Datadog has expanded its AWS Lambda integration to provide customers with enhanced visibility into their serverless functions, including real-time metrics and metadata collected at higher granularity than standard CloudWatch metrics. The new integration offers features such as alerting on errors in real time, tracking cold starts, estimating costs, and monitoring memory usage. With this enhanced integration, developers can gain deeper insights into the performance and resource utilization of their serverless functions, allowing them to optimize cost efficiency and troubleshoot issues more effectively.

To monitor all of your AWS Lambda functions with Datadog, you can start by enabling Datadog's AWS and Lambda integrations, collecting enhanced metrics and logs, and setting up alerts to detect trends and anomalies. You can also use Datadog's Lambda Library to collect custom business metrics, distributed traces, and enhanced metrics from your functions. With the Lambda extension, you can automatically surface actionable insights into your functions, including detailed metadata for your functions such as cold start and any custom tags you added to your function in the Lambda console. Additionally, you can visualize your AWS Lambda metrics with Datadog's out-of-the-box integration and enhanced metrics dashboards, giving you deep visibility into the performance of your serverless applications.

Since its launch in 2014, AWS Lambda has revolutionized application development through serverless computing by allowing developers to focus on customer-centric problem-solving rather than infrastructure management. However, monitoring serverless applications presents new challenges, necessitating insights into function-level performance. Datadog APM addresses this by providing distributed tracing and comprehensive visibility into AWS serverless services, including Lambda. Their lightweight Lambda extension enables detailed data collection and asynchronous transmission of metrics, logs, and traces to Datadog APM, even during function failures. The extension seamlessly integrates with AWS services while capturing critical performance metrics like cold starts and request payloads. Datadog's APM offers end-to-end visibility across serverless functions and infrastructure by correlating traces with metrics and logs, helping identify bottlenecks and errors impacting user experience. It supports various tracing methods and runtimes, enhancing application performance optimization and issue resolution. This comprehensive approach aids businesses in navigating the complexities of serverless architectures, ultimately contributing to improved user experiences and business outcomes.