September 2019 Summaries

Datadog has introduced a new integration with DingTalk, Alibaba's communication platform, to enhance real-time collaboration and issue resolution within teams using Alibaba Cloud environments. This integration allows teams to receive automated alerts for critical events, such as spikes in errors or memory deficits, directly in their DingTalk group chats, facilitating swift troubleshooting. Users can configure alerts to notify specific DingTalk groups and share annotated graphs and insights, which are useful for discussing trends or anomalies. The integration with DingTalk, alongside compatibility with other collaboration tools like PagerDuty and Slack, and various Alibaba Cloud services, aims to improve visibility and collaborative problem-solving in complex cloud infrastructures.

Alibaba Cloud has integrated with Datadog, providing users with comprehensive cloud computing services. The new integration enables businesses to monitor the health and performance of Alibaba Cloud services such as load balancers, managed databases, and more in Datadog. Key metrics and metadata are automatically collected from various Alibaba Cloud services including Elastic Compute Service (ECS), Server Load Balancer (SLB), ApsaraDB RDS, ApsaraDB for Redis, Content Delivery Network (CDN), Container Service, and Express Connect. Datadog uses Alibaba Cloud Monitor APIs to collect metrics and metadata from these services, allowing users to derive more useful insights by aggregating them across any scope that matters to them. By installing the Datadog Agent on Alibaba Cloud VMs, users can get even richer context around the metrics and metadata collected by the integration. The integration is now generally available for businesses worldwide.

Datadog's Log Rehydration feature allows organizations to efficiently archive and retrieve old logs for troubleshooting and analysis, overcoming challenges such as slow access, high costs, and lack of granularity in traditional log management solutions. With Log Rehydration, users can quickly rehydrate logs from cold storage into their Datadog account, analyze them using the Log Explorer, and perform root cause analysis without additional tooling or significant organizational overhead. The feature also enables organizations to organize old logs with archives, manage storage costs effectively, and gain greater visibility into their applications and infrastructure.

DDSketch is a new sketch algorithm designed to accurately compute percentiles on large-scale monitoring data. It was developed by Datadog, which handles vast amounts of distributed data daily. Unlike existing state-of-the-art quantile sketch algorithms, DDSketch provides relative-error guarantees that better reflect users' needs when looking at latency plots. This makes it more memory-efficient and accurate than other sketches with rank-error guarantees. DDSketch has a small memory footprint and is highly performant, making it suitable for use in monitoring systems. It is currently being used at scale at Datadog and has open source implementations available in Java, Go, and Python.

** We recently published a paper, DDSketch: A Fast and Fully-Mergeable Quantile Sketch with Relative-Error Guarantees, in PVLDB on August 28th. This paper introduces DDSketch, a new sketch algorithm designed to efficiently compute percentiles from large-scale monitoring data while maintaining high accuracy and low memory usage. Unlike existing algorithms, DDSketch provides a relative-error guarantee, ensuring that the computed percentile values are within a specified fraction of the actual value. The algorithm achieves this by using representative values and bucketization, allowing for efficient merging and minimizing memory footprint. Our benchmarks demonstrate that DDSketch outperforms other sketch algorithms in terms of performance, memory usage, and accuracy, making it an attractive solution for distributed data processing applications.

Logs are essential for troubleshooting Node.js applications, offering insights into issues and their root causes. While Node.js provides basic logging capabilities through `console.log`, the Winston library provides greater flexibility for customizing log metadata, formats, and exception handling. This guide details how to use Winston to enrich and centralize logs, allowing for deeper application visibility and efficient issue resolution. It covers configuring loggers, incorporating logging levels, adding metadata, and customizing formats. Additionally, the guide highlights the importance of centralizing logs for comprehensive monitoring and demonstrates how to visualize trends and correlate logs with distributed request traces using services like Datadog. By leveraging these practices, teams can efficiently manage logs, ensuring detailed insights into application health and performance.

Datadog is committed to open-source software and transparency, emphasizing its dedication to supporting open standards and integrating with projects like OpenTelemetry, which aims to provide a unified set of APIs for system instrumentation across various platforms. With the merger of OpenTracing and OpenCensus into OpenTelemetry, Datadog is contributing its tracing libraries to support this initiative, offering auto-instrumentation to collect comprehensive telemetry data without manual code changes. This collaboration intends to enhance observability by allowing companies to gather data from diverse systems and migrate it easily between monitoring providers, thereby reducing vendor lock-in. Datadog's existing libraries, used by many companies, have already contributed valuable feedback, which will further enrich the OpenTelemetry project. The partnership highlights Datadog's belief in the benefits of open-source instrumentation for improving system visibility and performance across the industry.

Since its release in 2015, the Serverless Framework has become a popular tool for managing and deploying serverless applications by allowing users to define infrastructure as code for easy sharing and version control. To enhance observability, Datadog developed a Serverless plugin that provides real-time visibility into AWS Lambda functions without requiring code changes, addressing challenges like monitoring performance and detecting errors. The plugin supports Node.js and Python functions and integrates with AWS X-Ray for end-to-end tracing, which is crucial as serverless applications often operate as part of larger systems. Users can configure custom metrics and explore key performance data via Datadog's platform, enabling efficient monitoring of serverless environments.

Oracle's Container Engine for Kubernetes (OKE) is a service that enables organizations to deploy, manage, and scale Kubernetes clusters in the cloud. With OKE, users can build dynamic containerized applications by incorporating Kubernetes with services running on their Oracle Cloud Infrastructure. The Datadog Agent provides comprehensive visibility into OKE container infrastructure, allowing users to visualize their cluster, monitor live processes, and track key metrics from all pods and containers in one place. OKE launches clusters with control plane and worker nodes in a node pool along with network resources needed for that cluster. Users can easily modify their cluster and download kubeconfig files for additional management tasks. Datadog's integration with OKE enables users to monitor Kubernetes, track load on clusters, pods, and individual nodes, collect metrics from services running in the cluster, and more. The integration provides built-in dashboards for high-level views of OKE clusters, Live Container view for real-time insights into containers' resource consumption, logs, and health, and Autodiscovery feature to automatically track containerized services running in dynamic environments.

Multi-line logs such as stack traces can provide valuable information for debugging and troubleshooting application problems. However, without proper configuration, log management services may treat each line of a multi-line log as an individual event, making it difficult to interpret the activity of your applications. To handle multi-line logs effectively, you can either log to JSON or use a log shipper that looks for specific patterns to indicate the beginning of a new log entry. Popular log shippers include Logstash, Rsyslog, Fluentd, Fluent Bit, Syslog-ng, NXLog, and the Datadog Agent. By aggregating multi-line logs as single events, you can more easily parse, search, and use them to identify problems in your environment.

The text discusses the challenges of collecting and processing multi-line logs for debugging and troubleshooting purposes. It highlights the issue that log management services treat each line separately, making it difficult to interpret application activity. The article presents strategies for handling multi-line logs, including logging to a file, using JSON format, and configuring popular log shippers such as Logstash, Rsyslog, Fluentd, Fluent Bit, Syslog-ng, NXLog, and Datadog. These solutions enable the aggregation of multi-line logs into single events, making it easier to identify problems and troubleshoot issues. The article concludes by emphasizing the importance of simplifying multi-line logs to extract their full value for troubleshooting and monitoring purposes.

The text outlines the integration of Okta, a cloud-based identity management service, with Datadog, which allows organizations to monitor and analyze Okta logs for enhanced security and compliance. By using Okta, businesses can streamline IT workflows by incorporating tools like single sign-on and multi-factor authentication, along with user management services. The integration with Datadog enables users to collect and oversee Okta logs, providing visibility into access and lifecycle events, which helps in threat detection, user activity tracking, and debugging authentication and authorization issues. The document details how Okta logs deliver insights into administrative and user activities, allowing for the detection of suspicious behavior and monitoring of authentication attempts. Furthermore, archiving Okta logs facilitates the creation of an audit trail crucial for compliance and security audits. Overall, the integration offers valuable insights through log analytics, aiding organizations in maintaining system health and security.