January 2017 Summaries

Outlier algorithms MAD and DBSCAN are effective in identifying hosts behaving differently from others, but may flag outliers among tightly grouped metrics due to their scale-independent nature. ScaledMAD and ScaledDBSCAN are newer scaled outlier algorithms that consider the relative scales of divergence and median data, making them more suitable for monitoring situations where dispersion in metrics is more meaningful in the context of overall magnitude. These algorithms can help detect anomalies within closely clustered groups of metrics when the overall scale of the metrics is large compared to the median distance between hosts and series.

The text discusses the challenges of detecting outliers in time series data, particularly when dealing with metrics that have similar values. The MAD (Median Absolute Deviation) and DBSCAN algorithms are effective in most situations, but may not be suitable when the dispersion is more meaningful in terms of overall magnitude. To address this, new scaled outlier algorithms ScaledMAD and ScaledDBSCAN have been introduced, which consider relative scales of divergence and median values to determine outliers. These algorithms can help identify anomalies without being affected by the overall scale of the metrics, making them suitable for applications where absolute deviation is more meaningful. The text also mentions that these new algorithms are part of a broader effort to improve monitoring features and provide dynamic graphing and alerting capabilities.

The Jenkins platform is an open-source Java-based continuous integration server that helps organizations automate the build, test, and deployment of projects. It is widely adopted by companies such as GitHub, Etsy, LinkedIn, and Datadog. The platform supports hundreds of plugins, enabling it to handle a wide range of use cases. The Datadog plugin provides additional insights into Jenkins job history and trends, allowing users to set alerts for important build failures, identify trends in build durations, and correlate Jenkins events with performance metrics from other parts of the infrastructure. The integration also collects real-time system and security events, enabling users to track Jenkins queues, monitor system and security events, and visualize Jenkins metrics like build status and job duration.

The Datadog agent was experiencing long gaps in metrics on some systems due to the os.statvfs function call taking a very long time during disk checks. This issue occurred when using NFS (Network File Systems) with hard mounts, which can cause programs to hang in an unkillable state by design. To address this problem, the Datadog agent now runs the statvfs call on a separate thread and times out if it's hanging, allowing the main thread to continue. This solution slightly increases memory usage but is necessary for the agent to operate in various environments with different NFS mount settings.

The Datadog team discovered a bug in their agent that caused it to hang indefinitely on systems with certain types of network file systems, specifically NFS. The issue was traced to the `os.statvfs` function, which is a Linux system call that can cause the system to hang if used in an unkillable state. This happens when trying to stat a remote directory mounted with NFS, and it's a known issue due to the way NFS handles connections. To fix this, the Datadog team had to implement a workaround by running the `statvfs` call on a separate thread and using a timeout to prevent the main thread from hanging. This solution comes with trade-offs, such as slightly increased memory usage on systems with hard-mounted NFS disks, but it allows the agent to operate in a wide range of heterogeneous environments.

Cory Watson's experience of integrating Datadog into Stripe's observability stack highlights the importance of fostering a culture of observability, which involves more than just measuring outputs but also developing a shared understanding across teams to make data-driven decisions. His approach emphasizes understanding each team's needs, establishing the value of new tools, laying a solid foundation, and ensuring effective communication throughout the organization. By identifying power users as champions and being open to feedback, the transition to a new monitoring system becomes smoother, promoting a deeper understanding of and adjustment to existing processes. Watson's strategy not only improved the monitoring and decision-making processes at Stripe but also encouraged a broader organizational shift towards a culture that values informed decision-making and continuous improvement.

Datadog recently launched a new feature called Notebooks, which allows users to save graphs and other information from a particular point in time along with text. The feature was developed by Marie-Laure Bardonnet, an intern from Lyon, France, during her seven-month internship at the company. Despite concerns about being assigned only minor tasks, Marie-Laure was given the opportunity to work on the Notebooks project under the mentorship of team lead Ivan DiLernia. She learned React, Redux, and Redux Saga while working on this feature. After completing her internship, she continued to work part-time on the Notebooks feature and has now accepted a full-time position at Datadog.

Amazon Kinesis Firehose is a fully managed service that enables users to stream and capture data into various Amazon storage services such as Kinesis Analytics, S3, Redshift, and Elasticsearch Service. It can be considered a drop-in replacement for systems like Apache Kafka or RabbitMQ. Datadog's Firehose integration allows users to monitor and alert on Firehose performance metrics alongside metrics from storage backends and the rest of their infrastructure. The integration collects producer-level API metrics, backend-specific metrics, and supports automatic alerting for unexpected drops in throughput or changes in seasonal behavior.

Marie-Laure Bardonnet, an intern from Lyon, France, built a significant portion of the Notebooks feature in the Datadog web application, which allows users to save graphs along with text and other information. The feature was developed over seven months, starting with small bug fixes, and gradually becoming more complex tasks. Marie-Laure's experience on this project not only helped her develop new skills but also provided her with mentorship from Ivan DiLernia, the team lead, who encouraged her to take ownership of the project. The success of the feature has led to Marie-Laure returning to Datadog after completing her studies and accepting a full-time position at the company.

Amazon Data Firehose is a fully managed service that enables users to stream and capture data into various Amazon storage services, such as S3, Redshift, and Elasticsearch Service. It can be used as a drop-in replacement for systems like Apache Kafka or RabbitMQ, offering auto-scaling capabilities and simplified administration and maintenance. Datadog's integration with Firehose provides real-time metrics and performance monitoring, enabling users to monitor and alert on Firehose performance metrics alongside other infrastructure metrics. The integration also offers automatic alerting for unexpected drops in throughput and tagging of metrics with stream names and regions, making it easy to identify and address pipeline issues.

Datadog's new Notebooks feature allows users to combine real-time or historical graphs with Markdown cells, enabling the creation of detailed postmortems, dynamic runbooks, and open-ended exploration for troubleshooting. Notebooks support adjustable timeframes for graphs, making it easier to identify warning signs and create alerts for future incidents. They also provide a platform for collaborative documentation and updating runbooks, improving response times during issues. Additionally, the feature allows users to freely explore their infrastructure or application metrics without affecting existing dashboards.

Datadog's new Notebooks feature allows users to create detailed postmortems that can be shared with their team, using real-time or historical graphs combined with Markdown cells. This feature enables the creation of dynamic runbooks and provides an open-ended exploration sandbox for troubleshooting metrics. With notebooks, users can easily organize and contextualize incidents, identify warning signs, and document them in a way that's accessible to the entire team. The feature also allows users to save their work and share it with others, making it easy to collaborate on internal documentation and improve engineering teams' productivity.

Azure Event Hubs is a Microsoft Azure cloud service that enables real-time data pipelines by storing events in sequence for a configurable retention time. It has similarities to Kafka and AWS Kinesis. Datadog now allows users to monitor their Event Hubs alongside other Azure services, providing full observability of infrastructure. Monitoring consumed capacity helps prevent data loss due to throttling exceptions, while tracking outbound traffic can identify resource limitations affecting consumer applications. With Datadog's integration with Azure and support for custom metrics in multiple languages, users gain access to advanced monitoring functionality for their Event Hubs and other Azure services.

Azure Event Hubs is a real-time data pipeline service in the Microsoft Azure cloud that stores events in sequence for a configurable retention time, allowing applications to stream or replay data as needed. The service has similarities with Kafka and AWS Kinesis services, and its capacity is constrained by throughput units provisioned for an Event Hub namespace. Monitoring all Event Hubs is crucial to confirm performance and avoid saturating provisioned capacity, which can result in throttling, dropped events, and exceptions. By integrating Datadog with Azure Event Hubs, users gain access to a full suite of metrics, including standard Azure metrics, custom metrics, and advanced monitoring functionality such as graphing, dashboards, and algorithmic alerting. This integration enables users to bring observability to their Azure infrastructure and monitor modern applications more effectively.

Amazon Elasticsearch Service is a managed service that helps users configure, deploy, and maintain their Elasticsearch clusters in AWS cloud environments. Datadog's integration with Amazon Elasticsearch Service allows automatic collection of key metrics, visualization on an out-of-the-box dashboard, and alerts for resource shortages or performance issues. The integration also enables connection to the hosted Elasticsearch cluster to pull additional metrics beyond those provided by AWS CloudWatch. Key features of Amazon Elasticsearch Service include automated data backups, availability through zone awareness, and monitoring via CloudWatch. Datadog's integration allows users to collect, visualize, and alert on various metrics such as cluster status, free storage space, JVM heap usage, CPU utilization, latency, and IOPS. Users can also set up alerts for potential issues like high CPU utilization or low free storage space. Additionally, the Datadog Agent provides access to more Elasticsearch metrics than those exposed by CloudWatch.

Datadog has integrated with Amazon Elasticsearch Service (Amazon ES), allowing users to automatically collect key metrics, visualize them in an out-of-the-box dashboard, and get alerts for resource shortages or performance issues. The integration simplifies the process of managing Elasticsearch clusters by automating daily backups and enabling zone awareness, which helps distribute instances across multiple availability zones. Datadog's integration also enables users to correlate performance metrics with metrics from other parts of their infrastructure. Additionally, users can access more advanced Elasticsearch metrics using the Datadog Agent, which requires installing the Agent on another host and pointing it at the Amazon ES domain endpoint. The integration is designed to provide increased visibility into Elasticsearch performance and resource utilization, helping users respond to potential issues before they become pressing concerns.