August 2017 Summaries

This exercise was conducted by Datadog to test the resilience of their Elasticsearch cluster. The team ran a game day, intentionally triggering failures in their services to observe how they responded in a controlled environment. They tested stopping individual nodes, including the leader node, and observed that while indexing and search requests returned 503s for a brief period during the leader transition, the applications recovered with minimal impact due to their buffering strategy. The team also discovered an issue with "dangling indices" that caused the cluster to go RED after deleting indices without acknowledging it across all nodes. Regular health checks for client nodes were found to be key in preventing issues. The exercise provided valuable insights into the cluster's behavior and highlighted the importance of monitoring and maintenance.

The Datadog Solutions Team uses Hashicorp's Vagrant to reproduce customer issues by creating virtual machines that mimic the unique environments of their clients. However, setting up these full reproduction environments can be time-consuming and complex due to the need for specific operating systems, kernels, and container technologies. To address this issue, the team leverages Vagrant's provisioning feature, which allows them to automate installation and configuration steps using configuration management tools or bash scripts. They maintain a shared GitHub repository containing virtual machines used for reproducing customer issues, with each VM having its own directory containing a Vagrantfile and setup script. This approach enables the team to share their reproduction environments and avoid redundant work when troubleshooting similar problems. Additionally, they have extended this solution by using Hashicorp's Terraform tool to manage remote instances in cloud environments like AWS. By maintaining a consistent directory structure and reusing Vagrant's setup scripts and data files, the team can quickly provision reproduction environments on EC2 instances that can be left running for longer periods and shared with other team members. This approach allows them to collaborate more effectively on reproducing customer issues and providing demonstrations of specific integrations. Future improvements include better modularization of setup scripts and exploring Hashicorp's Consul backend system for managing Terraform states.

The Datadog Solutions Team uses Vagrant to manage local virtual machine environments for reproducing customer issues, but they found setting up full reproduction environments took time and learning was a challenge. To expedite the setup process and make their team more efficient, they utilized Vagrant's provisioning feature to run configuration management tools like Chef, Puppet, or Ansible, and created a shared GitHub repository for virtual machines with pre-built Vagrantfiles and setup scripts. They later integrated Terraform into their workflow, using it to provision reproduction environments on small EC2 instances and share them with teammates, resulting in improved scalability and collaboration. The team is now exploring ways to further modularize their setup scripts and introduce additional features like Consul for managing terraform states.

Datadog has developed a robust serverless security monitoring and alerting pipeline for its operations in AWS cloud. The company uses CloudTrail to log all AWS API calls across multiple accounts, then narrows the focus by categorizing specific API calls into severity levels of log, notify, and alert. A centralized security orchestration framework integrates with Slack, Duo, and PagerDuty for notifications, authentication, and alerts. The pipeline architecture includes a Cloudwatch Event Rule that triggers an SNS Topic, which in turn sends the API call data to an SQS queue within the security AWS account. A lambda function processes the data from the SQS queue and interacts with Komand, a security orchestration and automation platform, for further processing and decision-making. The pipeline is monitored using Datadog, providing meaningful data and actionable intelligence to the security team.

Datadog has implemented a robust, largely serverless security monitoring and alerting pipeline to monitor its extensive AWS operations. The pipeline uses CloudTrail to log all AWS API calls on every account, but focuses on specific relevant calls and divides them into three severities: log, notify, and alert. It integrates with Slack, Duo, and PagerDuty to notify, authenticate, and alert potential security-relevant API calls. The pipeline sends data to a dedicated security-oriented AWS account while data collection is deployed to every Datadog AWS account via Terraform. A Cloudwatch Event Rule triggers an SNS Topic, which sends the call cross-account to an SQS queue within the security account. A lambda function processes the API call data and sends it to a security orchestration layer for further processing. Komand is used to create a workflow consisting of multiple individual plugins, decision points, and branches to parse the API call and its parameters, apply logic, and interact with multiple systems and APIs. The pipeline provides meaningful data and actionable intelligence to the security team, allowing efficient protection of the company and customers.

This article discusses the monitoring of Amazon ELB metrics using CloudWatch. It explains three methods to collect these metrics - AWS Management Console, AWS Command Line Interface (CLI), and a monitoring tool integrated with CloudWatch API. The article also highlights the usefulness of ELB access logs in investigating specific request issues. Additionally, it mentions that for more precision and granularity on backend instances' performance, one should consider monitoring their resources directly. Finally, it concludes by stating that in the next part of this series, readers will learn how to monitor ELB metrics using Datadog integration along with native metrics from backend instances for a complete view.

This article discusses Amazon Elastic Load Balancing (ELB), a service that distributes incoming web traffic across Amazon EC2 instances in different availability zones. Monitoring ELB is crucial for maintaining smooth user experience and ensuring fault tolerance. Key performance metrics to monitor include RequestCount, SurgeQueueLength, SpilloverCount, HTTPCode_ELB_4XX, and HTTPCode_ELB_5XX. Additionally, backend-related metrics such as HealthyHostCount, UnHealthyHostCount, Latency, and BackendConnectionErrors should be monitored to gain a comprehensive understanding of ELB's performance and health. Monitoring these metrics helps identify potential issues and ensures optimal performance for web applications using ELB.

Peloton, a technology company revolutionizing the fitness industry with its indoor-cycling bike and live/on-demand classes, has successfully scaled its business by using full-stack observability. With nearly half a million users, maintaining low latency for real-time features like rider leaderboards is crucial. By implementing Datadog APM, Peloton engineers were able to trace user requests in their application and identify inefficiencies, leading to significant improvements in response times for various endpoints within 30 days of implementation. This observability has not only improved the user experience but also helped the company scale its business rapidly.

Peloton Cycle, a technology company based in New York City, is revolutionizing the fitness industry with its indoor-cycling bike equipped with a 22-inch HD touchscreen. To deliver high-quality video feeds and minimize latency for real-time features like leaderboards, Peloton uses Datadog to monitor its AWS-based cloud infrastructure and collect custom user-experience metrics. By deploying full-stack observability, including application performance monitoring (APM), Peloton engineers can identify inefficiencies, trace code in the wild, and improve the user experience while scaling rapidly with speed. Within 30 days of implementing APM, the team reduced response times by a factor of three or more, significantly improving user engagement and overall application performance.

The Triggered Monitors page in Datadog has been improved with faster load times and powerful filters, allowing users to resolve incidents more quickly. The updated page provides a summary of alerts at the top of the sidebar, showing a breakdown by status and counts of recently triggered monitors. New faceted search options have also been added, including filtering groups by how recently they were triggered, group name, and monitor ID. The new Triggered Monitors page is built to scale for teams of all sizes and uses the same query syntax as the Manage Monitors page. Existing Datadog customers can access the updated page, while new users can sign up for a free trial to experience advanced alerting and collaboration features.

The Triggered Monitors page in Datadog has been improved with faster load times and powerful filters, allowing users to quickly identify and resolve incidents. The new page includes a faceted search option that enables teams to filter groups by how recently they triggered, group name, or monitor ID. This update is designed to scale for teams of all sizes, providing significantly faster load times and enhanced collaboration capabilities.