September 2015 Summaries

Datadog introduces outlier detection, a feature that automatically identifies any host or group of hosts behaving abnormally compared to their peers. This feature helps users monitor metrics without having to define ahead of time what constitutes "normal" versus "abnormal" values. Outlier detection can be used to alert when one machine starts reporting errors at an aberrant rate, identify the cause of latency spikes, and spot problem hosts on dashboards. The feature offers two algorithms for identifying outliers: DBSCAN (density-based spatial clustering of applications with noise) or MAD (median absolute deviation).

Outlier detection is a crucial aspect of maintaining healthy hosts within an infrastructure, minimizing service degradation and disruption. Datadog offers two algorithms for this purpose: DBSCAN (density-based spatial clustering of applications with noise) and MAD (median absolute deviation). DBSCAN works by greedily agglomerating points that are close to each other, while MAD is a robust measure of variability. Both algorithms have their strengths in detecting outliers based on the specific behavior of hosts. When setting up alerts, it's essential to consider the size of the time window analyzed and whether banding behavior requires separate monitoring for different groups.

Datadog is introducing outlier detection, a feature that automatically identifies hosts behaving abnormally compared to their peers. This allows users to set alerts without choosing fixed thresholds for "anomalous" metrics and run statistical analysis in real-time on all hosts to determine baseline values. Outlier detection can be used to identify problem hosts, automate alerts, and provide a comprehensive overview of monitored infrastructure. The feature is available with two algorithms: DBSCAN and MAD, and can be easily integrated into dashboards and monitors for added value in monitoring and alerting toolkits.

Homin Lee discusses the importance of identifying unhealthy hosts in an infrastructure and introduces outlier detection, a technique that can help minimize service degradation and disruption. He explains that traditional threshold-based alerts can be difficult to define and may trigger false alarms, especially for metrics with spikes or fluctuating baselines. Instead, outlier detection uses algorithms such as DBSCAN and MAD to compare each host against others in the group, alerting when a host deviates from the pack while avoiding false positives. The two algorithms differ in their approach: DBSCAN uses clustering to identify outliers, while MAD is a robust measure of variability that focuses on deviations from the median. Homin Lee provides guidance on setting parameters and tuning tolerance for each algorithm, as well as considerations for choosing between DBSCAN and MAD depending on the specific use case. Ultimately, outlier detection can be used in conjunction with other monitoring features to provide dynamic alerts and minimize service disruption.

Docker is a young container technology that provides easily-configured, lightweight VMs that start up fast, ideal for microservice architectures and environments that scale rapidly or release often. With the recent release of AWS's EC2 Container Service (ECS), Docker has taken another step towards maturity by providing a service that automatically manages your Docker containers for you. ECS balances load among containers, recovers unhealthy containers, provides scaling automation, and more. However, monitoring containers can be challenging due to their rapid presence and potential downtime. Datadog is a purpose-built tool designed to monitor highly dynamic infrastructure, including containers, with a tailored ECS integration that automatically tracks new container metrics and handles offline containers gracefully. With Datadog, users can track their containers, view individual metrics, group them dynamically, correlate with other infrastructure metrics, set service-level alerts, and gain immediate visibility into any problems affecting their Docker infrastructure.

The AWS re:Invent 2015 conference features an extensive schedule of over 380 events, with Datadog highlighting several key sessions and activities. Datadog's participation includes live demos of new outlier detection features and integrations with AWS services, as well as multiple sessions on monitoring strategies and infrastructure evolution. Notable sessions include insights into AdRoll's transition to a data-driven organization, Amazon's DevOps tools and processes, and best practices for Elastic Load Balancing. Additional sessions cover the use of AWS services like EC2, ECS, and Kinesis for scalable, real-time data processing and application management. Attendees are encouraged to explore vendor booths and join social events, such as the AWS Pub Crawl and hosted gatherings, to network and learn about the latest advancements.

Medium, a popular blogging platform, uses Amazon DynamoDB as one of its primary data stores to handle rapid scaling. Monitoring and managing the performance of DynamoDB is crucial for maintaining an optimal user experience. The main challenge faced by Medium is throttling, which can cause high latency and user-facing errors. To tackle this issue, Medium uses Datadog to track whole-table capacity usage and compares it with provisioned capacity. They also calculate the number of partitions per table and estimate the throughput limit for each partition. By tracking hot keys and throttling events, Medium can take action to increase capacity when needed. Additionally, they use Redis as a cache in front of DynamoDB to reduce throttling frequency and lower costs. Proper alerting is essential for resolving issues quickly, and Medium uses Datadog's alerting features to notify the right people with the appropriate channels and messages. Monitoring also helps Medium identify when they can scale down their provisioned throughput, optimizing infrastructure expenses.

This article discusses the process of collecting native DynamoDB metrics using Amazon CloudWatch. It outlines three methods to access these metrics: through the AWS Management Console, via the command-line interface (CLI), and by integrating with a monitoring tool that uses the CloudWatch API. The first method involves navigating the console, selecting desired metrics, and setting up alerts for when metric thresholds are crossed. The second method requires installing the AWS CLI and running queries to retrieve specific metrics. The third method involves using dedicated monitoring tools that can integrate with CloudWatch via its API, allowing for more advanced functionality such as dynamic slicing, aggregation, and filtering of metrics on any attribute. In Part 3 of this series, strategies used by Medium's engineering team to monitor DynamoDB are discussed.

DynamoDB is a hosted NoSQL database service offered by AWS that provides fast and scalable performance for applications with large amounts of data. It supports both document and key-value store models, has properties of both a database and a distributed hash table, and each table is synchronously replicated across three availability zones to ensure high availability and data durability. To correctly provision DynamoDB and keep applications running smoothly, it's essential to understand and track key performance metrics in various areas, including read and write queries, consumed capacity units, and errors. Monitoring these metrics can help identify issues before they impact application performance, such as latency, throttling, and errors. Additionally, monitoring Global Secondary Index creation and tuning provisioned capacities are crucial for maintaining good database health and performance. By tracking the right metrics and taking proactive measures, developers can ensure their applications run efficiently and effectively on DynamoDB.

Medium, a popular online publication, uses DynamoDB as its primary data store to scale rapidly and handle high traffic. To monitor performance and prevent throttling, Medium employs several strategies, including tracking whole-table capacity, partitioning, and hot keys. They use Datadog to collect metrics on reads and writes per second, compare actual usage to provisioned capacity, and track throttled requests. Medium also creates custom metrics, such as a "Last Backup Age" metric, to ensure regular backups of DynamoDB tables and trigger alerts when necessary. By properly monitoring performance and using tools like Datadog, Medium is able to optimize its infrastructure expenses, reduce throttling frequency, and provide an excellent user experience.

Jean-Mathieu Saponaro discusses the second part of a three-part series on monitoring DynamoDB, focusing on collecting native DynamoDB metrics using CloudWatch. He explains that CloudWatch metrics can be accessed through the AWS Management Console, command-line interface, and third-party monitoring tools integrating with the CloudWatch API. Using the management console allows for simple automated alerts and visualizing recent changes in individual metrics, while also enabling graphing of selected metrics and creating custom alerts. The command line interface offers more control over queries, including filtering options, and is useful for spot checks and ad hoc investigations. Integrating CloudWatch with third-party monitoring tools provides extended functionality, such as correlating metrics across different infrastructure components or dynamically slicing and filtering data.

Redis is a popular in-memory key/value data store known for its performance and simple onboarding, offering uses across industries and use cases such as databases, message queues, and memory caches. It's available as a free, open-source product with commercial support options, including managed Redis-as-a-service. Many high-traffic websites and applications like Twitter, GitHub, Docker, Pinterest, Datadog, and Stack Overflow utilize Redis. Monitoring Redis is crucial to catch problems in resource issues with the database itself and supporting infrastructure. Key metrics include performance metrics such as latency, instantaneous operations per second, and hit rate, as well as memory metrics like used memory, mem fragmentation ratio, and evicted keys. Persistence metrics track data writes and saves to disk, while error metrics monitor common errors like rejected connections and keyspace misses. The article concludes by highlighting the importance of monitoring Redis metrics to ensure optimal performance and health of the database infrastructure.

Redis offers a comprehensive suite of monitoring tools that allow users to assess its performance through command line interface commands such as `info`, which provides a snapshot of current metrics, including server statistics, memory usage, and client connections. Users can delve deeper into specific sections like persistence and replication to analyze particular aspects of performance. For latency issues, Redis provides tools such as the slowlog, latency monitor, and latency doctor to diagnose and address delays, offering detailed insights into execution times and potential solutions. Memory optimization is also a focus, with commands like `memory stats` and `memory usage` providing granular details on consumption. While Redis’s built-in tools are sufficient for real-time monitoring and troubleshooting, integrating with dedicated services like Datadog can enhance long-term performance analysis and alerting capabilities, ensuring a more streamlined and effective monitoring process for Redis instances.