July 2017 Summaries

This article discusses how to monitor Amazon Aurora using CloudWatch and collect both RDS metrics and database engine metrics. It provides detailed instructions for accessing these metrics through the AWS Management Console, command line interface, and monitoring tools with a CloudWatch integration. Additionally, it covers querying server status variables, performance schema, sys schema, MySQL Workbench GUI, and using MySQL-compatible monitoring tools to collect database engine metrics. The article concludes by stating that in the next part of this series, it will demonstrate how to set up comprehensive Aurora monitoring with Datadog.

Amazon Simple Storage Service (S3) is a highly scalable object store that allows users to upload and retrieve files through various interfaces. It organizes objects into directories and subdirectories within "buckets," which can store an unlimited number of objects up to 5 terabytes in size. S3 is suitable for web asset storage, static site hosting, backup and recovery, archiving, and general-purpose file storage. Datadog's AWS S3 integration collects and visualizes various metrics for monitoring service performance and usage. Throughput, error, and performance metrics are available, as well as resource metrics that provide data on bytes stored, transferred, and total objects. To monitor S3 metrics in Datadog, first install the main AWS integration with read-only IAM credentials, then enable S3 metric collection by checking the S3 box in the service sidebar. By integrating with more than 650 technologies, including CloudFront and other AWS services, users can track S3 metrics alongside data from these related services for complete visibility into their website resources.

Adam Michael Wood is discussing Amazon Simple Storage Service (S3), a highly scalable object store that allows files to be uploaded and retrieved through various interfaces. S3 is ideal for storing web assets, static site hosting, backup and recovery, archiving, and general-purpose file storage due to its unlimited capacity and up to 5 terabytes per object size. Datadog provides an integration with S3, collecting and visualizing metrics such as throughput, error, performance, and resource usage, allowing for in-depth monitoring and actionable insights. To use the integration, users need to install the main AWS integration and enable S3 metric collection, which can be correlated with other AWS services like CloudFront for full visibility into web resources.

The Datadog Summit will take place in Austin, Texas on September 28th. This one-day event offers attendees the opportunity to learn from Datadog staff and community members who are utilizing monitoring for building faster, more reliable systems. Sessions will cover observability best practices, new feature leveraging, fostering a culture of observability within organizations, and hands-on training led by Datadog engineers. The summit will be held at the Omni Hotel Barton Creek and includes food, cocktails, and networking opportunities. Registration is free but limited, so sign up now to secure your spot.

The Datadog Summit is taking place in Austin, Texas on September 28, offering a one-day event for attendees to meet and learn from Datadog staff and community members. The summit will feature sessions focused on observability best practices, leveraging new features, and fostering a culture of observability at organizations. Hands-on training will also be provided by Datadog engineers, covering topics such as writing custom integrations and learning about Autodiscovery with container orchestrators like Kubernetes. The event is free but space is limited, so registration is recommended to secure a spot.

This post discusses recent improvements in alerting and algorithmic monitoring by Datadog. The new features aim to create smarter, more effective alerts. These include anomaly detection for metrics with natural fluctuations or changing baselines over time, APM service monitors that tie alerts directly to the health of specific services, composite monitors that trigger based on multiple indicators, and a zippy faceted monitor search for easier navigation through infrastructure issues.

This article highlights recent feature enhancements by Datadog, focusing on visualization and collaboration features. The new Notebooks feature allows users to explore, investigate, and document findings in a collaborative format accessible to the whole team. Additionally, two new types of visualizations have been developed for application performance monitoring: flame graphs and service-level dashboards. Furthermore, Datadog now supports read-only users, allowing account admins to manage access levels effectively. These enhancements aim to improve data-driven collaboration across organizations and provide a comprehensive monitoring solution.

Datadog has recently enhanced its platform with new visualization, collaboration, and management features. The company's Notebooks feature allows teams to create descriptive postmortems and runbooks in a visual format that's accessible to the whole team. Additionally, Datadog has introduced flame graphs and service-level dashboards to help users visualize application performance. These features are designed to improve data collection, alerting, and collaboration capabilities. Furthermore, Datadog has added read-only users, which enable account admins to share access to key metrics with specific individuals without granting them edit privileges. Overall, these new features aim to provide a more comprehensive monitoring and observability experience for users.

Datadog has recently introduced new features to enhance its alerting and algorithmic monitoring capabilities, aiming to provide smarter and more effective alerts that are actionable, clear, and customizable. These enhancements include anomaly detection for metrics with natural fluctuations or changing baselines, APM service monitors to track targeted performance indicators from specific services, composite monitors to capture complexity by combining multiple indicators, and a new faceted monitor search feature on the Manage Monitors page. The new features aim to reduce noise while ensuring timely notification of pressing problems, making it easier for users to quickly identify key issues with performance and availability.

Datadog has introduced a series of feature enhancements aimed at improving observability by expanding its integrations and data collection capabilities. This first post in a series highlights the addition of application performance monitoring (APM), which is now included with the Datadog Agent, allowing for easy deployment and open-source customization. APM supports multiple programming languages and web frameworks, enabling users to trace application performance and issues across service boundaries. Datadog has also expanded its integrations with major cloud providers like AWS, Google Cloud Platform, and Microsoft Azure, alongside other tools and services, to enhance visibility and monitoring. The Autodiscovery feature simplifies tracking and monitoring of containerized services, crucial in dynamic environments where containers frequently change. These advancements aim to help users efficiently collect and analyze data to quickly identify and resolve performance issues, setting the stage for further enhancements in alerting and algorithmic monitoring discussed in subsequent posts.

Application Performance Monitoring (APM) by Datadog helps improve application performance by tracking service interactions and providing detailed flame graphs for individual request traces. The new APM monitors notify users of changes in service-level indicators like latency and error rates, allowing them to identify issues, prioritize fixes, and optimize overall application performance. These monitors can be set up directly from the service dashboard or on the "Manage Monitors" page. When an APM monitor is triggered, users receive a notification with contextual information and links to investigate further. Datadog's integration with various technologies enables seamless investigation of issues across different components of the application stack.

Datadog Application Performance Monitoring (APM) is a tool that helps developers understand and improve application performance by tracking how services interact to serve real requests. It automatically gathers high-level performance metrics and decomposes individual request traces in detailed flame graphs to isolate issues, prioritize fixes, and optimize overall application performance. Datadog APM monitors are designed to notify users of changes in service-level indicators such as latency and error rate for each service, providing a simple way to automate monitoring of service-level performance. The monitors can be created directly from the service dashboard via the monitor status bar, and they include suggested alerts that serve as a guide to the metrics to consider alerting on. When an APM monitor is triggered, team members receive a notification with contextual information, including links back to a service overview and end-to-end traces of recent requests. The tool provides better alerting and allows developers to immediately begin investigating issues, which can lead to improved service performance.

Google Cloud SQL is a fully managed service that allows users to easily set up, maintain, and scale MySQL databases with PostgreSQL support in beta. Hosted on Google Cloud Platform, it automatically handles software updates and enables automated backups and failover for high availability. Datadog's Google Cloud SQL integration helps monitor key metrics such as disk utilization, CPU usage, connections, and questions (for MySQL). The integration also allows users to correlate Cloud SQL performance with other Google Cloud services and set up automatic alerts for unexpected behavior.

Google Cloud SQL is a fully managed service that enables users to easily set up, maintain, and scale MySQL databases with PostgreSQL support in beta. It automatically handles software updates, backups, and failover for high availability. The Google Cloud SQL integration with Datadog provides visibility into key metrics such as disk utilization, CPU usage, connections, questions, and more, allowing users to monitor their database's health and performance. Additionally, the integration enables users to correlate Cloud SQL performance with other Google Cloud services, set up automatic alerts for unexpected behavior, and create custom dashboards and alerts using a robust alerting system. With Datadog's GCP integration, users can start monitoring Cloud SQL in just a few minutes or take advantage of a free 14-day full-featured trial.

Stephen Kappel discusses Datadog’s method of automating piecewise regression to analyze timeseries data, addressing the challenges of identifying breakpoints and determining the number of segments without manual input. The approach involves using a greedy algorithm to efficiently navigate the exponential solution search space by initially overfitting the data with numerous segments and then iteratively merging segments to minimize error while preventing overfitting. The stopping criterion is based on the increase in total sum of squared errors, ensuring the algorithm halts merging when the error increase is minimal. Datadog's solution is implemented in a Python library, which also supports variations such as using different error metrics or fitting step functions, providing flexibility in analyzing different types of data.

Scott Dixon, a solutions engineer, created a Datadog dashboard to monitor the real-time status of the New York City subway system due to frequent service issues. He utilized the MTA's text updates on service status, which categorize lines by service division and status, to parse and send custom metrics to Datadog. Dixon wrote a custom Agent check to parse updates and generate the metric <code>mta.line_service</code>, which indicates if a line has "good service" or is experiencing issues like planned work or delays. By tagging the metrics with line names and statuses, the dashboard provides insights into the number of lines with good service and the performance of individual lines. Service-check widgets help quickly identify lines with issues, and an additional check monitors the availability of the MTA website, which is crucial for data reliability. This project highlights the versatility of Datadog, as users have applied it to various monitoring scenarios, encouraging others to explore its potential with a free trial.

Amazon CodeDeploy is a service that automates application deployment across EC2 instances or on-premise hosts. It integrates with other AWS services and continuous integration/continuous delivery tools such as Ansible, Chef, Puppet, and Jenkins. Datadog's new integration helps monitor AWS CodeDeploy by tracking failed deployments, execution times, and the impact of newly deployed code on application performance. The integration also provides complete observability of AWS applications and infrastructure in one place. AWS CodeDeploy works with rolling updates to EC2 instances or Auto Scaling groups and can use Elastic Load Balancing for traffic routing during deployment. Datadog's integration allows users to visualize CodeDeploy performance, create custom dashboards and alerts, and track deployment events.

AWS CodeDeploy is a service that simplifies the process of deploying changes to applications across EC2 instances or on-premise hosts. It integrates with other AWS services and tools like Ansible, Chef, Puppet, and Jenkins to automate deployment. Datadog's new integration helps monitor AWS CodeDeploy by tracking deployment failures, execution time, and impact on application performance. The integration provides complete observability of AWS applications and infrastructure in one place. AWS CodeDeploy can deploy to on-premise instances but is particularly useful for rolling out updates to AWS applications. It allows users to configure deployments to apply revisions to Auto Scaling groups or use Elastic Load Balancing to route traffic away from instances during deployment. The service also enables automatic rollbacks if a deployment fails and manual stopping of deployments as needed. Datadog's integration with CodeDeploy provides rich context to help identify and fix deployment issues quickly, including tracking deploys, setting up alerts, and visualizing performance trends.