September 2017 Summaries

Datadog introduces Live Container view, offering real-time insights into the status and performance of all containers in an infrastructure. With Docker adoption on the rise, monitoring challenges arise due to short lifetimes and hardware-agnostic behavior of containers. The Live Containers view provides detailed information about containers' health, resource consumption, and deployment, along with integrations for Docker, Kubernetes, ECS, and other container technologies. Datadog's tagging allows users to filter and focus on specific layers of their infrastructure. Additionally, the table offers summary graphs for longer timespan analysis, individual container inspection, and visibility into provisioning.

AWS Lambda is a popular service for running code on demand without managing underlying compute instances. Monitoring serverless applications like those in AWS Lambda presents unique challenges compared to traditional application monitoring, as there's no long-lived host to monitor and collect telemetry data. However, proper instrumentation of applications and supporting services can ensure observability even in the absence of metrics on CPU, memory, etc. Datadog provides a serverless monitoring solution for AWS Lambda that includes custom metrics from Lambda functions, integration with other AWS services, and alerting features.

The text discusses the management of multiple AWS accounts for security purposes. Having separate accounts can provide natural security boundaries and isolation between workloads. However, managing multiple accounts can add operational complexity. To manage this effectively, the author outlines a set of requirements that include having IAM users only in one account, limiting API access by default, requiring MFA for privilege escalation, and grouping privileged API calls together by topic. The text then provides an example implementation using IAM constructs such as users, groups, and roles to demonstrate how these requirements can be met.

The text discusses account sprawl in AWS, a common problem where multiple accounts are created for different teams or individuals, leading to operational complexity and security risks. The author proposes a solution to manage multiple AWS accounts securely by creating a central management account (M) that can assume roles in the other accounts. The proposed design includes requirements such as IAM users only existing in one account, limited API access to other accounts, role delegation with MFA, and grouping privileged API calls by topic. The author provides an example implementation for three accounts and three users, demonstrating how to create groups, roles, and policies to manage access and permissions. However, the solution is limited to 10 IAM users per group, and the author invites feedback and critiques on their approach.

AWS Lambda is an event-driven compute service that abstracts away the underlying physical computing infrastructure, allowing developers to focus on their code rather than the execution environment. Monitoring applications running in AWS Lambda presents unique challenges due to its serverless nature, but proper instrumentation can ensure observability even without traditional system metrics. Key performance metrics for Lambda functions include duration, invocations, errors, and throttles, which are available through AWS CloudWatch console. Custom metrics from Lambda functions can be used to track application performance, such as requests, responses, latency, and work done to service requests. Instrumenting entry and exit points of the application, as well as code segments where actual work is performed, allows for capturing high-resolution metrics on request rate, response time, and function duration. Combining and correlating metrics from all data sources provides a comprehensive view to diagnose performance issues and monitor overall system health. Datadog's machine learning-powered alerting features can automatically notify of potential issues, ensuring that the right person is alerted when necessary.

Datadog APM's Python tracing client now supports integration with several asynchronous libraries such as asyncio, aiohttp, gevent, and Tornado. This enables developers to gain visibility into their distributed applications using end-to-end flame graphs with minimal configuration. Asynchronous programming optimizes I/O-bound Python applications by implementing cooperative multitasking instead of multithreading. Datadog's Python APM client has been designed specifically for async support, allowing users to trace and debug asynchronous code more effectively. The end-to-end flame graphs provide insights into how tasks are executed concurrently across the environment, helping developers identify errors or latency issues in their applications. To start tracing asynchronous Python applications, users need to configure the tracer with the correct context provider based on the async framework or library they're using.

Datadog APM's Python tracing client now integrates with several asynchronous libraries to provide end-to-end visibility into distributed applications. Asynchronous programming optimizes I/O-bound Python applications by implementing cooperative multitasking, but it can make tracing and debugging more challenging due to context switching. Datadog's Python APM client has been designed to include async support, allowing users to easily instrument synchronous or asynchronous code. The client provides end-to-end flame graphs that enable tracing of individual requests across service boundaries, helping identify errors and latency issues. By integrating with popular async libraries, the client can auto-instrument applications and provide targeted alerts for service-level indicators such as p95 latency and 5xx error rates.

Drupal Association has centralized its monitoring data using Datadog to manage its complex and evolving infrastructure, which comprises various sites and services. The organization previously used a patchwork of solutions for monitoring but switched to Datadog for better visibility and control over their infrastructure. With Datadog's integrations, they can now monitor all systems, applications, and services in one place, reducing the time spent maintaining their infrastructure and increasing its reliability and performance.

The Drupal Association, led by Director of Engineering Tim Lehnen, has centralized its monitoring data in one place using Datadog for monitoring and alerting. The project's infrastructure is complex and evolving, comprising multiple sites and services, including content delivery networks, code repositories, automated testing, and more. Prior to switching to Datadog, the team used an ad hoc system that produced redundant alerts and made it difficult to separate signal from noise. With centralized monitoring, they can now track everything in one place, including applications, services, and infrastructure components, using integrations with various tools such as Jenkins and New Relic. This allows them to quickly identify issues, reduce maintenance time, increase uptime and performance, and provide better insights over time.

The guide provides a comprehensive overview of setting up monitoring in a Kubernetes cluster using Datadog. It covers various techniques, including collecting resource metrics and cluster status from Docker and Kubernetes, using Autodiscovery to collect metrics from services with simple configurations, creating pod annotations for more complex use cases, collecting custom metrics from containerized applications via DogStatsD, and watching the orchestrator. The guide also provides instructions on how to set up monitoring in a real-world scenario using a guestbook application as an example.

Datadog, a company founded in 2010 with the aim of providing comprehensive observability for teams across development, operations, and business lines in public and private cloud environments, has recently announced the addition of log processing and analytics to its platform through the acquisition of Logmatic.io. The integration of logs into Datadog's existing metrics and tracing features creates a unified solution that provides the "three pillars of observability": metrics for overall performance insights, traces for end-to-end visibility into requests, and logs for rich context around discrete events. This acquisition aligns with Datadog's core design principles and enhances its platform by bringing all three pillars together in one place. The company is currently working on integrating Logmatic technology into Datadog to provide a seamless customer experience, with plans to unveil the fully integrated product later this fall.

We created Datadog in 2010 with the goal of providing unified and complete observability for teams spanning development, operations, and business lines in public and private cloud environments. For the first phase of our observability roadmap we built a product that provides seamless, deep metrics coverage for all modern infrastructure components. Then, earlier this year, we launched our APM product to provide end-to-end tracing of requests and business transactions as well as automated instrumentation of applications. Today we are very pleased to announce the addition of log processing and analytics to Datadog, through our acquisition of Logmatic.io. We believe that logs, when combined with other sources of information we collect, provide new insights forming the “three pillars of observability”: metrics, traces, and logs. These three pillars will now be integrated into one place, providing a seamless customer experience. The addition of log processing and analytics to Datadog is expected to be formally unveiled later this fall.

Charles Masson's discussion focuses on the use of robust statistical distances for designing effective anomaly and outlier detection algorithms in machine learning applications. The text delves into various statistical distances, such as the Kolmogorov-Smirnov Distance, Earth Mover's Distance, and Cramér-von Mises Distance, explaining their roles in comparing data distributions. Visual tools like Q-Q plots are also explored as initial heuristic methods for distribution comparison. Each distance measure has unique properties and applications, with the Kolmogorov-Smirnov Distance being sensitive to local deformations, the Earth Mover's Distance providing insights into distributions with long tails, and the Cramér-von Mises offering a balance between the two. The article emphasizes the advantages and limitations of each distance measure, as well as their implementations, including recent enhancements to SciPy, and provides an interactive visualization tool for practical experimentation.