January 2020 Summaries

The Docker logging driver and delivery mode chosen by developers significantly impact the performance of their containerized applications. The json-file driver, which writes logs to a local file on the host where the container is running, is recommended for most use cases due to its speed and reliability. However, developers can choose from various drivers that forward logs to remote endpoints or use logging driver plugins to create custom solutions. The delivery mode, which determines how the container prioritizes logging operations relative to other tasks, also affects performance, with blocking mode potentially introducing latency and non-blocking mode ensuring high volumes of logging activity don't impact application execution. By selecting the appropriate logging driver and delivery mode, developers can optimize logging reliability and application performance, making it easier to monitor and analyze their containers' logs.

Datadog has introduced Continuous Profiler, an always-on, production code profiler that enables developers to analyze code-level performance across their entire environment with minimal overhead. The tool provides a flame graph of stack traces, allowing developers to visualize how their programs execute in production and identify potential bottlenecks. Datadog aggregates profiles to help users understand overall resource consumption and find hotspots in their code, while also providing features such as profile tagging, correlation with distributed tracing, and actionable insights for performance improvements. Continuous Profiler is now available for Java, Python, and Go, with support for other languages coming soon.

Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that enables developers to build highly available and scalable applications on Kafka. Datadog's new integration with Amazon MSK provides deep visibility into managed Kafka streams, allowing users to monitor their health and performance in real-time. The integration includes an out-of-the-box dashboard providing key metrics such as offline partition count and disk usage of brokers. Monitoring ZooKeeper alongside Amazon MSK offers a comprehensive view of the managed cluster. With Datadog, users can monitor Amazon MSK alongside more than 650 popular technologies, including other Amazon services like AWS Lambda and Fargate.

Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that allows developers to build highly available and scalable applications on Kafka, enabling migration of existing Kafka applications to AWS while handling provisioning and maintenance of Kafka and ZooKeeper nodes. The new integration with Datadog provides deep visibility into managed Kafka streams, monitoring health and performance in real-time through an out-of-the-box dashboard. To ensure reliability, alerts can be set up for disk usage thresholds, and machine learning-powered forecasts can predict when disk usage will exceed a threshold. The integration also tracks offline partition counts, allowing developers to track the number of partitions that are not serving data. Additionally, it provides metrics on ZooKeeper request latency, enabling comprehensive monitoring of managed clusters. With Datadog, developers can monitor Amazon MSK alongside hundreds of other popular technologies and tools, ensuring uninterrupted streaming and performance.

Istio's observability features provide comprehensive insights into service mesh performance and internal operations through metrics, logs, and traces. Metrics are collected either from Envoy filters processing network traffic or from Istio components like Pilot, Galley, Citadel, and Mixer, which are instrumented for Prometheus. Visualization tools such as Grafana, ControlZ, and Kiali enhance monitoring by offering dashboards and graphical interfaces to track mesh-level metrics, component-specific data, and traffic patterns. While Grafana displays real-time service metrics, ControlZ provides per-component insights, and Kiali visualizes service communication within the mesh. Envoy and Istio components also generate logs for troubleshooting, and tracing capabilities using tools like Zipkin help identify latency issues across service requests. Istio's Pilot component offers debugging endpoints to analyze configuration and traffic routing behavior, ensuring effective traffic management within the service mesh.

Istio is an open-source service mesh released in 2017 by Google, IBM, and Lyft, designed to manage network architecture by abstracting network routes, thus simplifying service management without altering application code. It facilitates canary deployments, load balancing, service discovery, and more, while providing built-in telemetry and security features. Visibility into the Istio mesh is crucial for maintaining system health, as it involves monitoring service performance and ensuring Istio components function without overburdening resources. Istio operates via Kubernetes deployment, utilizing components like Envoy, Pilot, Galley, Citadel, and Mixer, each contributing to traffic management, configuration validation, security, and telemetry. Key metrics such as request count, request duration, and response sizes help assess network traffic performance, while component-specific metrics provide insights into the health of Istio's infrastructure, ensuring optimal performance and resource allocation within the service mesh.

The post outlines how to leverage Datadog for monitoring an Istio service mesh, highlighting the ease of collecting metrics, traces, and logs from Istio's components and the services within the mesh without the need for a standalone Prometheus server. It explains the setup process for the Datadog Agent and Cluster Agent using a Helm chart, ensuring integration with Kubernetes API for enriched data collection. The guide also covers setting up Datadog's Istio integration, which uses Autodiscovery to efficiently query Istio's Prometheus endpoints, and details the use of Datadog's APM for tracing requests within the mesh. Additionally, it discusses the visualization of Istio metrics through customizable dashboards, the role of Datadog's Network Performance Monitoring in understanding network topology, and the benefits of setting up alerts for automated monitoring. The post emphasizes the advantages of Datadog's integrated views for managing the complexity of service meshes and offers a free trial for new users.

Web server logs and access logs from technologies such as NGINX, Apache, and AWS Elastic Load Balancing (ELB) provide valuable key performance indicators (KPIs) for monitoring the health and performance of applications. However, storing large volumes of log events can be expensive and difficult to maintain. Generating metrics from high-volume logs allows users to visualize overall trends in application activity while reducing costs associated with indexing large volumes of logs. By identifying important KPIs and creating metrics based on specific log attributes, users can monitor trends and gain insight into their applications' performance. Datadog's log processing and analytics make it easy to create log-based metrics that can be visualized, alerted on, and correlated with other infrastructure metrics and traces.

Watchdog for Infra is an AI-powered tool that automatically detects performance anomalies in infrastructure without any setup required. It supports popular technologies like Redis, PostgreSQL, and Amazon Web Services (AWS) and provides guidance on how to resolve detected issues. The tool identifies anomalous patterns in various areas of your infrastructure, such as host-level memory usage and TCP retransmit rate. Watchdog for Infra creates stories that provide an easy-to-read description of the issue and recommended next steps. Users can create monitors to notify them of detected issues and prevent alert fatigue. The tool is now generally available and does not require any configuration, allowing users to start viewing stories and enabling alerts right away.

Maxim Brown discusses the challenges of storing and analyzing large volumes of web access logs from technologies like NGINX, Apache, and AWS Elastic Load Balancing (ELB). He explains that these logs can provide valuable key performance indicators (KPIs) for monitoring application health and user experience. However, storing all log data can be expensive and difficult to maintain, making it essential to find ways to extract meaningful insights from the noise. Maxim Brown introduces the concept of log-based metrics, which involve generating metrics from specific parts of log events to track trends and identify issues without indexing large volumes of logs. He provides best practices for creating these metrics, such as identifying what information in logs is valuable to monitor, using attributes like client IP, requested URL path, and response status code, and converting other attributes into tags to filter and aggregate metrics. Maxim Brown also discusses how log-based metrics can be used with centralized monitoring platforms like Datadog to visualize, alert on, and correlate logs with infrastructure metrics and traces, making it easier to track long-term trends and perform anomaly detection and forecasting.

Watchdog for Infra is a new feature that automatically detects performance problems in infrastructure without requiring any setup or configuration. It continuously evaluates infrastructure metrics to determine normal baseline ranges, and creates stories that highlight anomalies, providing guidance on how to resolve them. Watchdog can monitor a wide range of technologies, including Redis, PostgreSQL, AWS, and more, and provides automatic alerts when issues are detected, allowing users to take corrective action right away. The feature also includes Metric Correlations, which searches across multiple data sources for similar abnormalities, and allows users to create custom monitors to notify their teams about specific issues.