April 2019 Summaries

You can use PHP's system logger and native logging functions to log errors, but also track the performance of API calls and function calls, or count the occurrence of significant events in your applications. Storing logs in a central file gives you flexibility for processing and analyzing them later on. The Monolog library provides more options for customizing how your logs are formatted and routed. You can create JSON-formatted logs with metadata using the `JsonFormatter`. Monolog also allows you to use processors to log uniform data, assign appropriate log levels to events of different types, and capture PHP exceptions and arbitrary events. Centralizing and storing your logs in a platform like Datadog enables you to offload log processing and long-term storage, aggregate logs from all hosts, and troubleshoot incidents more efficiently.

Datadog has been recognized by Forrester Research as a Leader in its report, The Forrester Wave: Intelligent Application And Service Monitoring, Q2 2019. Unlike previous industry analyst reports, this one gives a nod to the changing landscape where customers want a unified view across all components of their software for faster problem detection and diagnosis. According to the report, Datadog provides a unified dashboard that keeps practitioners in-context when troubleshooting performance issues, giving staff members greater visibility than previously deployed tools, allowing them to react faster. The report validates Datadog's strategic goals, including releasing new features such as Synthetics API Tests and Browser Tests to address customers' needs for monitoring user experiences. Forrester evaluated 13 vendors across three categories, with Datadog scoring highest in the strategy category, and the company is committed to building a unified experience for its customers by expanding its service offerings to cover adjacent areas that are important to them.

Twistlock is a platform that manages security and compliance within various environments, including virtual machines, containers, and serverless functions. Datadog has integrated with Twistlock to allow users to track security and compliance risks alongside their existing metrics, traces, and logs. The integration provides an out-of-the-box dashboard displaying the number of vulnerable hosts and container images over time, as well as lists of CVEs. Users can create custom dashboards that display Twistlock metrics alongside other integrations' data for better visibility into their system's health, performance, and security. Datadog also enriches Twistlock logs with attributes to help users analyze trends in vulnerabilities over time and filter logs by affected infrastructure parts.

MongoDB Atlas is a fully managed NoSQL database service deployed onto AWS, Azure, or GCP platforms. It provides built-in security features and high availability through automatic distribution across availability zones. Datadog has introduced an integration that enables users to monitor MongoDB Atlas health and performance metrics alongside their cloud infrastructure and applications. The new integration offers real-time throughput metrics, broken down by operation type, as well as read and write latency tracking. Users can also track current connection count and set up alerts for unexpected changes in workloads or approaching connection limits. Datadog supports over 650 technologies, including MongoDB Atlas, providing a comprehensive view of all the components within an organization's stack.

MongoDB Atlas is a fully managed NoSQL database that can be deployed on AWS, Azure, or GCP. The new integration with Datadog allows for real-time monitoring of MongoDB Atlas health and performance metrics alongside cloud infrastructure and applications, providing visibility into throughput, read and write latency, and current connection counts. This enables the detection and diagnosis of potential issues such as network problems or database overload, allowing for proactive scaling and optimization. Additionally, the integration provides out-of-the-box dashboards and real-time search metrics for Atlas Vector Search, enabling optimal system memory allocation and performance monitoring. The Datadog integration also supports 850+ technologies, including AWS, Azure, and GCP services, providing an in-depth view of all technologies in the stack, allowing for correlation with performance data from dependent applications.

Datadog offers an automatic context propagation feature that unifies all telemetry data for a single request by linking logs from disparate functions, processes, hosts, containers, and cloud services. This feature is built into Datadog's distributed tracing libraries and requires zero developer time or maintenance. By enhancing tracing libraries to inject the same request-scoped context into logs, users can easily view the logs associated with a request trace, pivot from logs to traces in one click, isolate all the logs from a single request, and flip the switch for auto-correlation of logs and request traces. This solution helps developers troubleshoot faster by providing a comprehensive view of telemetry data for a given request.

The Datadog Unix Agent for IBM AIX is now generally available, providing comprehensive monitoring capabilities for large, complex systems that require accurate, real-time performance tracking. The open-source agent supports multiple AIX versions and offers a native installation experience with easy control using familiar SRC commands. It collects system-level metrics such as CPU, memory, and network usage, as well as extends support to lparstat, iostat, individual process monitoring, custom integration checks, and Dogstatsd for custom metrics. Future updates will include increased integration support on AIX, starting with metrics from the IBM Hardware Management Console (HMC).

Datadog's auto-instrumentation feature automatically brings together all the logs for a given request and links them seamlessly to tracing data, allowing developers to quickly diagnose issues without manual instrumentation. The company's distributed tracing libraries were built to solve the problem of propagating context across services and infrastructure boundaries, enabling the reconstruction of full request lifespans in a single visualization. By enhancing these tracing libraries to inject request-scoped context into logs, Datadog unifies all logs and traces for a given request, requiring zero developer time or maintenance. Once configured, users can easily view logs associated with a request trace, pivot from logs to traces, and isolate all logs from a single request, enabling faster troubleshooting and better application performance visibility.

The Datadog Unix Agent for AIX is now generally available, offering a native experience with an AIX-native .BFF file format installation script. The agent runs as a daemon and supports various system-level metrics such as CPU, load, memory, and network usage, which can be visualized in Datadog. Beyond core system-level metrics, the agent also collects data from custom integrations running on AIX, lparstat and iostat tools, individual processes, and allows for custom metric submission via DogStatsD. The agent is open source and runs as a separate non-root user with options for privileged access. Users can try the agent with a 14-day free trial or provide feedback to increase integration support on AIX in the future.

Datadog's new automated browser tests enable developers to automate user experience monitoring and ensure that users can complete actions like signing up for a new account or adding items to a cart. The tests are simple to implement, requiring only basic knowledge of the application and no coding skills. With machine learning, the tests automatically detect changes to the application and update accordingly. This allows teams to monitor user experience alongside metrics, distributed traces, and logs from their applications and infrastructure. The browser tests also provide end-to-end visibility for troubleshooting, showing screenshots of what users are seeing when a test fails. Additionally, the self-maintaining nature of these tests reduces false alarms caused by flaky tests, allowing teams to focus on building new features instead of fixing broken tests.

Python's built-in logging module is designed to give critical visibility into applications with minimal setup. The main parameters of `basicConfig()` are level, handler, and format, which can be customized to suit specific needs. Logging to a file allows for more customization and routing options, making it easier to troubleshoot issues. Customizing log levels, handlers, and formats can help capture lower-priority logs and exceptions, providing deeper insights into application performance and errors. The logging module also supports JSON formatting, which is easily parseable by external log management services. By centralizing logs with a service like Datadog, developers can explore them alongside distributed request traces and infrastructure metrics to get a full picture of their applications' performance and behavior.

Google Cloud's Stackdriver Logging is now integrated with Datadog, allowing users to collect and analyze logs from their GCP services and applications in a single platform. With this integration, users can search, filter, analyze, and alert on logs alongside metrics and distributed request traces. Stackdriver logs are automatically parsed by Datadog, enabling users to create facets for grouping and analyzing logs. This integration enhances visibility into Google Cloud infrastructure and applications, allowing users to monitor, visualize, and alert on logs alongside platform metrics and application performance data.

Datadog's GCP integration now includes Stackdriver Logging, allowing users to collect and analyze logs from Google Cloud Platform services and applications in a single platform. With this integration, users can export logs to specified endpoints or sinks for further analysis, and Datadog will automatically parse attributes from JSON-formatted logs. The ability to collect Stackdriver logs enables users to monitor their entire GCP infrastructure, including platform metrics and application performance data, giving them end-to-end coverage of their environment. Users can start collecting Stackdriver logs immediately if they already use Datadog, or sign up for a free trial to get started.

The text outlines Datadog's integration with Google Hangouts Chat, emphasizing its utility in enhancing real-time communication and collaboration during outages. By allowing teams to share annotated graphs and receive alerts in designated chat rooms, the integration facilitates swift issue resolution. Users can easily share graph snapshots across multiple Hangouts Chat rooms, initiating discussions to determine if anomalies are expected or require investigation. Alerts can be configured to notify specific chat rooms, ensuring that the right teams receive pertinent information promptly. The integration also includes direct links to Datadog dashboards for further exploration of logs and metrics, providing a comprehensive approach to monitoring and addressing technical issues efficiently. The setup process is straightforward, encouraging teams unfamiliar with Datadog to try out the service with a 14-day free trial.

Google Cloud Run is a compute platform designed to run stateless containers without the need to manage infrastructure, available as a fully managed service or through Cloud Run for Anthos, which operates across Google Kubernetes Engine, multi-cloud, on-premise, and hybrid environments. It leverages Knative for serverless computing, enabling users to deploy containers with ease by specifying configurations such as environment variables and memory limits. Datadog, a launch partner, offers integrations that provide visibility into Cloud Run services, allowing users to monitor metrics, logs, and traces across both cloud-based and on-premise infrastructures. Through Datadog's integration, users can automatically collect and analyze performance data, including latency and resource usage, and utilize tags for efficient data management. Additionally, Cloud Run supports audit logs for detailed activity tracking, which can be integrated with Datadog's Stackdriver Logs for comprehensive monitoring. Whether managing infrastructure locally or in the cloud, Cloud Run offers scalable solutions with Datadog providing robust monitoring capabilities.

Service Level Agreements (SLAs) are crucial for improving the performance and reliability of services, benefiting both service providers and users. SLAs involve defining clear objectives using Service Level Objectives (SLOs) and Service Level Indicators (SLIs). To set reasonable SLAs and SLOs, it is essential to collect data on key metrics such as latency, throughput, and error rates from user-facing applications and subcomponents. Datadog Synthetics simulates user conditions and helps establish performance expectations, while Datadog APM tracks real user interactions and identifies underperforming services or code-level inefficiencies. By analyzing the entire distribution of metrics rather than just averages, more accurate insights can be gained. Datadog APM generates dashboards with SLI metrics for each internal service, allowing teams to define objectives that make sense for their specific stack components. Customizable, comprehensive dashboards enable monitoring and assessing the health of services and their underlying infrastructure-level components. SLO-driven alerts can be set up in real time to trigger at increasing levels of severity as metrics approach internal and external SLA thresholds. Integrating APM with infrastructure monitoring allows for tracing requests across various services, making it easier to investigate issues and identify potential bottlenecks throughout the entire stack. To implement an effective SLA strategy, organizations should first establish a monitoring platform, followed by setting up dashboards and alerts that reflect their SLAs and key resources or services.

Microsoft's .NET Framework has gained popularity since its introduction in 2002, with organizations such as UPS, Stack Overflow, and Jet.com using it. The rise of the .NET Core runtime now supports cross-platform development for this high-performance framework. Datadog APM and distributed tracing are now generally available for both .NET Framework and .NET Core applications, allowing deeper visibility into these environments. Datadog's .NET monitoring provides real-time insights into application performance, helping developers identify bottlenecks and optimize their applications. The Service Map automatically maps out microservices architecture based on data collected from APM, while the App Analytics feature helps find specific request traces for investigating errors and bottlenecks. Watchdog uses machine learning to alert users of anomalies in .NET apps. Datadog integrates with cloud services like Azure and AWS, allowing users to monitor their .NET applications and infrastructure in one place.

Microsoft's .NET Framework has gained significant traction since its introduction in 2002, with a large user base including notable organizations. The rise of the .NET Core runtime has expanded its reach to cross-platform development. Datadog APM now supports both .NET Framework and .NET Core applications, providing deeper visibility into these environments. This addition brings .NET into line with other supported frameworks and languages in APM. Datadog's .NET client automatically instruments various data stores, web frameworks, and databases, making it easy to integrate and monitor .NET applications without code changes. The platform offers features such as the Service Map, service overview dashboards, App Analytics, Watchdog, and flame graphs to analyze and understand application performance in real-time. These tools help identify dependencies between services, optimize application performance, investigate issues, and provide customer-level visibility. Datadog's integration with cloud services like Azure and AWS enables monitoring of both .NET applications and infrastructure in one place, making it easier to connect the dots across environments.

Maxim Brown discusses the importance of monitoring an EKS cluster and provides various methods for accessing Kubernetes metrics, including Kubernetes-native methods, AWS CloudWatch, and a dedicated monitoring service. He explains how to use tools like `kubectl`, Kubernetes Dashboard, and AWS CLI to collect and visualize metrics from the cluster. He also highlights the limitations of these methods and introduces a dedicated monitoring service that can centralize all sources of data into one platform, providing a more complete picture of the EKS cluster's health and performance.

Datadog's integration with Kubernetes, Docker, and AWS provides comprehensive monitoring for Amazon EKS clusters by collecting and visualizing metrics, logs, and traces. The Datadog Agent, which can be deployed across EKS clusters, gathers Kubernetes state metrics and resource metrics from nodes and containers. The integration with AWS CloudWatch allows for the collection of AWS service metrics, enhancing visibility into the health and performance of infrastructure components like EBS volumes and ELB load balancers. By employing features such as Autodiscovery, detailed tagging, and customizable dashboards, Datadog enables real-time tracking and analysis of dynamic infrastructure and applications. Furthermore, it supports advanced alerting mechanisms, including machine-learning-driven alerts and integrations with notification services like PagerDuty and Slack. This setup allows users to proactively manage their EKS clusters, automatically scale resources, and respond to potential issues efficiently.

Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes platform by AWS that provides Kubernetes-as-a-service, automating the provisioning and management of infrastructure to ensure high availability across multiple availability zones. EKS integrates with AWS services like VPC, ELB, EBS, and EC2 or AWS Fargate to enhance network isolation, load balancing, persistent storage, and compute resource provisioning. Monitoring EKS clusters is crucial for maintaining application performance and identifying resource bottlenecks, focusing on metrics like resource utilization, pod availability, and cluster state, as well as AWS service metrics such as those from EC2 and EBS. The article emphasizes the importance of tracking events and metrics to ensure cluster health, offering insights into Kubernetes operations and AWS infrastructure performance, which are integral for capacity planning and troubleshooting. It also previews further guidance on collecting these metrics using Kubernetes tools and AWS CloudWatch in the next part of the series.

At Datadog, ensuring reliable data pipelines is crucial due to the massive volume of data processed daily. Reliability is defined as a system's ability to produce correct outputs up to a given time. A highly reliable pipeline doesn't necessarily mean it never fails; instead, it consistently delivers data on time even if it crashes occasionally. To guarantee reliability, several factors must be considered when designing pipelines: fault tolerance, good monitoring, and preparedness for failure recovery. Datadog uses a simplified architecture consisting of an object store for historical data, clusters running Spark data pipelines, Luigi workers for task and workflow management, and Spark workers for compiling code and sending it to the appropriate cluster. Datadog's approach involves using separate clusters for each job instead of one giant cluster, which provides isolation between jobs, easier monitoring, customized tuning for specific jobs, and easy scaling up or down as needed. The use of spot instances in AWS also forces fault tolerance design. To avoid long-running jobs that can lead to significant work loss during failures, pipelines should be broken vertically by separating transformations into multiple jobs with intermediate data checkpoints and horizontally by partitioning input data and running multiple jobs to process the whole dataset. Monitoring is essential for early detection of failures, using system metrics, job metrics, and data latency metrics. In case of failure, quick recovery and minimizing customer-facing impact are crucial goals. This can be achieved through breaking down jobs into smaller pieces, increasing cluster size, switching from spot to on-demand clusters, and establishing easy ways to rerun jobs. Additionally, having backup plans for query systems in case of pipeline delays helps maintain service operability at the cost of slightly degraded performance.

Quentin Francois, a data engineer at Datadog, shares the company's best practices for building reliable data pipelines. According to Francois, reliability is not about never failing, but rather ensuring that data pipelines deliver good data in time. To achieve this, it's essential to design pipelines with fault tolerance and monitoring in mind. This includes using object stores, cloud Hadoop/Spark services, and spot instances, which can fail at any time due to supply and demand fluctuations. However, with proper architecture, clustering, and monitoring, failures can be anticipated and recovered from quickly. Key strategies include breaking down jobs into smaller pieces, monitoring cluster metrics, job metrics, and data latencies, and thinking ahead of potential failures. By following these best practices, data engineers can build more reliable data pipelines that meet the needs of downstream users.