January 2024 Summaries

Datadog has expanded its Live Processes feature to include applications running on AWS Fargate, a fully managed compute engine for Amazon ECS and EKS. This allows DevOps engineers, SREs, and application developers to monitor processes running on their serverless infrastructure, which is critical for understanding the performance of these applications. Datadog Live Processes enables users to see every process running across all their ECS tasks in one place, monitor resource metrics, isolate problematic processes causing crashes or latency, investigate anomalous behavior with Watchdog, and spot suspicious processes running on serverless containers. This feature helps ensure that teams have the visibility they need to investigate problems with applications, services, and infrastructure running in AWS Fargate.

Datadog has expanded its Live Processes feature to support applications running on AWS Fargate, a fully managed compute engine for Amazon ECS and EKS. This enables teams to gain process-level visibility into their serverless infrastructure, monitor resource metrics, isolate problematic processes, investigate anomalies, and spot suspicious activity. With this expansion, Datadog provides a more comprehensive monitoring solution for AWS Fargate environments, allowing developers and DevOps engineers to quickly identify issues and resolve them before they impact the application. The feature offers various facets, including task-specific facets and an AWS Fargate facet, which enable filtering and investigation of processes running in ECS or EKS containers. By using Datadog Live Processes for AWS Fargate, teams can ensure visibility into their applications' performance and infrastructure, enabling them to take swift action to resolve issues and maintain application reliability.

This article provides a comprehensive guide on understanding and diagnosing memory issues in Go applications. It covers an overview of Go application memory, how to analyze Go memory usage, and new APM runtime metrics dashboards. The article breaks down process memory into Go memory (managed by the Go runtime) and non-Go memory (allocated using cgo or syscalls). It also explains how to estimate physical Go memory usage and provides a simplified list of newer memory metrics that can be mapped to older MemStats metrics. Furthermore, it discusses heap profiling and goroutine profiling as methods for breaking down heap usage and identifying oversized goroutine pools, respectively. The article concludes with tips on using runtime metrics to resolve memory issues and recommends referring to the official documentation for more information about Go garbage collection and memory optimization.

The article aims to provide a comprehensive primer on Go memory metrics, helping engineers diagnose and resolve memory issues in their applications. It covers the basics of Go application memory, including process memory and kernel memory, as well as how to analyze Go memory usage using various tools and techniques such as profiling, heap profiling, and goroutine profiling. The article also discusses new APM runtime metrics dashboards that provide a breakdown of memory usage, making it easier for developers to monitor and diagnose memory issues in their applications. By understanding these concepts and tools, engineers can better manage the memory usage of their Go applications and resolve common memory-related issues such as memory leaks and OOM kills.

Datadog Data Streams Monitoring (DSM) is an advanced tool that helps optimize event-driven applications using streaming data pipelines such as Kafka or RabbitMQ. DSM provides end-to-end visibility from data ingestion to processing and output, enabling users to detect issues quickly, scale flexibly, analyze complex events, and troubleshoot problems in real-time. The integration of DSM with Application Performance Monitoring (APM) allows for a high-level overview of streaming data dependencies, investigation without workflow disruption, and identification of performance issues such as blocked messages, offline consumers, and high-latency queues. This integration helps users quickly pinpoint the root cause of performance issues and improve application performance.

Datadog's Data Streams Monitoring (DSM) integration in Application Performance Monitoring (APM) provides robust capabilities for monitoring streaming data pipelines, enabling users to optimize event-driven applications and detect issues quickly. DSM offers visibility into all services and queues across the pipeline in one place, allowing users to track performance, identify bottlenecks, and troubleshoot issues without disrupting their workflow. The integration displays a high-level overview of the streaming data architecture, making it easier to analyze application metrics alongside topology information to remediate bottlenecks and improve application performance. With DSM, users can conduct investigations without disruption, pinpoint root causes with precision, and measure how resources are affected by issues in their pipeline.

Azure Kubernetes Service (AKS) is a Microsoft service that allows users to easily deploy and manage containerized applications in Azure. However, monitoring the health of AKS clusters can be challenging due to the large number of containers being orchestrated. Datadog's AKS integration provides complete visibility into AKS clusters by collecting metrics and logs from the entire setup and organizing them into high-level visualizations. The new Datadog cluster extension for AKS enables users to deploy the Datadog Agent directly within Azure, simplifying workflows and reducing overhead. This post explains how to quickly deploy the Datadog Agent across AKS clusters and visualize AKS cluster and control plane activity using Datadog's monitors and OOTB AKS dashboard.

The Azure Kubernetes Service (AKS) provides a platform for deploying and managing containerized applications in Azure, but its sheer volume of containers can make monitoring challenging. Datadog's AKS integration offers complete visibility into AKS clusters by collecting metrics and logs from the entire setup and organizing them into high-level visualizations. However, this integration often requires additional tools to install the Datadog Agent on clusters, adding complexity to workflows. The Datadog cluster extension for AKS solves this issue by allowing users to easily deploy the Datadog Agent directly within Azure, streamlining the deployment process and reducing overhead. Once deployed, users can visualize AKS cluster and control plane activity in real-time, detect issues before they cause bottlenecks, and take steps to debug and troubleshoot problematic nodes. With the extension, users can start monitoring their AKS clusters within minutes, catching issues across all Azure clusters without relying on third-party tools.

Google Cloud Platform's BigQuery is a fully managed serverless data warehouse that enables large-scale data analysis and storage while eliminating infrastructure management overhead. However, users face challenges in tracking storage and compute costs due to various pricing models and shared usage within organizations. Datadog's BigQuery integration helps track and analyze BigQuery usage, providing an out-of-the-box dashboard that visualizes key metrics for cost drivers, resource consumption, query performance, and more. This comprehensive visibility into the data analytics stack enables users to optimize costs, monitor performance, and troubleshoot issues.

Datadog has introduced an out-of-the-box (OOTB) dashboard for its BigQuery integration, which provides comprehensive visibility into BigQuery costs and query efficiency. The dashboard helps users track and analyze their BigQuery usage, gauge the efficiency of their queries, and optimize costs and performance. It offers various key metrics to visualize compute resources and storage consumption, including jobs in flight, bytes scanned, and BigQuery storage. Additionally, it provides dedicated sections for tracking slot allocation and storage usage, as well as monitoring query execution times and optimizing query performance. The dashboard can help users get comprehensive visibility into their data analytics stack, troubleshoot issues, and reduce cloud spend.

Datadog has expanded its Database Monitoring (DBM) service to support Oracle databases, allowing teams using Oracle databases to monitor their resources alongside telemetry from across their environments. DBM supports various deployment configurations of Oracle databases and provides features such as the Databases view for vital metrics, normalized queries for incident investigation, Query Samples for granular insights into database performance, and an out-of-the-box dashboard for cross-team collaboration. This increased visibility helps teams understand resource utilization, identify high-cost operations, and optimize their databases for better application efficiency.

Datadog has released Database Monitoring (DBM) for Oracle databases, providing host-level and query performance metrics and insights to teams operating Oracle databases. DBM supports various deployment configurations of Oracle databases, including self-managed, RDS, RAC, Exadata, Autonomous Database, and Automatic Storage Management. It enables teams to monitor the health and performance of their Oracle databases by exploring the Databases view, using normalized queries to investigate incidents, understanding resource utilization with Query Samples, and facilitating cross-team collaboration with an out-of-the-box dashboard. DBM provides visibility into vital metrics such as queries per second, active connections, wait groups, query performance, and system health, allowing teams to optimize their database operations, identify issues quickly, and improve overall application efficiency.

DevSecOps is an essential practice for organizations to foster in order to deploy resilient and secure applications and services. It breaks down the barriers between development, operations, and security teams by integrating security into all workstreams. This approach shortens feedback loops, empowers engineers to contribute to security guidelines, and provides security teams with context and understanding for effective investigations and reviews. DevSecOps can be implemented using existing tools such as automation and monitoring, and fostering a culture of collaboration between teams. Datadog has successfully integrated DevSecOps into its operations by creating strong communication channels across formerly isolated teams and integrating security into site reliability goals.

DevSecOps is an organizational-wide culture and practice that fosters the integration of security into every workstream, breaking down silos between development, operations, and security teams. This approach has been adopted by organizations to address the long-standing problem of siloed workstreams, resulting in a more resilient and secure software development and release cycle. By unifying workstreams, DevSecOps enables faster troubleshooting, empowers engineers to contribute to security guidelines, and gives security teams context and understanding to conduct higher-quality investigations and share findings with others. In practice, DevSecOps applies automation tools, monitoring, and collaboration between security and engineering teams to create a shared vernacular, context, and goals among developers, operations, and security folks. The benefits of DevSecOps include shortened feedback loops, empowered engineers, and unified workstreams that result in more resilient applications and infrastructure.

The Madrid office of Datadog was founded by Fernando Mayo and Borja Burgos, who previously co-founded Undefined Labs to improve how developers test applications. The team joined Datadog in 2020 through the acquisition of Undefined Labs and has been expanding rapidly, with a focus on accelerating growth in the Madrid hub. The Madrid team is led autonomously, with ownership and impact being core principles of the engineering culture at Datadog. The team's main focus is on building and delivering on the mission of shift-left observability, specifically features such as Static Analysis and Quality Gates. With an international team, the Madrid office brings together knowledge and experience from diverse teams around the world, enabling access to unique challenges and talent. The team is rapidly expanding, with 40 people currently working in different teams, and plans to grow into a more established engineering hub.

The blog post introduces the Profiling Engineering team at Datadog and their work on developing a continuous .NET profiler, which is designed for 24/7 production use with minimal impact on application performance. It distinguishes the profiler from other tools by its ability to run continuously in production environments, collecting data on CPU usage, wall time, exceptions, lock contention, and memory allocations. The post details the architectural components of the .NET profiler, including individual profilers, samplers, aggregators, and exporters that work together to collect, serialize, and send data to Datadog's backend for analysis. The post also discusses the challenges and decisions involved in implementing the profiler, such as choosing native code over sidecar applications, and the mechanisms used to manage CLR event notifications and call stack cleanup. The profiler relies heavily on CLR services to function efficiently and is initialized at application startup, ensuring compatibility with various .NET runtime versions. The post serves as an introduction to a series that will explore the technical choices made to ensure the profiler's efficiency and low impact on production systems.

Modern engineering teams rely on continuous integration and continuous delivery (CI/CD) providers to automate their pipeline processes. However, managing and maintaining the performance of these systems becomes increasingly challenging as teams grow in size and complexity. To address this issue, organizations can leverage tools like CI Visibility to gain shared context around CI/CD workflows. By implementing best practices such as effectively troubleshooting issues, narrowing the scope of investigations with dashboards, drilling down into pipeline issues by tracing CI runners, and creating monitors that span the entire CI/CD system, teams can maintain the speed and reliability of their pipelines while scaling. Additionally, monitoring CI trends over time helps establish baselines for performance and identify performance regressions proactively.

Bowen Chen discusses the challenges of monitoring complex CI/CD systems and shares strategies for troubleshooting issues and creating sustainable practices. Modern engineering teams rely on providers like GitHub Actions, GitLab, and Jenkins to build automated pipelines and testing tools. However, improving performance and troubleshooting failures can be challenging due to varying provider levels of visibility and terminology. Tools like CI Visibility help give teams shared context around CI/CD workflows. To effectively troubleshoot issues, platform engineers can use dashboards, alerting, and monitoring to track pipeline health over time and identify performance regressions. By implementing best practices such as establishing baselines for performance, identifying performance and reliability regressions, and gaining end-to-end visibility into the CI/CD system with tools like Datadog CI Visibility, teams can maintain the speed and reliability of their pipelines while scaling their teams and workflows.

Zendesk is an integrated solution that helps support teams manage customer inquiries and feedback. However, as organizations grow, managing the increasing number of support tickets becomes challenging. Datadog's Zendesk integrations provide a detailed view of customer experience through high-level overviews and granular visibility into customers' sessions within applications. The primary integration allows for tracking customer experience as an organization scales and automatically creates Zendesk tickets based on telemetry data. Additionally, the new integration with Datadog RUM Session Replay enables precise troubleshooting of customer issues by allowing support teams to replicate users' navigation in their apps. Overall, these integrations help organizations improve customer satisfaction and reliability by tracking key metrics and quickly addressing reported issues.

Seagence is a tool that helps developers detect and debug concurrency issues in Java applications by analyzing how requests are processed in real-time. It integrates with Datadog, allowing developers to track defects as events and visualize the root cause in out-of-the-box dashboards. The integration includes a preconfigured monitor that can notify users about Seagence's findings in real time. By using Seagence and Datadog together, developers can proactively detect and debug code-level issues in their Java applications before they affect end users.

Seagence is a tool that helps developers detect and debug code-level issues in Java applications by analyzing how application requests are processed in real time. The Seagence integration with Datadog enables developers to track defects as Datadog events, visualize the root cause in out-of-the-box (OOTB) dashboards, and get deep visibility into Java errors and exceptions. With Seagence's preconfigured monitor, developers can automatically receive notifications about detected issues in real-time, and use Datadog Workflow Automation to kick off automated remediation processes. This integration provides a proactive approach to detecting and debugging code-level issues before they affect end-user experience, allowing developers to address issues promptly and prevent potential problems.

The text discusses the issue of alert fatigue in monitoring systems, where excessive or irrelevant alerts hinder the ability to identify critical issues, leading to disruptions in production and loss of trust in monitoring systems. To combat this, organizations should adopt a continuous improvement methodology by regularly reviewing and updating their monitoring strategies to minimize unnecessary alerts. Key strategies include identifying noisy alerts, specifically predictable and flappy alerts, and implementing team-level adjustments such as increasing evaluation windows, adding recovery thresholds, consolidating alerts, leveraging conditional variables, and scheduling downtimes. Datadog is highlighted as a tool that provides features to reduce alert fatigue through customizable dashboards, integration with other services, and options for precise alert conditions, thereby enhancing communication and troubleshooting efficiency. The text concludes by emphasizing the importance of a well-defined monitoring strategy to prevent alert fatigue and encourages new users to explore Datadog with a free trial.

Datadog has expanded its Workflow Automation capabilities to include Azure actions, allowing engineers to create automated workflows for their Azure resources. With nearly 80 Azure actions available and more on the way, automation can be used to address disruptions, improve response times, and boost overall health and security of Azure-based systems. The new actions use integrations to cover an array of Azure services, providing the means to orchestrate complex workflows across different technologies. This expansion enables teams to automate remediation in response to alerts from Datadog monitors and trigger workflows either manually or automatically in response to security signals.

Datadog has announced an expansion of its Workflow Automation capabilities to include Azure actions, allowing developers to create automated workflows for their Azure resources. This enables engineers to address disruptions, improve response times, and boost the overall health and security of their Azure-based systems. With nearly 80 Azure actions available today, users can automate tasks such as restarting critical services, securing applications, and containing security incidents with just a few clicks. The new capabilities use integrations to cover various Azure services, providing means to orchestrate complex workflows across different technologies. By combining Datadog's alerting capabilities with Azure actions in Workflow Automation, developers can greatly accelerate their response to disruptions, reduce downtime, and maintain clear communications surrounding incident resolution.