January 2025 Summaries

Datadog's dbt Cloud integration provides visibility into the performance and health of dbt job runs, models, and tests, enabling users to monitor runtime insights, model execution and compilation times, and test run results in one unified platform. The integration includes a top list of longest-running jobs, runtime metrics, and test success rate, allowing users to quickly identify issues and optimize their data pipeline operations. With this integration, DevOps teams can scale their dbt runs and models while maintaining visibility into their performance and health.

Datadog has released its On-Call feature, which unifies observability and paging into one platform, enabling teams to build schedules, correlate pages to application errors, and bring together stakeholders in a single place. This new feature is now generally available for users. Additionally, Datadog has released several other features and updates this month, including support for Pinecone vector databases, Mobile Session Replay, and integrations with Google services and Windows Blue Screen errors. These new features aim to improve the overall incident response experience for teams using Datadog.

Husky is a distributed storage system designed for observability data, which means it processes and stores large amounts of new data with minimal updates to existing records. To achieve this, Husky uses a combination of size-tiered compaction and locality compaction methods. Size-tiered compaction involves merging small fragments into larger ones based on exponentially increasing size classes, while locality compaction generates fragments that are as lexically narrow as possible by dividing them into individual levels with exponentially larger sizes. This approach aims to minimize the number of small fragments, reducing object storage GET requests and improving query performance. The system also uses a custom fragment file storage format, a sorting schema, and pruning mechanisms to optimize for query execution, ensuring that only relevant fragments are scanned, reducing latency and cost. Overall, Husky's compaction system is designed to efficiently process trillions of events per day while minimizing the impact on queries and object storage costs.

The text discusses the challenges of protecting applications, resources, and data in cloud environments. The National Security Agency (NSA) and Cybersecurity and Infrastructure Security Agency (CISA) have identified strategies for mitigating these risks, which are also echoed by cloud providers, researchers, engineers, and security vendors. Datadog's security researchers contribute to this effort by publishing timely insights, tools, and recommendations to help mitigate cloud risk. The post focuses on two categories: cloud infrastructure and data security, and cloud identity and access management. It highlights the importance of understanding the Cloud Shared Responsibility Model and managing complex cloud permissions and access controls. Datadog has compiled a roundup of its research, findings, and guides from 2024 and previous years to provide actionable strategies for securing cloud environments. The post also covers DevSecOps practices, threat detection and response, AI security, and other related topics.

The NSA has shared its cloud mitigation strategies in two key areas: DevSecOps practices and threat detection and response. To support these strategies, Datadog Security Labs and The Monitor have contributed research, insights, and guides on integrating security into existing development and operations workflows, defending CI/CD environments, using Infrastructure as Code to secure deployments, and adopting a DevSecOps approach at various stages of the software development lifecycle. Additionally, the authors discuss the challenges associated with cloud security, including maintaining secure code, pipelines, and deployments, and provide research on strengthening threat detection, collecting and interpreting authentication, audit, and activity logs, and monitoring LLMs for vulnerabilities and threats. The post concludes by highlighting Datadog's security platform as a solution to help monitor and secure cloud environments.

The text explains the concept of cloud unit economics, which measures how effectively an organization utilizes cloud services relative to business outcomes, such as conversions or sign-ups. By focusing on the cost per unit of revenue-generating activities rather than the overall cloud bill, businesses can better understand the efficiency and value of their cloud spending. The article highlights how Datadog Cloud Cost Management (CCM) aids companies in using unit economics to track and optimize cloud expenses, allowing them to define custom metrics, visualize performance, and set alerts for changes in unit economics. This approach helps organizations correlate cloud costs with business outcomes, enabling teams to optimize performance and costs actively. Additionally, CCM's dashboards and alerts provide essential visibility and insights for stakeholders, helping them manage budgets and drive business success.

Datadog Change Tracking offers real-time visibility into system changes, surfacing relevant changes directly within Datadog's monitors, dashboards, and service pages. This provides an immediate and unified view of changes such as application deployments, feature toggle updates, Kubernetes events, and database modifications. Teams can promptly identify relevant changes and take informed action, accelerating root cause analysis and reducing time to resolution. Change Tracking helps teams view changes directly in context, correlate recent changes with shifts in health and performance metrics, and analyze detailed change insights. It also provides access to remediation workflows and tools within the Datadog platform, enabling teams to efficiently respond to incidents and improve efficiency and reliability.

The Datadog Summit is an event celebrating the company's community of engineers, SREs, and developers who contribute to its growth. The summit will take place in London on March 4 at Convene Sancroft, St. Paul's, and is free to attend with limited space available. The program includes a keynote featuring customers and members of the Datadog Product team, as well as four hands-on workshops led by experts. Additionally, there will be two breakout sessions focused on SRE concepts and practices, an expo hall with product demos and gamified activities, and a networking reception to conclude the event.

Rust's strong memory safety and efficient code execution make it a top choice for building robust, high-performance systems. However, monitoring Rust applications in production environments can still face challenges such as latency spikes, resource contention, and unexpected bottlenecks. To address this information gap, Datadog provides a guide on how to instrument a Rust application with OpenTelemetry (OTel) and send its collected telemetry data to Datadog. The guide covers preparing the application for telemetry collection, determining how to send OTLP data to Datadog, adding OpenTelemetry dependencies to the manifest file, configuring OpenTelemetry at startup, and configuring the runtime environment. It also provides examples of instrumenting traces, logs, and metrics, as well as using the collected data in Datadog to gain insights into performance and quickly identify issues. By following this guide, developers can ensure their Rust applications meet performance expectations and remain reliable under load.

Watchdog Cloud & API Outage Detection is an AI-driven capability that automatically identifies external cloud and API provider degradations impacting services, helping to save time on investigations. The technology proactively detects outages, providing visibility into the blast radius of the problem, directing teams to take action via notifications, and accelerating root cause analysis. By continuously monitoring for elevated error rates in requests to external providers, Datadog can detect service degradation as soon as it occurs, giving teams a head start in identifying and mitigating issues before they escalate. With this capability, customers can minimize downtime and maintain business continuity, even in the face of external outages, by receiving clear impact visibility and actionable notifications.

The text discusses the challenges of deploying microservice-based applications with Kubernetes and how GitOps can address these issues. GitOps is a declarative approach to continuous delivery that uses version control to manage infrastructure and applications in a centralized repository. It aims to bring familiar developer workflows to infrastructure management, providing a single source of truth for managing both application code and infrastructure. The text explains how GitOps works in practice, including the use of CI/CD tools such as Argo CD and Flux, and how it can help promote releases across environments by enabling automated promotion criteria and visualization of different application instances. It also introduces best practices for implementing GitOps in a Kubernetes environment, including the use of version control, continuous monitoring, and visibility into deployment processes.

Datadog's Database Monitoring (DBM) Recommendations provides a clear and prioritized view of performance bottlenecks in databases, helping teams optimize database performance and resolve issues before they impact production. By surfacing key findings across database hosts and queries, prioritizing issues by severity, and offering detailed summaries and actionable next steps, DBM Recommendations enables targeted troubleshooting and improvement. The tool also provides a centralized view of performance issues, allowing teams to quickly pinpoint and address problems, and offers customization options such as creating custom monitors for query performance or high resource utilization alerts. By cutting through noise and highlighting important findings, DBM Recommendations empowers teams to proactively improve database performance and stability.

The summary of the text is as follows: Postgres was experiencing crashes with a segmentation fault error on multiple EC2 nodes, specifically after running an expensive query that scanned a large database table. The team investigated the issue and found that it was caused by a bug in LLVM's JIT compilation for Arm64 architectures. The bug resulted in incorrect memory allocation and relocation of sections, leading to undefined behavior and crashes. The team identified the root cause as a failure to enforce the Arm64 ABI requirements, which led to gaps in the memory allocation. They discovered that modern versions of LLVM use a new dynamic linker, JITLink, which aims to deprecate RuntimeDyld, and thus was not impacted by this bug. To resolve the issue, Thomas Munro, a Postgres developer, patched MikaelSmith's work into the Postgres source tree and shipped Postgres with a fixed memory manager that respects the Arm64 ABI, fixing the issue in Postgres versions 17.1, 16.5, 15.9, 14.14, 13.17, and 12.21.

Datadog offers Cloud Run metric collection and visualization through its Google Cloud integration. To set up the integration, you need to use service account impersonation, which enables Datadog to gain visibility into your serverless containerized workloads. The default methodology for instrumenting a Cloud Run application is through a sidecar container, which runs alongside your Cloud Run Functions as it collects critical monitoring data. You can also instrument a Cloud Run application through either a Dockerfile or buildpack. Once the integration is enabled and set up, Datadog will automatically start collecting monitoring data from your serverless containers and populating an out-of-the-box dashboard with key metrics. The Cloud Run dashboard includes an overview widget, enabling you to quickly gauge the state of your Cloud Run environment, including a count of serverless containers and requests, the rate of errors, and top lists of revisions using the most memory and CPU. Datadog's integration allows you to set up critical alerts that help maintain the health and efficiency of your services, such as alerts for high rates of 5xx or 4xx errors, enabling you to quickly address issues that may be negatively impacting user experience. With Datadog, you can view Cloud Run monitoring data within the Datadog Serverless view, which surfaces key metrics alongside traces and logs so you can spot errors and quickly pivot between them all. Additionally, Datadog collects and visualizes critical Cloud Run job metrics such as a count of task attempts and completions, as well as a count of running and completed executions. If you are already using the Datadog Google Cloud integration, then Cloud Run logs will automatically be collected; otherwise, you need to set the DD_LOGS_ENABLED environment variable true within your cloud provider's environment settings or in your container configuration to ensure that application logs are captured and sent to Datadog.

You can access key Cloud Run metrics through the Cloud Run UI or within Google's Cloud Monitoring service. Cloud Run is automatically integrated with Cloud Monitoring, allowing you to view metrics without further setup or configuration. The Cloud Run UI provides a quick overview of performance and allows for advanced filtering options such as custom metric analysis and historical data comparisons. Additionally, Cloud Run services emit logs containing details about request and responses, container startups and shutdowns, security alerts, and other important information. You can view these logs within the Cloud Run UI or via the Google Cloud CLI. Furthermore, Cloud Monitoring enables you to collect and visualize metrics from across several different Google tools, allowing for a unified platform for monitoring your serverless applications.

Watchdog, an AI engine from Datadog, is expanding its capabilities to automatically detect faulty Kubernetes changes and provide immediate guidance for rapid remediation. This expansion aims to bridge the gap between deployment activities and their downstream effects, empowering teams to quickly identify root causes of issues, respond faster, and minimize downtime. By leveraging real-time change tracking and impact analysis, Watchdog enables teams to detect potential problems before they become incidents, troubleshoot faster with context and recommended next steps, and ensure no critical Kubernetes issue goes unnoticed. This new capability is available to all current Datadog customers and can be used for free trials or by signing up for a paid subscription.

mark summary: Google Cloud Run is a fully managed platform that enables developers to deploy and scale container-based serverless workloads. It abstracts away infrastructure management, providing an easy-to-use interface for deploying and managing applications without worrying about underlying infrastructure. To ensure optimal performance and cost-effectiveness, it's essential to monitor key metrics such as CPU utilization, memory allocation time, request count, request latency, billable instance time, and running job executions count, among others. These metrics provide insights into resource usage, concurrency, and scaling behavior, enabling developers to make data-driven decisions to optimize their applications' performance and costs.

Datadog distribution metrics provide globally accurate percentiles across distributed systems and applications, allowing organizations to summarize metric data accurately. Distribution metrics capture high-resolution data that can be visualized using heatmaps, enabling exploration of seasonal patterns over time. They also enable the creation of threshold queries for setting service level objectives (SLOs), precise granular monitors for proactive alerting about issues, and identification of statistically calculated outliers with standard deviations. By utilizing distribution metrics, organizations can unlock advanced analysis and monitoring, providing insights at scale to inform data-driven decisions.

The text discusses the challenges of diagnosing memory-related issues in containerized applications, particularly those caused by memory leaks, which can lead to crashes and increased latency. It highlights Datadog's new Memory Leaks section in the Service Catalog, which aids in the investigation of these issues by centralizing telemetry data and providing a guided workflow. This feature allows developers to visualize memory usage, identify outliers, and analyze trends using tools like flame graphs and overlays for OOM (Out-Of-Memory) kills. A detailed example is provided for a service written in Go, demonstrating how the process helps in pinpointing the source of a leak, such as a misconfigured cache, thereby facilitating quicker remediation. The conclusion emphasizes the benefits of using Datadog's Memory Leaks workflow to improve service health and invites users to explore its capabilities through a free trial.

Organizations in regulated industries face challenges in meeting stringent data residency, privacy, and regulatory requirements while maintaining system visibility; this is particularly true when logs need to remain in specific regions or environments. To address these issues without sacrificing observability or requiring multiple logging tools, Datadog has announced its integration with Quickwit, an open source, cloud-native search engine designed for efficient log management. Quickwit enables organizations to deploy and manage logs in their own environments, supporting both cloud and on-premise data centers. This integration aims to enhance Datadog's real-time observability capabilities, offering seamless monitoring and optimization, while Quickwit will continue to support the open source community with an upcoming major update under the Apache License 2.

The Doctor Droid integration with Datadog automates the process of investigating incoming alerts, providing critical context and insights to help users quickly evaluate the severity of an alert and decide on next steps. By running triage workflows on incoming alerts, Doctor Droid groups relevant alerts together and determines their legitimacy, providing a quality score that helps identify noisy alerts and evaluates various aspects of the alerting strategy. The integration also enables users to configure custom playbooks within Doctor Droid to perform initial investigative actions, such as fetching metrics from monitoring platforms like Datadog or querying relevant databases. These summaries are then enriched with organization-specific information, providing a high-level overview of impacted resources and potential root causes. With the Datadog Doctor Droid integration, users can easily access these summaries as Datadog events, which can be configured to send findings from any connected runbooks to the alerting platform of their choice when these monitors are triggered. This enables users to receive enriched alerts within tools that are already part of their workflows, such as PagerDuty, Slack, and Teams. The integration also helps users conduct more in-depth root cause analysis by viewing Doctor Droid's analyses of resources impacted by recent incidents, which can be accessed through the Datadog dashboard. By integrating Doctor Droid with Datadog, users can quickly assess critical issues, reduce alerting noise, and enhance their monitoring tools with automated playbooks and detailed evaluations.

Datadog's Snowflake integration provides full visibility into Snowflake's infrastructure and data retrieval, enabling teams to improve performance and reduce costs by uncovering long-running queries and rooting out inefficient resource utilization. The integration collects telemetry from Snowflake, including logs on query history, security, and event tables, as well as metrics from Account usage and Organization usage schemas. Datadog offers an out-of-the-box dashboard for monitoring Snowflake performance metrics, including key performance indicators such as execution time, average bytes written, and average bytes scanned. The integration also enables teams to monitor virtual warehouses with Log Analytics, optimize their spend with Cloud Cost Management, and create custom dashboards to track additional metrics. By leveraging Datadog's Snowflake integration, teams can gain deeper insights into their Snowflake data and make data-driven decisions to improve performance and reduce costs.

Snowflake is a self-managed data platform that enables users to easily store, process, analyze, and share high volumes of structured and semi-structured data. It has gained widespread adoption due to its unified, scalable, and high-performance platform, flexibility, and built-in dashboards and integration with business intelligence (BI) tools like Tableau and Power BI. Data scientists use Snowflake's integrations with tools like Python and TensorFlow to train machine learning (ML) models, while media platforms can use real-time analytics from Snowflake to track user engagement and optimize content delivery. However, monitoring Snowflake is important for teams due to its complexity and the potential for high costs, particularly if they are using it for multiple purposes or processing large datasets. To monitor Snowflake, teams should focus on tracking metrics such as warehouse query performance, storage usage, and task history to ensure optimal performance, cost optimization, and data accuracy.

The text discusses how Snowflake's built-in web interface, Snowsight, enhances data collection and monitoring capabilities by providing tools to track and analyze key metrics related to virtual warehouse performance, query activity, cost management, and data loading. It outlines how Snowsight's dashboards and views, such as the Warehouse Monitoring view and Query History view, allow users to visualize and optimize resource usage, manage costs, and troubleshoot issues. The text emphasizes the importance of monitoring data loading processes for timely analytics and mentions the use of resource monitors to control compute credit usage effectively. Additionally, it highlights the challenges of monitoring Snowflake in a distributed system and hints at a future discussion on integrating Datadog for comprehensive visibility into Snowflake databases and related infrastructure.

Datadog Cloud Cost Management (CCM) enables teams to understand cloud costs by using the FinOps Open Cost and Usage Specification (FOCUS), which normalizes cost data from multiple providers into a unified format. CCM automatically tags ingested costs with FOCUS data, making it easier to visualize costs across all clouds and SaaS providers, explore cross-cloud costs with a single query, and upload custom cost data for analysis. The platform provides a unified view of cloud costs and cost trends, enabling teams to gain insights into their spending and make data-driven decisions.

Datadog LLM Observability has been integrated with Google Gemini, allowing users to monitor, troubleshoot, improve, and secure their Google Gemini LLM applications. The platform offers automatic instrumentation for detecting and tracing API calls made by the application without manual annotations, simplifying setup and enabling end-to-end tracing of LLM chain executions. It also provides out-of-the-box quality and safety evaluations for detecting hallucinations, toxic responses, and other issues, as well as real-time metrics to monitor performance and token usage. Additionally, it offers an Operational Insights dashboard for collating key usage metrics, allowing users to quickly focus on sources of errors, latency, and unexpected usage patterns. This integration enables users to gain complete visibility and control over their Gemini applications, ensuring cost efficiency and performance while maintaining compliance with regulations.