Monitor Databricks with Datadog

Datadog’s Databricks integration combines Agent-based Spark and infrastructure telemetry, API polling for job execution data, and system-table queries for cost and lineage information to provide visibility across analytics, data engineering, and AI/ML workloads. Its Jobs Monitoring features track job and cluster health, performance, logs, traces, resource utilization, failures, delays, and costs, while offering anomaly detection, alerting, troubleshooting tools, and rightsizing recommendations that can be linked to cloud infrastructure telemetry and Jira workflows. Data Observability Quality Monitoring extends coverage to Delta and Unity Catalog table freshness, volume, column metrics, custom rules, and lineage, while Data Streams Monitoring maps streaming pipelines and helps correlate upstream issues such as Kafka lag with Spark job failures. The integration also monitors Model Serving endpoints through latency, throughput, errors, CPU, memory, and GPU metrics, with dashboards and alert templates for detecting scaling or reliability problems. Datadog Cloud Cost Management contextualizes Databricks DBU spending with discounted cloud costs, tag-based attribution, anomaly detection, forecasts, and custom metrics for analyzing costs of training, ETL, and inference workloads. Reference Tables further enrich logs, metrics, and events with business and operational metadata to support ownership, cost allocation, targeted alerts, and configuration analysis.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.