Databricks’ native monitoring resources

Databricks provides several complementary sources of telemetry for monitoring data engineering, analytics, and Model Serving workloads, with system tables serving as the primary resource for analyzing performance, costs, usage, audit activity, and lineage across Unity Catalog-enabled workspaces. Because system-table data is not real time, live monitoring relies on options such as job notifications, APIs, host-based agents, cloud-provider telemetry, and Model Serving metrics endpoints. Compute monitoring includes system tables for clusters, nodes, instances, and SQL warehouses, while classic clusters can also send logs and metrics externally and be investigated through the Spark UI for execution timelines, DAGs, skew, spill, shuffle, streaming behavior, and driver or executor logs. Job and pipeline performance can be assessed through Lakeflow system tables, Databricks job and pipeline interfaces, event logs, and Spark Streaming Query Listener, with notifications available for immediate status and duration alerts. Query performance is available through query-history tables, APIs, UI profiling, and serverless query insights, while data quality and lineage capabilities include Unity Catalog profiles, anomaly detection, SQL alerts, audit logs, lineage tables, Catalog Explorer graphs, and optional OpenLineage integration. For Model Serving, Databricks offers endpoint health metrics, logs, inference tables, model-quality profiles, LLM evaluation tools, and AI Gateway usage data to analyze request volume, errors, latency, compute consumption, model behavior, and costs.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.