Spark Analyzer: Measure Wasted Executor Time in Spark

Spark Analyzer is a free Onehouse command-line tool designed to analyze completed Spark applications through History Server APIs, helping teams assess real workloads rather than relying solely on standardized benchmarks such as TPC-DS. It reports resource efficiency, idle-compute waste, stage-level consumption, and the distribution of work across extract, transform, and load phases, allowing organizations to identify costly patterns such as skew, spilling, shuffle-heavy joins, and excessive small-file writes. Tests on deliberately flawed Spark jobs showed that skew could leave 40.8% of allocated compute idle, while a small-files workload used 81.8% of compute on file writing despite appearing highly efficient by utilization metrics. The tool works with open-source Spark, EMR, Databricks, Dataproc, and Microsoft Fabric, can run entirely locally with outputs saved as JSON and Excel reports, and is intended to complement Spark UI by identifying which applications and stages deserve detailed investigation. It also provides conservative stage-based estimates of potential acceleration with Onehouse’s Quanton engine, while reporting similar performance when no expected benefit exists.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.