Spark UDF Performance: Quanton Up to 1.5x Faster Than EMR

Quanton reports that it improves Spark jobs dominated by Scala UDFs and wide rows by retaining UDF execution in the JVM while reducing surrounding overhead from row copying, columnar-row conversions, shuffle processing, caching, spilling, and partitioned writes. In customer-run AWS tests of a FinOps vendor’s approximately 500-column pipeline, Quanton reduced runtime versus tuned Amazon EMR by 13.5% for a median tenant, 26.2% for a p97 tenant, and 34.4% for a p99 tenant without code changes. For the largest workload, which expanded from 7.45 billion input rows to 19.5 billion rows, Quanton completed in 7,281 seconds on 180 nodes, while EMR exhausted local disk and failed at that size before completing; EMR required 300 nodes to finish in 7,092 seconds. Quanton attributed its lower resource use to compact columnar shuffle, spill, and cache handling, reporting 67% less local-disk data written and 38.4% fewer core-hours than EMR’s successful 300-node run. The vendor subsequently migrated the pipeline to Quanton on Amazon EKS, while the report argues that conventional JVM engines incur costly wide-row copies and native engines can suffer expensive transitions between native columnar execution and JVM-based UDFs.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.