Spark Memory Management: A Sizing Guide for Executor Memory

Apache Spark executor memory management extends beyond `spark.executor.memory`, which sets only the JVM heap, while YARN and Kubernetes enforce a larger container limit that also includes memory overhead, optional off-heap memory, and PySpark worker memory. Within the heap, Spark reserves 300 MiB, assigns a default 60% of the remainder to a unified execution and storage pool, and leaves the rest for user objects and internal structures; execution can evict cached storage but may spill to disk when its share is insufficient. Because executor tasks share execution memory according to the number of concurrent cores, increasing cores without increasing heap can reduce memory available per task and increase spilling. The guide recommends sizing executors from node resources, accounting for operating-system capacity, container limits, overhead, cores, task parallelism, and cluster-manager constraints rather than relying on heap size alone. It distinguishes Java heap failures, commonly reported as exit code 52, from container-level memory kills such as YARN physical-memory violations or Kubernetes exit code 137, which often indicate excessive native, Python, or off-heap memory and may require more overhead rather than a larger heap. Spark UI metrics including peak execution memory, JVM memory, spill volume, failed tasks, and garbage-collection time can help identify whether workloads need more partitions, skew mitigation, fewer executor cores, reduced caching, altered heap sizes, or increased overhead, while driver memory must separately accommodate operations such as collecting results, converting to pandas, and building broadcast joins.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.