Spark Configuration Generator tool

• Spark Configuration Generator
• References

TL;DR

• Enter total nodes, cores per node and memory per node; the tool returns executor count, executor cores, executor memory and memory overhead.
• It offers three shapes: Tiny (one core per executor), Fat (one executor per node using every core) and Balanced (five cores per executor).
• The Balanced calculation reserves one core per node for the OS and node manager, subtracts one executor for the YARN ApplicationMaster, and sets spark.executor.memoryOverhead to 10% of the per-executor memory.
• Five cores per executor is a convention, not a derived number. Treat every output as a starting point and confirm it against your own stage metrics in the Spark UI.

Spark Configuration Generator

Spark Configuration Generator tool will generate the spark configuration based on hardware configuration.

References

• Spark configuration reference for the executor memory and core properties this tool emits
• Running Spark on YARN for the ApplicationMaster overhead and container sizing rules
• Spark tuning guide for memory management and the data-locality trade-offs behind the defaults
• Spark JVM troubleshooting playbook for the JVM flags that go alongside these sizes