Spark Submit Command Formatter tool

• Spark Submit Command Formatter/Minifier
• References

TL;DR

• Paste a spark-submit command written on one line and get it back split across lines with a trailing backslash per option, or minified back to one line.
• Each Spark option is resolved to the configuration property it actually sets, so --num-executors shows as spark.executor.instances and you can compare a command against spark-defaults.conf directly.
• Your application’s own arguments are kept separate from Spark’s, so a job that takes --input and --output no longer has them mistaken for Spark options.
• Quoted values survive the round trip intact, including an extraJavaOptions string containing spaces.

Spark Submit Command Formatter/Minifier

Format a spark-submit command across lines for review, or minify it back onto one line for a scheduler that wants a single string. Both directions preserve the command exactly.

The useful part is the breakdown underneath. A long spark-submit line hides duplicated or contradictory settings, and the same setting can arrive as either a flag or a --conf, which makes them hard to compare by eye. The parameter table resolves every option to its configuration property, so --executor-memory 18g and --conf spark.executor.memory=18g line up as the same row.

The option-to-property mapping follows the OptionAssigner list in SparkSubmit.scala and the flag names in SparkSubmitOptionParser.java at the Spark v4.2.0 tag, so --principal and --keytab resolve to spark.kerberos.principal and spark.kerberos.keytab rather than the pre-Spark-3 spark.yarn.* names.

References

• Submitting applications for the spark-submit argument reference
• SparkSubmit.scala at v4.2.0, the OptionAssigner list this tool’s flag-to-property mapping follows
• Spark configuration reference for what each --conf key means and its precedence
• Spark Configuration Generator to work out the executor sizes before formatting the command
• Spark JVM troubleshooting playbook for the extraJavaOptions gotchas a long command tends to hide