Spark Shuffle Partition Generator tool
Enter the shuffle input size, executor count and cores per executor, and get a spark.sql.shuffle.partitions value back, sized so each partition stays inside the range Spark handles well rather than left at the default 200.
TL;DR
spark.sql.shuffle.partitionsdefaults to 200 regardless of your data size or cluster, which is right for almost nobody.- Size it from the shuffle input, not the input file size. This tool targets one partition per 100 MB, then rounds up, so a 20 GB shuffle on 40 cores gives 240 partitions at roughly 85 MB each.
- A second constraint matters as much: the value is rounded to a multiple of total executor cores, or the last wave leaves cores idle.
- Adaptive execution coalesces partitions down after the shuffle is written, so this value sets the upper bound rather than the final count.
Spark Shuffle Partition Generator
Used to generate the spark shuffle partition value based on shuffle input size.
References
- Spark SQL performance tuning for
spark.sql.shuffle.partitionsand the adaptive coalescing that acts on it - Data skew in Apache Spark for why raising this value does not fix one hot key
- Spark memory management for the execution memory each concurrent task divides
Trademarks
Apache Spark and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.
Found this useful?
These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.