Spark internals interview questions: data skew, shuffle and performance tuning
Ninety interview questions with answers on three areas where Spark jobs actually fail: data skew, the shuffle subsystem and performance tuning. Each answer gives the decision, the mechanism and what it costs, and links to the measured walkthrough.
TL;DR
- Ninety questions in three groups: data skew, shuffle internals and performance tuning, thirty each.
- Every answer is collapsed. Open one at a time and answer it aloud before reading.
- Each answer links to the post that measures the mechanism, so you can see the numbers behind the claim.
Data skew
Data skew: what it is, how to find it, what it breaks and how to fix it. The answers link into the data skew post, where each mechanism is measured.
What skew is
1. What is data skew in Apache Spark?
An uneven distribution of rows across partitions, such that one partition holds far more than the others. It matters because a stage finishes when its slowest task finishes, and a task processes exactly one partition on one thread. So a single oversized partition makes the whole stage serial no matter how many executors are idle.
The number that captures it is the ratio of the largest partition to the median. On the dataset in this post that ratio is 27x before any fix.
2. What causes data skew in Spark?
A shuffle assigns each row to a partition by hash(key) % numPartitions, so the
distribution of partitions is the distribution of the key. Common sources:
- Natural frequency. A few customers, products or tenants genuinely account for most of the rows.
- Placeholder values.
null,-1,"unknown",""and epoch-zero dates all hash to one partition and are often the single largest key. - Low cardinality. Joining on
countryorstatusgives you as many useful partitions as there are distinct values, whateverspark.sql.shuffle.partitionssays. - Upstream layout. One enormous input file, or a partitioned source where one directory dwarfs the rest.
Which kind of skew is it? separates these, because the fix differs: a placeholder key is usually deleted, a natural hot key is usually isolated or salted.
3. How does data skew affect Spark job performance?
Not as a uniform slowdown. The symptom is a stage sitting at 199 of 200 tasks complete while one task runs on, with the cluster otherwise idle. Adding executors does nothing, because the work that was already finishing early just finishes earlier.
Past a point it stops being slow and starts failing — spill, then GC pressure,
then OutOfMemoryError or an executor lost to the heartbeat timeout. See
what it looks like when skew wins.
6. What is a hot key?
A single key value whose rows are numerous enough to dominate the partition it hashes to. “Hot” is relative to the median key, not an absolute count: a key with a million rows is not hot if every other key also has a million.
7. How are hot keys related to data skew?
A hot key is the usual cause; skew is the effect on the partition distribution. The distinction matters because it tells you where to look. If one key is hot, you can isolate or salt that key specifically (Fix 4). If the distribution is lumpy with no single dominant key, key-specific fixes do nothing and you want a layout or partitioner change (Fix 6, Fix 8).
10. What is partition imbalance in Spark?
The general form of the same problem: partitions differing enough in size that
task durations diverge. Skew from a hot key is one cause. Others are an
unbalanced input layout, a filter that happens to eliminate most rows from some
partitions, or repartition(n, "col") on a column with few distinct values.
Imbalance is measured, not assumed — max, median and their ratio, as in Step 1: measure it.
Finding it
4. How can you identify data skew in a Spark application?
Measure the partition distribution directly rather than inferring it from
runtime. spark_partition_id() grouped and counted gives you max, median and the
ratio between them in one query — that is the whole of
Step 1.
Two cheaper signals worth knowing: the Spark UI’s task metrics summary for the stage, and a pre-flight count of the top keys before you run the expensive job at all (can you detect skew before running?).
5. How can you identify skewed partitions using Spark UI?
Open the stage and read the summary metrics table, which reports min, 25th percentile, median, 75th percentile and max for each metric. Skew shows up as a large gap between the median and max rows for Duration, Shuffle Read Size / Records and Spill.
A median of 2s against a max of 90s is skew. A median of 2s against a max of 3s with a long stage is simply a lot of work. Reading the task metrics summary walks through the distinction, which is the part people get wrong — a long stage is not evidence of skew, a wide max-to-median gap is.
23. How can you identify a hot key using SQL or PySpark?
Count by key and look at the head of the distribution:
(df.groupBy("customer_id").count()
.orderBy(F.desc("count"))
.show(10, truncate=False))
Compare the top count against the median count rather than against zero. If you cannot afford a full count on the real table, sample it — the rank of a hot key survives sampling even though the count does not, which is the point of pre-flight detection.
Do not forget to count null separately; groupBy keeps it as a group, but it is
easy to miss when scanning output.
28. How would you troubleshoot a Spark job where one task takes significantly longer than others?
In this order, because each step is cheaper than the next:
- Is it skew or one bad executor? Check whether the slow task moves to a different host on retry. If the same partition is slow wherever it runs, it is skew; if the same host is slow whatever it runs, it is hardware.
- Read the stage summary metrics. A max-to-median gap in shuffle read confirms skew and tells you roughly how bad.
- Find the key. Count by the join or group key on the input to that stage.
- Check for placeholders first.
nulland sentinel values are the most common single answer and the easiest to fix. - Then pick a fix by what the data allows, not by preference — which skew fix should you choose?
What it breaks
8. How does data skew affect joins?
A shuffle join co-locates matching keys, so all rows with the hot key on both
sides land in the same task. The cost is multiplicative rather than additive: if
the hot key has n rows on the left and m on the right, one task produces
n x m output rows.
That is why a skewed join can be far worse than a skewed aggregation on the same data, and why joins get most of the attention in this post.
9. How does data skew affect groupBy() operations?
Less severely than joins, because there is no second side to multiply against and
because Spark partially aggregates on the map side first. For algebraic
aggregations — count, sum, min, max, avg — the map-side combine reduces
each key to one partial row per input partition before the shuffle, so the hot key
arrives at the reducer as a handful of rows rather than millions.
The exceptions are the ones that cannot combine: collect_list, collect_set,
exact count(distinct), and anything inside a UDAF that holds state per group.
Those ship every row and skew exactly like a join.
26. How does data skew cause executor memory issues?
Because several structures are sized per task, not per cluster. A task’s share of execution memory holds its sort buffers and hash map; a skewed task needs far more than its share and spills to disk. Spill is survivable but slow, and it is the warning sign immediately before failure.
The specific killers are a hash join build side that will not fit, a sort that spills repeatedly, and any aggregation holding unbounded state per group. Add the join multiplication from question 8 and one task can be asked to materialise orders of magnitude more than the others.
27. How can data skew lead to task failures?
The usual progression is spill, then GC thrash, then either
java.lang.OutOfMemoryError or the executor being declared lost after missing
heartbeats while paused in GC. Retries make it worse: the retried task is the same
partition, so it fails the same way until the stage hits spark.task.maxFailures
and the job dies.
A stage that fails on attempt 4 having failed identically on 1, 2 and 3 is diagnostic — random infrastructure failures do not reproduce that precisely.
Removing the shuffle, and letting the engine help
15. When would you use a Broadcast Join to solve data skew?
Whenever one side fits in memory, and it is the first thing to try. Broadcasting sends the small side to every executor and joins locally, which removes the shuffle — and skew is a property of a shuffle. No shuffle, no skewed partition, regardless of how lopsided the key is.
The limit is spark.sql.autoBroadcastJoinThreshold, 10 MB by default, applied
to Spark’s size estimate rather than the file size. Raising it is reasonable; the
memory is consumed once per executor, not once per task. It stops being an option
when the small side is genuinely large, which is what
Fix 7 exists for.
See Fix 1.
16. What is Adaptive Query Execution (AQE)?
Re-planning at runtime. Spark plans a query from statistics it has before execution, which are often wrong. AQE lets it revise the plan at shuffle boundaries using the statistics it has just measured — the actual sizes of the shuffle it has already written.
It does three things: coalesces small partitions, switches join strategies when
the measured size makes a broadcast viable, and splits skewed partitions. It is on
by default (spark.sql.adaptive.enabled is true).
17. How does AQE handle skewed joins?
At the shuffle boundary it looks at the map output sizes. A partition is treated as skewed when it is both:
- larger than
skewedPartitionFactortimes the median partition, and - larger than
skewedPartitionThresholdInBytesin absolute terms.
A partition meeting both is split into several smaller ones, and the matching partition on the other side is duplicated so every split still sees the rows it must join against. The correctness argument is that duplication, and it is why this only applies to joins.
Both conditions must hold, which is the part that surprises people — see the next question.
18. What is spark.sql.adaptive.skewJoin.enabled?
The switch for that rule, true by default. The two thresholds beside it matter
more:
| Setting | Default |
|---|---|
spark.sql.adaptive.enabled |
true |
spark.sql.adaptive.skewJoin.enabled |
true |
spark.sql.adaptive.skewJoin.skewedPartitionFactor |
5.0 |
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes |
256 MB |
spark.sql.adaptive.advisoryPartitionSizeInBytes |
64 MB |
Read off a running cluster rather than the documentation, and identical on both Spark 3.5 and 4.1.
The absolute threshold is the trap: a partition can be a hundred times the median and AQE will leave it alone if it is under 256 MB. That is the right default for a cluster — splitting a 20 MB partition buys nothing — but it means AQE does nothing on laptop-scale data however skewed it is, which is exactly the situation people test in before concluding the feature is broken. Lower the threshold to see it fire on small data. See Fix 2.
19. What is the difference between salting and AQE for handling skew?
AQE splits a partition that is already skewed, after the shuffle has been written, and needs no code change. Salting changes the key so the skewed partition is never created, and requires you to rewrite the join and undo the salt afterwards.
| AQE skew join | Salting | |
|---|---|---|
| Code change | none | rewrite join, replicate, strip salt |
| When it acts | after the shuffle, at runtime | before the shuffle, by construction |
| Small data | does nothing under 256 MB | works at any size |
| Aggregations | joins only | works for groupBy too |
| Correctness risk | handled by the engine | yours — mismatched replication silently drops or duplicates rows |
Try AQE first because it is free. Reach for salting when the data is too small for AQE to fire, when the operation is an aggregation rather than a join, or when the split is not enough.
Salting
12. What is salting, and how does it solve data skew?
Adding a random component to the join key so that one hot key becomes n
moderately warm ones. If the key was customer_id, the salted key is
(customer_id, salt) where salt is a random integer in 0..n-1. Hashing now
spreads those rows across n partitions instead of one.
The cost is on the other side of the join: because the large side’s rows are
scattered across n buckets, the small side must be replicated into all n,
so every bucket can find its match. That replication is the whole correctness
story.
On the dataset here salting takes the max-to-median ratio from 27x to 1.9x — see Fix 3.
13. How would you implement salting in PySpark?
Salt the large side, replicate the small side, join on both components, then drop the salt:
from pyspark.sql import functions as F
SALTS = 16
large_salted = large.withColumn(
"salt", (F.rand() * SALTS).cast("int"))
small_replicated = (small
.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALTS)]))))
joined = (large_salted
.join(small_replicated, ["customer_id", "salt"])
.drop("salt"))
Then assert the row count matches the unsalted join. Every salted example in this post carries that check, because the failure mode is silent.
For an aggregation there is no replication step at all: aggregate by
(key, salt), then aggregate the partials by key.
14. What are the limitations of salting?
- It multiplies the small side by the salt count. 16 salts means 16 copies; the small side stops being small. Over-salting measures this.
- It is a correctness risk. Replication that does not match the salting drops or duplicates rows with no error. Check the row count in the job, not just while investigating.
- It salts everything, not just the hot key. Every key pays the replication cost so that one key stops being hot. Isolating the hot key (Fix 4) avoids that.
- The salt count is a tuning parameter with no good default. Too few and skew remains; too many and the small side explodes.
- It changes the code, so it has to be maintained and understood by whoever reads the job next.
The harder cases
11. How would you handle data skew during a large join?
In order of how much they change:
- Broadcast if either side fits — removes the shuffle entirely.
- Let AQE split it if the partition clears 256 MB — free, no code change.
- Filter first. Push predicates and drop placeholder keys before the shuffle; less data cannot skew.
- Pre-aggregate if the join is followed by an aggregation (Fix 5).
- Isolate the hot key, joining it separately and unioning (Fix 4).
- Salt (Fix 3).
The ordering is deliberate: the early options remove or shrink the shuffle, the later ones redistribute it. A problem removed needs no tuning.
20. How would you handle skew when both sides of a join are large?
Broadcasting is out, so the choices narrow:
- AQE skew join, which is designed for exactly this and needs no code change.
- Isolate the hot key: handle the few hot keys separately — often the hot key’s rows on the small-ish side are few enough to broadcast even though the table is not — then union with the normal join of everything else.
- Salting on both sides, with the replication on whichever side is smaller.
- Iterative broadcast (Fix 7): break the larger side into chunks that each fit a broadcast, join each, union the results.
- Change the layout (Fix 6): bucket or sort both tables on the join key so the shuffle disappears on every future run. The most work, and the only one that fixes it permanently.
21. How can repartitioning help with data skew?
repartition(n) with no column shuffles rows round-robin, which produces an even
distribution regardless of key. That genuinely fixes the imbalance — and it is
usually useless before a join, because the join immediately re-shuffles by the
join key and recreates the original distribution.
Where it does help is when the imbalance came from the input layout rather than a key: one huge file, or a source whose partitions are uneven, feeding a job that is not keyed at all.
repartition(n, "key") does not help. It reproduces exactly the distribution that
is causing the problem, by construction.
22. Why doesn't simply increasing the number of partitions always solve data skew?
Because every row with the same key hashes to the same partition, whatever the
partition count is. Doubling spark.sql.shuffle.partitions halves the average
partition size and leaves the hot key’s partition exactly as it was — you have
added empty partitions, not divided the big one.
It also has a cost: more partitions means more tasks, more scheduling overhead and more small shuffle files. Raising the partition count measures both halves of that.
The exception is when there is no hot key and the problem is simply too few partitions for the cluster, which is a different problem wearing the same symptoms.
24. How would you handle data skew in a groupBy() aggregation?
Usually you do not have to — map-side combine already handles algebraic aggregations (question 9). When it does not:
- Two-phase aggregation, which is salting without the replication: aggregate
by
(key, salt), then aggregate those partials bykey. This is straightforward precisely because there is no second side. - Replace non-combinable aggregations.
count(distinct)becomesapprox_count_distinctif approximation is acceptable;collect_liston a hot key is often a design problem rather than a tuning one. - Filter placeholders first, since
nullis frequently the hot group.
Fix 5 shows the two-phase shape.
25. How would you handle a customer with millions of transactions while other customers have very few?
This is the canonical hot-key case, and the answer depends on what comes after the join.
If you are aggregating per customer, pre-aggregate before the join: reduce that customer’s millions of transactions to one row first, and the join never sees them (Fix 5).
If you need the joined rows themselves, isolate that customer: filter them out, join them separately — their side of the dimension table is one row, so it broadcasts — and union the result back (Fix 4). Salting works too, but it makes every other customer pay for the one.
If this is a recurring pipeline, the durable answer is layout: bucket the fact
table on customer_id so the shuffle stops happening at all
(Fix 6).
In production
29. How would you optimize a production pipeline suffering from severe data skew?
Treat it as a measurement problem before a tuning one:
- Instrument. Log the max-to-median partition ratio for the keyed stages, so skew is a metric with a history rather than an incident.
- Fix the data if you can. Placeholder keys deleted upstream, or a layout change at the source, fix it for every job that reads that table rather than for yours.
- Take the free fixes: confirm AQE is on and that the partitions actually clear the 256 MB threshold; broadcast whatever fits.
- Then change the job, cheapest first — filter, pre-aggregate, isolate, salt.
- Assert correctness in the job. Any fix that replicates or splits can change the answer; a row-count check is one line.
- Alert on the ratio, not the duration. Duration drifts for many reasons; a ratio that climbs from 3x to 30x is skew arriving, usually before anyone notices the job is slow.
30. Describe a real-world scenario where you identified and resolved data skew.
A useful answer has four parts, and the structure matters more than the anecdote:
- The symptom. “A nightly join sat at 199 of 200 tasks for 40 minutes; the cluster was idle.”
- The measurement. “Stage summary metrics showed a median shuffle read of
8 MB and a max of 4 GB. Counting by the join key, one
merchant_idheld 60% of the rows — it was the placeholder the upstream system wrote for card-present transactions.” - The fix and why that one. “Those rows were not joinable, so they were filtered before the shuffle rather than salted. Max-to-median went to 3x and the stage to 4 minutes.”
- What stopped it recurring. “A check on the share of the top key, alerting when it exceeds a threshold, plus the filter pushed upstream.”
The strongest version names a fix you rejected and why — the honest answer is usually that salting was considered and dropped because the hot key turned out to be garbage data, which is Fix 9.
Shuffle internals
Shuffle internals: which writer you get, what lands on disk, the reduce side and tuning. The answers link into the shuffle internals post, where each mechanism is measured.
What a shuffle is, and what it costs
1. What is a shuffle in Spark, and why does one exist?
A redistribution of rows across partitions so that all records needing to be
processed together end up in the same place. It exists because operations like a
join or a groupBy need every row with a given key on one machine, and the
partitioning the data arrived with says nothing about that key.
Structurally it is a disk barrier between two stages: map tasks write their output to local disk, the stage ends, and reduce tasks fetch what belongs to them. That barrier is why a shuffle is the expensive thing in most jobs.
2. Which operations cause a shuffle?
Anything that has to repartition by key: groupByKey, reduceByKey,
aggregateByKey, join, cogroup, distinct, sortBy, repartition, and
window functions with a PARTITION BY.
coalesce to a smaller count does not shuffle, which is exactly why it can
leave you with uneven partitions. map, filter and mapPartitions are narrow
and never shuffle. See which operations shuffle.
3. Where does the cost of a shuffle actually go?
Four places, and they are not equal: serializing records, writing them to local disk, transferring them over the network, and on the reduce side deserializing and often sorting or spilling them again.
The one people underweight is disk. Every shuffled record is written to local disk by the map side even when the consumer is on the same host — the write is not optional, because the reduce side may fetch it much later or after a retry.
4. What is the difference between a narrow and a wide dependency?
A narrow dependency means each parent partition feeds at most one child partition, so the work can be pipelined inside one stage with no data movement. A wide dependency means a child partition draws from many parents, which requires a shuffle.
Stage boundaries in Spark are exactly the wide dependencies. Counting Exchange
nodes in a plan is counting wide dependencies.
5. Why did sort-based shuffle replace hash-based shuffle?
File count. Hash shuffle wrote one file per map-and-reduce pair: M x R
files, so 1000 map tasks into 1000 partitions produced a million files and a
million open file handles’ worth of pressure.
Sort-based shuffle writes one data file per map task with the records ordered
by partition id, plus a small index of offsets — M files rather than M x R.
Collapsing that count is what let the shuffle scale. See
why hash shuffle lost.
Which writer you get
6. What are the three shuffle writers?
BypassMergeSortShuffleWriter, UnsafeShuffleWriter and SortShuffleWriter.
SortShuffleManager.registerShuffle picks one per shuffle, once, and the choice
is fixed for that shuffle’s life.
They correspond to three handle classes — BypassMergeSortShuffleHandle,
SerializedShuffleHandle and BaseShuffleHandle — which is what you look at if
you want to know which one you got. See
what decides which writer you get.
7. How does Spark decide between them?
One if/else if/else on three properties of the dependency:
- Bypass if there is no map-side combine and the partition count is at most
spark.shuffle.sort.bypassMergeThreshold(200). - Otherwise serialized (
UnsafeShuffleWriter) if the serializer supports relocation, there is no map-side combine, and there are at most 16777216 partitions. - Otherwise
SortShuffleWriter, the general path.
Note what is absent: the data size. The decision is structural — aggregation, partition count, serializer — not based on how much data there turns out to be.
8. How can you tell which writer a shuffle actually used?
Nothing in the Spark UI names it, but the handle is reachable from the dependency:
val dep = rdd.dependencies.head.asInstanceOf[ShuffleDependency[_, _, _]]
println(dep.shuffleHandle.getClass.getSimpleName)
Run against four shuffles that differ only in the three deciding properties, that prints each branch of the tree in turn:
| Shuffle | Handle |
|---|---|
200 partitions, no combine, (Int, Int) records |
BypassMergeSortShuffleHandle |
201 partitions, no combine, (Int, Int) records |
SerializedShuffleHandle |
201 partitions, no combine, (Int, (Int, Int)) records |
BaseShuffleHandle |
201 partitions, reduceByKey, (Int, Int) records |
BaseShuffleHandle |
Every row is one predicate changing. That is the decision tree, observed rather than described.
9. What exactly is the bypass threshold, and is the comparison inclusive?
spark.shuffle.sort.bypassMergeThreshold, default 200, and the comparison is
<=. So a shuffle with 200 partitions takes the bypass path and the same shuffle
with 201 does not — the boundary is exact, not approximate, and the table above
shows it flipping.
This matters more than it looks, because spark.sql.shuffle.partitions also
defaults to 200. A SQL shuffle at the default sits exactly on the boundary.
10. Why does the bypass writer have a threshold at all?
Because it opens one file and one buffer per partition, per map task, and skips sorting entirely by writing each record straight to its partition’s file, concatenating them at the end.
That is cheap for few partitions and expensive for many. At 2000 partitions each
map task would want 2000 open files and, at the 32k default for
spark.shuffle.file.buffer, around 62 MiB of buffers. The default of 200 is
roughly where that trade stops paying. See
what should you configure.
11. What does "serializer relocation" mean, and which serializers support it?
The serialized path sorts records without deserializing them — it moves already-serialized bytes around inside a buffer and expects them to still be readable afterwards. A serializer that writes stream-level state, such as a back-reference table, breaks under that.
Asked directly of the JVM:
| Serializer | supportsRelocationOfSerializedObjects |
|---|---|
JavaSerializer |
false |
KryoSerializer |
true |
So Java serialization disqualifies UnsafeShuffleWriter outright. The 16777216
partition ceiling on the same path comes from PackedRecordPointer, which packs
a partition id and a memory address into one 64-bit long, leaving the partition
id 24 bits — and 2^24 is exactly 16777216.
12. If the default serializer is Java, how does any RDD shuffle reach the fast path?
Because spark.serializer is not the whole answer. ShuffledRDD asks
SerializerManager for a serializer, and it auto-selects Kryo when both the
key and the value class tags are primitives, primitive arrays or String —
regardless of what spark.serializer says.
So (Int, Int) records get Kryo automatically and can reach
UnsafeShuffleWriter, while spark.serializer remains unset at its
JavaSerializer default. Setting spark.serializer to Kryo changes the
fallback for types the auto-pick rejects; it does not override the auto-pick
and it does not help when there is map-side aggregation.
13. Can changing a value type change which writer runs?
Yes, and it is the most surprising consequence of the above. Changing a value
from Int to (Int, Int) makes the value class tag a tuple rather than a
primitive, the auto-pick declines Kryo, the serializer falls back to Java, Java
cannot relocate — and the same job moves from SerializedShuffleHandle to
BaseShuffleHandle.
That is rows two and three of the table in question 8, and the only thing that
changed is a type. SortShuffleWriter then pays to deserialize and re-serialize
records when merging spills, which the serialized path avoids entirely.
What lands on disk
14. What files does a map task write?
Three, whichever writer ran: a .data file holding every partition’s bytes
concatenated, a .index file of partition offsets, and a .checksum file.
shuffle_0_0_0.data # the records
shuffle_0_0_0.index # numPartitions + 1 offsets, as 8-byte longs
shuffle_0_0_0.checksum.ADLER32 # one checksum per partition
The sizes are arithmetic rather than mystery: for 4 partitions the index is
(4 + 1) x 8 = 40 bytes and the checksum file 4 x 8 = 32 bytes. See
what lands on disk.
15. Why is the last component of shuffle_0_0_0 always zero?
It is IndexShuffleBlockResolver.NOOP_REDUCE_ID, a placeholder kept so the block
id keeps its three-part shape. Its constancy is the entire point of sort-based
shuffle: there is no per-reducer file, so there is no reducer id to put there.
Under hash shuffle that component was meaningful, and that is precisely why there
were M x R files.
16. How does a reduce task find its bytes without reading the whole file?
The index file holds one 8-byte offset per partition plus a final end-of-file offset. A reduce task reads the two offsets bracketing its partition and then reads exactly that byte range from the data file.
So locating a partition is two small reads of the index rather than a scan, which is what makes one-file-per-map-task workable.
17. How is a shuffle write committed, and why does that make speculation safe?
The map task writes data and index to temporary files, then calls
IndexShuffleBlockResolver.writeMetadataFileAndCommit, which renames them into
place under a lock. That rename is the commit.
Which is why speculative execution is safe here: two attempts at the same map task can both write their temporaries, whichever commits first wins, and the loser’s files are discarded. There is never a half-written file visible to a reader.
18. Does sort-based shuffle sort your data?
It sorts records by partition id, so each partition’s bytes are contiguous in the data file. It does not order records by key within a partition unless a key ordering was explicitly requested.
People assume a sortBy came free with the shuffle. It did not.
The reduce side
19. What does the reduce side actually do?
Asks the driver’s MapOutputTracker where the blocks for its partition live,
fetches those byte ranges from each map output — locally where it can, over the
network where it cannot — decompresses and deserializes them, and then
aggregates or sorts if the operation needs it.
spark.reducer.maxSizeInFlight, 48m by default, bounds how much fetched data one
reduce task holds at a time. See what the reduce side does.
20. What is a FetchFailedException, and why is it expensive?
A reduce task could not retrieve a block it needs. It is expensive because the recovery is not a task retry — the map output is gone, so Spark must re-run the map stage to regenerate it, and everything downstream waits.
spark.shuffle.io.maxRetries (3) and spark.shuffle.io.retryWait govern how hard
a fetch tries before giving up. On a busy or flaky cluster raising them converts
expensive stage recomputation into cheap waiting. See
what happens when a fetch fails.
21. Who serves shuffle files when the executor that wrote them is gone?
The external shuffle service, if it is enabled. It runs per node rather than per executor, so shuffle output survives the executor exiting — which is what makes dynamic allocation viable, since Spark can only release an idle executor if something else can still serve its files.
Without it, losing an executor means losing its map output and re-running the stage that produced it. See who serves the files.
22. What does push-based shuffle change?
Map tasks additionally push their blocks to remote shuffle services, which merge the fragments belonging to each reduce partition. The reduce side then reads a few large sequential merged blocks rather than thousands of small random ones.
The problem it targets is the many-small-reads pattern that hurts most at large scale, not correctness. See why hash shuffle lost.
23. What does adaptive execution change about a shuffle?
It cannot un-write a shuffle that already happened, but at the boundary it can coalesce many small post-shuffle partitions into fewer reasonable ones, split skewed partitions, and switch a sort-merge join to a broadcast now that it knows the real size.
The evidence is AQEShuffleRead in the final plan. See
what AQE changes.
Spill, memory and tuning
24. What is spilling, and when does it happen?
Writing the in-memory collection to disk because the sorter’s share of execution
memory is exhausted. ExternalSorter and ExternalAppendOnlyMap spill sorted
runs and merge them later.
It is a symptom of the partition being too large for the memory available to that
task, which is why raising the partition count is usually a better first move than
raising spark.memory.fraction. See
what does spilling actually cost.
25. Why are memoryBytesSpilled and diskBytesSpilled so different?
They measure different things. memoryBytesSpilled is the in-memory object
size of what was spilled; diskBytesSpilled is the serialized, compressed bytes
that actually reached disk. On the measured run the disk figure was about a tenth
of the memory one.
So a large memoryBytesSpilled is not “we wrote that much” — it is “objects of
that size left memory”. Compare the two rather than reading either alone.
26. Which shuffle metrics are worth reading?
Shuffle write and read bytes for volume; shuffle read records against bytes to spot pathological record sizes; spill (both kinds) for memory pressure; and the max-to-median gap across tasks for skew.
The ratio that most often explains a slow stage is not a shuffle metric at all — it is the largest task’s shuffle read against the median. See which numbers tell you something.
27. What is actually worth configuring?
Mostly one number, and it is not a shuffle setting: the partition count. It
decides which writer runs, how many files exist, how large each reduce partition
is, and whether tasks spill. For SQL that is spark.sql.shuffle.partitions, and
better still whatever AQE coalesces it to.
After that, in rough order of usefulness: bypassMergeThreshold if you understand
the file-count trade, spark.shuffle.file.buffer (32k) on large shuffles,
spark.reducer.maxSizeInFlight (48m) when fetch-bound, and
spark.shuffle.io.maxRetries on a flaky cluster. Leave compression and checksums
alone unless measuring — spark.io.compression.codec defaults to lz4.
28. How do you avoid a shuffle entirely?
- Broadcast the small side of a join — no exchange at all.
- Bucket both tables on the join key, so the data arrives pre-partitioned and the exchange disappears permanently.
- Combine before the shuffle:
reduceByKeyovergroupByKey, or a DataFrame aggregation that gets a map-side partial automatically. - Filter and project first, so less is shuffled even when a shuffle remains.
coalesceinstead ofrepartitionwhen only reducing partitions, since it merges without shuffling.
29. When is the shuffle not the problem?
When the stage is slow but shuffle read and write are small, spill is zero, and task durations are even. Then you are looking at a scan, a UDF, a skewed input rather than a skewed shuffle, or simply too little parallelism.
The check is cheap: compare shuffle bytes against input bytes. If the shuffle is a small fraction of what the job read, tuning it cannot help much. See when the shuffle is not your problem.
30. A stage is slow and you suspect the shuffle. What do you check, in order?
- Is it skew? Max-versus-median shuffle read across tasks. One oversized partition defeats everything else, and the fix is different — see the data skew post.
- Is it volume? Shuffle bytes against input bytes. If they are comparable, the work is to shuffle less, not to tune the shuffle.
- Is it spill? Both spill metrics non-zero means partitions too large for the memory available; raise the partition count first.
- Is it fetch failures?
FetchFailedExceptionin the logs means stage re-runs, and the answer is retries or the external shuffle service, not partition tuning. - Only then the writer. If the shuffle is large, has no aggregation, and your
record types are forcing
BaseShuffleHandle, making the types Kryo-friendly moves it onto the serialized path — check with the one-liner in question 8.
Performance optimization
Performance optimization: proving a change worked, reading less, shuffling less and everything else. The answers link into the performance optimization post, where each mechanism is measured.
Proving a change worked
1. How do you know a Spark optimization actually fired?
Read the plan, not the clock. df.explain("formatted") names the optimization
directly — PartitionFilters, PushedFilters, ReadSchema,
BroadcastHashJoin, AQEShuffleRead — and those attributes do not move when the
cluster is busy.
This matters because most Spark optimizations are conditional: pushdown needs the filter to be expressible in the format, a broadcast needs the build side to be small enough at planning time, partition pruning needs the filter to name the partition column. Each silently does nothing when its condition is not met, and a slow job looks identical either way. See how do you read the proof?
2. Why is a before-and-after timing comparison unreliable?
Because on a shared cluster the difference you measured is usually smaller than the variance between two runs of the unchanged job. The input was cached differently, the executors landed on different hosts, another job was competing for the same disks.
A plan attribute is binary and reproducible: the filter either appears under
PushedFilters or it does not. That is why every technique in this post is
justified by a plan line or a counted number rather than a duration.
3. What is the difference between explain(), explain("extended") and explain("formatted")?
explain() prints the physical plan only. explain("extended") prints all four
trees — parsed, analyzed, optimized logical, and physical — which is what you want
when you are asking why the optimizer did something. explain("formatted")
prints a compact tree plus a numbered details section listing each node’s
attributes, which is where PushedFilters and ReadSchema actually appear.
For tuning, "formatted" is the one to reach for, because the attributes are the
evidence.
4. What is isFinalPlan, and why can inspecting a plan mislead you?
With adaptive execution on, the plan printed before execution is a proposal.
Spark re-plans at shuffle boundaries using measured statistics, so what ran may
differ from what you printed. isFinalPlan=true marks the plan that actually
executed.
The trap is subtle: calling count() to force execution before inspecting builds
a different query — one whose plan is not the plan you were asking about — and
reports isFinalPlan=false. Run the real action, then read the plan.
5. Which plan attributes are worth knowing by name?
| Attribute | Means |
|---|---|
PartitionFilters |
partition pruning applied — whole directories never opened |
PushedFilters |
predicate pushed into the scan |
ReadSchema |
the columns actually decoded; the proof of column pruning |
BroadcastHashJoin / BroadcastExchange |
the small side was broadcast |
Exchange |
a shuffle; count them, and try to remove them |
AQEShuffleRead |
adaptive execution coalesced or split partitions at runtime |
BatchEvalPython |
a Python UDF, and a codegen boundary |
*(n) |
a whole-stage codegen stage; fewer distinct n is better |
A change that alters none of these did not do anything, however much the wall clock moved.
Reading less
6. What is partition pruning, and what turns it off?
Skipping whole directories because the filter names the partition column, so those files are never opened. It is the cheapest optimization available, because the data is not read at all.
It turns off when the filter does not reference the partition column directly —
wrapping it in a function, comparing it against a differently-typed literal, or
filtering on a derived column instead. The evidence is PartitionFilters in the
plan; if that list is empty, pruning did not happen. See
prune partitions.
7. What is predicate pushdown, and does it only work on Parquet?
Pushing the filter down into the scan so rows are eliminated as early as possible.
And no — PushedFilters appears for CSV and JSON too.
What differs is what the format can do with the pushed filter. Parquet holds min/max statistics per row group, so it can skip entire blocks without decoding them. A row format still parses every record and simply discards non-matching rows sooner. Same plan attribute, very different saving. See pushdown and pruning.
8. What does select * actually cost?
Column pruning. On the same query against the same data, an explicit projection
gave a ReadSchema of 2 columns and select * gave 5. On a columnar
format that is a direct multiple on bytes decoded.
It also makes every downstream optimization work harder and makes the job fragile
to upstream schema changes. ReadSchema in the plan is the proof.
9. How much does file format choice matter?
More than most settings. Converting the measured dataset from JSON to Parquet was 7.5x on read — and unlike a tuning flag, it applies to every job that reads that table afterwards.
Parquet gives columnar decoding, per-row-group statistics for skipping, and better compression on typed columns. See file format and compression.
10. Does compression codec choice matter, and what is splittability?
A file is splittable if Spark can start reading it from the middle, which is what lets one file be processed by many tasks. A gzipped CSV is not splittable, so a 10 GB file becomes exactly one task no matter how large the cluster.
Inside Parquet this is less of a concern, because compression is per column chunk and the file remains splittable at row-group boundaries. Snappy is the usual default — cheaper CPU, larger files; zstd trades the other way.
Shuffling less
11. Why is a shuffle the thing to attack?
Because it is the only operation that writes every row to disk and sends it over the network, and it forces a stage boundary. Everything inside a stage is pipelined in memory; everything between stages is materialised.
The plan shows them as Exchange nodes. Counting Exchange before and after a
change is one of the most useful tuning habits there is: removing a join’s
broadcast turned one BroadcastExchange into a SortMergeJoin with two
Exchange nodes.
12. When does Spark choose a broadcast join?
When one side’s estimated size at planning time is under
spark.sql.autoBroadcastJoinThreshold, 10 MB by default. The small side is sent
whole to every executor and the join happens locally, removing the shuffle
entirely.
Two things commonly stop it: the estimate being wrong because statistics are missing or stale, and the build side genuinely being too large. With adaptive execution on, Spark can also switch to a broadcast at runtime once it has measured the real size — which is one of the better arguments for leaving AQE alone. See broadcast joins.
13. What is the difference between a broadcast join and a broadcast variable?
A broadcast join is a join strategy the optimizer picks; you influence it with a
threshold or a hint, and the evidence is BroadcastHashJoin in the plan.
A broadcast variable is an API for shipping a read-only value to every executor once instead of once per task — a lookup map used inside a UDF, say. It does not make joins faster and it does not appear in the plan. They share a word and nothing else. See broadcast variables.
14. reduceByKey or groupByKey, and why does it matter?
reduceByKey combines on the map side before the shuffle; groupByKey ships
every row and combines after. Same answer, 11x the shuffle bytes measured.
The DataFrame API makes this mostly moot — groupBy().agg() gets a partial
aggregate automatically for algebraic aggregations — which is itself an argument
for the DataFrame API over RDD lambdas. See combine early.
15. repartition or coalesce?
repartition(n) shuffles and can increase or decrease the count, producing even
partitions. coalesce(n) only decreases, and avoids a shuffle by merging existing
partitions — which is why it can leave you with uneven ones.
The trap is coalesce(1) before a write: because it avoids the shuffle, it pushes
single-threaded execution back up into the stage that produces the data.
repartition(1) shuffles into one partition and leaves the upstream stage
parallel. They are not interchangeable. See
repartition versus coalesce.
16. What do partitioning and bucketing do, and when is bucketing worth it?
Partitioning splits data into directories by column value, which enables partition pruning at read time. Bucketing pre-hashes rows into a fixed number of files by a key, so a join on that key needs no shuffle at all — the exchange disappears permanently rather than being tuned.
Bucketing costs a write-time shuffle and locks in the bucket count, so it pays off for a table joined the same way repeatedly, not for ad-hoc analysis. See partitioning and bucketing.
Adaptive query execution
17. What does AQE actually change, and what does it leave alone?
It re-plans at shuffle boundaries using statistics it has just measured, rather than estimated beforehand. Three things: it coalesces small post-shuffle partitions, it switches a join to a broadcast when the measured size allows, and it splits skewed partitions.
What it does not do is change the logical plan — it will not push a filter you
did not write or reorder your joins. It is on by default
(spark.sql.adaptive.enabled is true). See
let adaptive execution finish.
18. If AQE coalesces partitions, does spark.sql.shuffle.partitions still matter?
It becomes an upper bound rather than a target. AQE cannot create more partitions
than the shuffle produced, so the setting still decides the ceiling; it merges
downward toward spark.sql.adaptive.advisoryPartitionSizeInBytes, 64 MB by
default.
So the modern advice inverts: set shuffle.partitions generously and let AQE
coalesce, rather than trying to guess the right number in advance. The evidence
is AQEShuffleRead coalesced in the final plan.
19. Why does AQE sometimes appear to do nothing about skew?
Because its skew rule requires two conditions, and the absolute one is easy to
miss. A partition must be larger than
spark.sql.adaptive.skewJoin.skewedPartitionFactor (5.0) times the median and
larger than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes
(256 MB).
A partition can be a hundred times the median and be left alone because it is under 256 MB — which is exactly the situation on laptop-scale test data, where people conclude the feature is broken. The data skew post covers this in detail.
20. Should you ever turn AQE off?
Rarely, and mostly for diagnosis — turning it off makes the pre-execution plan the plan that runs, which is convenient when you are trying to reason about the optimizer in isolation.
In production the argument for leaving it on is that it has information you do not: the real size of a shuffle it has already written. Guessing partition counts in advance is strictly worse than measuring them after the fact.
Boundaries and APIs
21. What does a Python UDF really cost?
More than “a bit slower”. It inserts BatchEvalPython into the plan and splits
whole-stage code generation around itself, because the generated JVM code cannot
contain Python. Measured on the same trivial query, the built-in version planned
as one codegen stage and the UDF version as three — identical on Spark
3.5.9 and 4.1.3.
On top of that, every row crosses a serialization boundary into a Python worker and back. So the cost is structural, not proportional: it does not shrink because your function is simple. See keep the work inside the JVM.
22. When is a UDF justified, and what should you try first?
Check pyspark.sql.functions first — the built-in library is much larger than
most people assume, and anything there stays inside the JVM and inside codegen.
If you genuinely need Python, prefer a pandas UDF (functionType of
SCALAR, GROUPED_MAP, and so on): it moves data in Arrow batches rather than
row by row, which removes most of the serialization cost though not the codegen
split. A plain row-at-a-time UDF should be the last resort.
23. RDD, DataFrame or Dataset?
DataFrame by default. It is the only one the Catalyst optimizer can see into: an
RDD map with a lambda is an opaque function, so no predicate pushdown, no column
pruning, no join reordering, no codegen.
Dataset gives compile-time type safety in Scala at the cost of some deserialization; typed lambdas on a Dataset are similarly opaque to the optimizer. RDDs remain right for genuinely unstructured work and fine control over partitioning. See which API.
24. What is whole-stage code generation?
Spark compiling a chain of operators into a single generated Java method, so rows move through filter, project and join in tight loops rather than via a virtual call per operator per row.
You can read it in the plan: stages are marked *(1), *(2) and so on. Fewer
distinct stage numbers means longer fused pipelines. Anything the generator cannot
express — a Python UDF being the common case — forces a boundary and a new stage.
Memory, caching and serialization
25. When does caching make a job slower?
When the DataFrame is read once. Caching costs the time to materialise it plus the memory it then occupies, and that memory comes out of the same pool execution wants. A read-once DataFrame is slower cached than not.
Cache when something is genuinely reused across multiple actions, and unpersist when done. See caching.
26. Which storage level should you cache in?
MEMORY_AND_DISK is the DataFrame default and usually right. The choice that
actually moves the needle is serialized versus deserialized: the deserialized form
held 3x the memory of the serialized one on the measured data.
Deserialized is faster to read and much larger. Serialized costs CPU on every access and fits far more in the same pool. On a memory-pressured job that trade is usually worth taking.
27. What is the default serializer, and should you change it?
spark.serializer is unset by default, which means JavaSerializer — on both
Spark 3.5.9 and 4.1.3. This is worth checking rather than assuming, because so
many cluster images and templates set Kryo in spark-defaults.conf that people
report Kryo as “the default”.
Kryo is faster and more compact and is worth enabling for RDD-heavy or
cache-heavy jobs. It matters less than people expect for DataFrame work, which
mostly uses Spark’s own Tungsten encoders rather than spark.serializer at all.
See serialization.
28. How is executor memory divided, and what causes OOM?
Broadly: a reserved slice, then a unified region split between execution (shuffle, sort, join buffers) and storage (cache), which borrow from each other, and outside the JVM heap the overhead region for off-heap and Python workers.
OOM usually comes from one of a few places rather than “not enough memory
overall”: a skewed task asked to hold far more than its share, a broadcast build
side larger than estimated, an unbounded aggregation like collect_list on a hot
key, or collect() pulling a large result into the driver. See
memory and GC.
Cluster, stragglers and method
29. Why doesn't speculative execution fix a straggler caused by skew?
Because speculation re-runs the same partition on another node. It is designed for stragglers caused by a slow or failing node — bad disk, noisy neighbour — where a fresh attempt elsewhere finishes sooner.
A skewed task is slow because of the data it was given, so the duplicate gets the same oversized partition and is equally slow. You have spent twice the resources for the same duration. Diagnose which kind you have before enabling it — see resources, locality and speculation.
30. You are handed a slow Spark job. What do you do, in order?
Work down the list, because each step reduces what the later ones have to be clever about:
- Measure skew first. It is the exception that jumps the queue — one oversized partition defeats everything below.
- Read less. Partition pruning, column pruning, columnar format. Usually a one-line change; JSON to Parquet was 7.5x on its own.
- Shuffle less. Broadcast what fits, combine before the shuffle, bucket if the join repeats. Nothing you tune inside a shuffle beats not having it.
- Let AQE size the shuffles you keep.
- Use the API the optimizer can see — DataFrame over RDD lambdas.
- Remove boundary crossings — built-ins over Python UDFs.
- Then representation — storage level, serializer, what you cache.
- Only then cluster settings — executor sizing, locality, speculation.
And at each step, confirm from the plan that the change did what you think. See in what order?
References
- Handling data skew in Spark for the measured fixes behind the skew questions
- Spark shuffle internals for the writers, the files and the reduce side
- Spark performance optimization for the tuning questions
- Top Apache Spark interview questions for the broader question bank
Trademarks
Apache Spark, Apache Parquet and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.
Found this useful?
These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.
Comments
Questions, corrections and reviews are welcome. Sign in with GitHub to post; comments are stored in this site's public GitHub Discussions.