Spark performance tuning: seventeen techniques, and how to prove each one worked
Most Spark tuning advice is a list of settings with no way to tell whether any of them fired. These seventeen techniques each come with the code, the evidence that the optimization applied, and the failure mode that silently turns it off again, followed by thirty interview questions on tuning and proving it.
- Why is Spark tuning so hard to verify?
- The dataset these examples run against
- How do you read the proof?
- 1. Prune partitions before anything is opened
- 2. Push the filter down, and take only the columns you need
- 3. Let the small side be broadcast
- 4. Let adaptive execution finish the plan
- 5. Keep the work inside the JVM
- 6. Choose between repartition and coalesce deliberately
- 7. Know your skew before you tune for it
- 8. Combine early: the transformation you choose
- 9. Broadcast variables, and what they are not
- 10. File format and compression
- 11. Partitioning and bucketing
- 12. Which API: RDD, DataFrame or Dataset?
- 13. Serialization
- 14. Caching, and the form you cache in
- 15. Memory and garbage collection
- 16. Resources, locality and speculation
- 17. Regular expressions, more precisely
- In what order should you try these?
- Key takeaways
- References
- Trademarks
TL;DR
- Prove it from the plan, not the clock. A wall-clock comparison on a busy cluster cannot separate your change from the noise.
explain("formatted")names the optimization that fired:PartitionFilters,PushedFilters,ReadSchema,BroadcastHashJoin,AQEShuffleRead.select *costs you column pruning. The same filter read 2 columns with an explicit projection and 5 columns without one, from the identical query.- A join is broadcast or not depending on whether the build side fits
spark.sql.autoBroadcastJoinThreshold. Disabling it turned oneBroadcastExchangeinto aSortMergeJoinwith twoExchangenodes.- A Python UDF inserts
BatchEvalPythoninto the plan and splits whole-stage code generation around it. The built-in equivalent stays inside one generated stage.- Thirty interview questions on this topic are in Spark internals interview questions.
- Read the plan that actually ran. Calling
count()to inspect an adaptive plan builds a different query and reportsisFinalPlan=false, describing a plan that never executed.
Why is Spark tuning so hard to verify?
The usual tuning cycle is: change a setting, re-run the job, watch the clock, keep the change if the number went down. On a shared cluster that loop cannot tell you anything. The input was cached differently, the executors landed on different hosts, another job was competing for the same disks, and the difference you measured is smaller than the variance between two runs of the unchanged job.
Worse, most optimizations are conditional. Predicate pushdown applies only if the filter is expressible in the file format. A broadcast join happens only if the build side is small enough at planning time. Partition pruning needs the filter to reference the partition column directly. Each of these silently does nothing when its condition is not met, and a slow job looks exactly the same whether the optimization was skipped or applied.
So this post is organised around a different question. Not “is it faster?”, but “did the optimization actually fire?” That question has a definite answer, it is visible in the query plan, and it does not move when the cluster is busy.
Seventeen techniques follow. Each has the code, the evidence that it applied — a line in the plan, or a measured number — and the mistake that turns it off.
The dataset these examples run against
Everything below was run on Spark 4.1.3 in local[4] mode against two Parquet
tables written for the purpose. They are small enough to reproduce and shaped
like something real rather than an id, value toy.
orders = (spark.range(0, 4_000_000)
.withColumn("order_id", F.col("id"))
# 70% of rows land on 10 customers: a realistic heavy-hitter skew
.withColumn("customer_id",
F.when(F.rand(42) < 0.7, (F.rand(7) * 10).cast("int"))
.otherwise((F.rand(9) * 200000).cast("int")))
.withColumn("order_date",
F.date_add(F.lit("2026-01-01").cast("date"), (F.col("id") % 30).cast("int")))
.withColumn("amount", (F.rand(3) * 1000).cast("decimal(10,2)"))
.withColumn("status",
F.when(F.col("id") % 17 == 0, F.lit("CANCELLED")).otherwise(F.lit("SHIPPED")))
.withColumn("notes", F.concat(F.lit("note-"), F.col("id").cast("string")))
.drop("id"))
orders.write.mode("overwrite").partitionBy("order_date").parquet("/work/data/orders")
What that produced, measured rather than assumed:
| Table | Rows | Files | Bytes on disk |
|---|---|---|---|
orders |
4,000,000 | 120 across 30 date partitions | 66,125,993 |
customers |
200,000 | 4 | 812,232 |
The customers table being well under a megabyte matters later: it is why the
join in technique 3 is broadcast without anyone asking for it.
How do you read the proof?
One method underlies every technique here.
df.explain("formatted")
The formatted plan prints each operator with its own block of attributes, and five attribute names carry most of the signal:
| In the plan | What it proves |
|---|---|
PartitionFilters |
directory-level pruning; whole partitions never opened |
PushedFilters |
the filter was handed to the Parquet reader |
ReadSchema |
exactly which columns leave the scan |
BroadcastHashJoin / SortMergeJoin |
which join strategy was chosen |
AQEShuffleRead |
adaptive execution changed the shuffle at runtime |
Read the plan that ran, not a plan. For anything adaptive, the physical plan
printed before execution is a prediction. The plan that ran is available only
after the query has run, from the same QueryExecution object:
agg = orders.groupBy("status").agg(F.count("*").alias("n"))
rows = agg.collect() # executes THIS QueryExecution
print(agg._jdf.queryExecution().executedPlan().toString())
Using count() here instead of collect() is a trap worth naming, because it
looks equivalent and is not. count() builds a different query, so the plan
you then inspect is one that never executed. Doing exactly that while writing
this post produced AdaptiveSparkPlan isFinalPlan=false, describing a plan
Spark had discarded. The collect() version reports isFinalPlan=true.
1. Prune partitions before anything is opened
The cheapest data to process is data never read. When a table is partitioned on disk and the filter names the partition column, Spark resolves the filter against directory names and never opens the rest.
orders.filter(F.col("order_date") == "2026-01-05").select("order_id", "amount")
The proof is one line in the scan block:
(1) Scan parquet
Location: InMemoryFileIndex [file:/work/data/orders]
PartitionFilters: [isnotnull(order_date#5), (order_date#5 = 2026-01-05)]
ReadSchema: struct<order_id:bigint,amount:decimal(10,2)>
PartitionFilters carrying your predicate means 29 of the 30 date directories
were never listed or opened.
How it silently stops working. Wrap the partition column in a function and
the filter is no longer a partition filter, because Spark cannot invert the
function to decide which directories match. F.year(F.col("order_date")) == 2026
moves the predicate out of PartitionFilters and into a post-scan Filter, and
every partition gets read. Compare the partition column to a literal, and do the
arithmetic on the other side of the comparison.
2. Push the filter down, and take only the columns you need
Two separate optimizations live in the same scan block, and one of them is undone by a habit almost everyone has.
orders.filter(F.col("status") == "CANCELLED").select("order_id", "status")
PushedFilters: [IsNotNull(status), EqualTo(status,CANCELLED)]
ReadSchema: struct<order_id:bigint,status:string>
PushedFilters means the Parquet reader itself skips row groups whose statistics
rule out a match, so the rows never reach Spark. ReadSchema with two fields
means only two columns were decoded.
Now the same query without the projection:
orders.filter(F.col("status") == "CANCELLED")
PushedFilters: [IsNotNull(status), EqualTo(status,CANCELLED)]
ReadSchema: struct<order_id:bigint,customer_id:int,amount:decimal(10,2),status:string,notes:string>
The pushdown is unchanged. The ReadSchema went from 2 columns to 5, including
notes, the widest column in the table. Column pruning is driven by what you
project, and select * switches it off while leaving every other optimization
looking healthy.
This is the single cheapest change in this post: name your columns.
3. Let the small side be broadcast
A join between a large table and a small one does not need a shuffle. If the small side fits in memory, Spark ships a copy to every executor and joins locally.
orders.join(customers, "customer_id").select("order_id", "segment")
+- Project (8)
+- BroadcastHashJoin Inner BuildRight (7)
+- BroadcastExchange (6)
BroadcastHashJoin with BuildRight means customers was the side broadcast.
It qualified because its 812,232 bytes are comfortably under
spark.sql.autoBroadcastJoinThreshold, which defaults to 10 MiB.
Disabling the threshold on the identical query gives the contrast:
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
+- Project (11)
+- SortMergeJoin Inner (10)
: +- Exchange (4)
+- Exchange (8)
One BroadcastExchange became a SortMergeJoin with two Exchange nodes:
both sides of the join now shuffle. That is the cost the broadcast was avoiding,
and it is why this is usually the highest-value line in a plan to check.
How it silently stops working. The decision uses the estimated size at
planning time. If the small side is the result of earlier transformations rather
than a table read, the estimate can exceed the threshold even when the real
output is tiny, and the plan quietly falls back to a sort-merge join. The
estimate comes from statistics, so a table that has never had statistics
collected can be mis-sized in either direction. Where you are certain, say so
explicitly with F.broadcast(customers), which bypasses the estimate.
The full strategy-selection logic is its own subject, covered in Spark joins in depth.
4. Let adaptive execution finish the plan
spark.sql.shuffle.partitions is a number chosen before Spark knows how much
data there is. Adaptive query execution re-plans after each shuffle writes,
using the sizes it actually observed.
agg = orders.groupBy("status").agg(F.count("*").alias("n"))
agg.collect()
print(agg._jdf.queryExecution().executedPlan().toString())
With spark.sql.adaptive.enabled on:
AdaptiveSparkPlan isFinalPlan=true
+- *(2) HashAggregate(keys=[status#3], ...)
+- AQEShuffleRead coalesced
+- Exchange hashpartitioning(status#3, 200), ENSURE_REQUIREMENTS
And with it off, the same query:
*(2) HashAggregate(keys=[status#3], ...)
+- Exchange hashpartitioning(status#3, 200), ENSURE_REQUIREMENTS
AQEShuffleRead coalesced is the proof. The Exchange still says 200,
because that is what was configured and written; the coalescing happens on the
read side, where AQE merged those 200 partitions into far fewer tasks after
seeing how little data each held. The aggregate produces two rows, so 200 reduce
tasks were 198 tasks’ worth of scheduling overhead for nothing.
isFinalPlan=true is the other half of the proof, and it is the reason the
collect() discipline from earlier matters. Without it you are reading a
prediction.
The re-planning AQE performs, and what it declines to touch, is covered in adaptive query execution.
5. Keep the work inside the JVM
A Python UDF is not just slower arithmetic. It changes the shape of the plan.
upper_udf = F.udf(lambda s: s.upper() if s else None, StringType())
orders.select(upper_udf(F.col("status")).alias("s"))
* Project (5)
+- BatchEvalPython (4)
+- * Project (3)
+- * ColumnarToRow (2)
+- Scan parquet (1)
The built-in equivalent:
orders.select(F.upper(F.col("status")).alias("s"))
* Project (3)
+- * ColumnarToRow (2)
+- Scan parquet (1)
Two things changed. BatchEvalPython appeared, which is the operator that
serializes rows, ships them to a Python worker process, and reads results back.
And notice the asterisks: * marks an operator fused into a whole-stage
generated function. BatchEvalPython has no asterisk, so it splits the
generated stage in two, and the operators either side of it can no longer be
fused together.
The cost is therefore not only the Python interpreter. It is the serialization round trip per batch plus the loss of code generation across the boundary.
The rule. Reach for a built-in first; the pyspark.sql.functions catalogue
covers more than most people assume. Where a UDF is genuinely necessary, a
pandas UDF amortizes the boundary crossing over a whole batch instead of paying
it per row, and a Scala implementation avoids the process boundary altogether.
6. Choose between repartition and coalesce deliberately
These two look interchangeable and are not.
print("source partitions =", orders.rdd.getNumPartitions())
print("after repartition(50) =", orders.repartition(50).rdd.getNumPartitions())
print("after coalesce(2) =", orders.coalesce(2).rdd.getNumPartitions())
print("after repartition(customer) =", orders.repartition(50, "customer_id").rdd.getNumPartitions())
source partitions = 5
after repartition(50) = 50
after coalesce(2) = 2
after repartition(customer) = 50
Both reach a partition count, by different means. repartition shuffles and
can raise or lower the count, distributing rows evenly, and with a column
argument it co-locates rows sharing a key. coalesce does not shuffle; it
merges existing partitions into fewer, which is nearly free and is the right
tool immediately before a write to reduce output file count.
The trap is that coalesce reaches backwards. Because it introduces no
shuffle, the reduced parallelism applies to the stage that produces the data, not
just the write. coalesce(2) before a heavy aggregation does not aggregate with
full parallelism and then narrow; it runs the aggregation itself on two tasks.
When you want fewer output files without throttling the computation that makes
them, repartition is the one that costs a shuffle and keeps the upstream
parallelism.
7. Know your skew before you tune for it
Skew is the failure mode most often misdiagnosed as a memory problem, and it is measurable in one query.
top = orders.groupBy("customer_id").count().orderBy(F.desc("count")).limit(5).collect()
customer_id= 5 rows= 280751 share=7.02%
customer_id= 2 rows= 280675 share=7.02%
customer_id= 8 rows= 280332 share=7.01%
customer_id= 0 rows= 280332 share=7.01%
customer_id= 9 rows= 280304 share=7.01%
total rows = 4000000
Five keys holding 7% of the table each is a partition that will be roughly fourteen thousand times the size of an average one when there are 200,000 distinct keys. No memory setting fixes that; one task simply has more work than the others and the stage waits for it.
Measure this before reaching for a fix. If the distribution is flat, your slow stage is not skew and salting it will only add a shuffle. The fixes themselves are a subject of their own, in data skew in Apache Spark.
8. Combine early: the transformation you choose
Two ways to sum by key produce the same answer and move very different amounts of data.
val base = sc.parallelize(1 to 2000000, 8).map(i => (i % 1000, 1))
base.reduceByKey(_ + _).collect() // combines on the map side
base.groupByKey().mapValues(_.sum).collect() // ships every value
Measured shuffle write for each, from the stage metrics:
| Transformation | distinct keys | shuffle write bytes |
|---|---|---|
reduceByKey |
1000 | 35,680 |
groupByKey |
1000 | 424,472 |
Eleven times the shuffle traffic for an identical result. reduceByKey
performs map-side aggregation, so each partition sends one partial sum per key.
groupByKey sends every individual value and sums them on the reduce side.
The same distinction applies to aggregateByKey and combineByKey, which also
combine on the map side and additionally let the accumulator type differ from the
value type. The rule generalises: prefer any transformation that reduces before
the shuffle over one that reduces after it.
groupByKey is not always wrong. When you genuinely need every value for a key
together, such as collecting them into a list, there is nothing to combine. But
when the operation is associative, choosing it costs an order of magnitude.
In DataFrame code this choice mostly disappears, because groupBy().agg() plans
a partial aggregate before the exchange automatically. It is RDD code where the
decision is still yours.
9. Broadcast variables, and what they are not
A broadcast join and a broadcast variable are different things that share a name.
A broadcast join is a plan decision Spark makes about a DataFrame. A broadcast variable is something you create explicitly to ship read-only data to executors once, rather than once per task:
lookup = spark.sparkContext.broadcast(build_dict()) # shipped once per executor
rdd.map(lambda r: lookup.value[r.key])
Without it, a lambda that closes over build_dict() serializes that dictionary
into every task. A thousand tasks means a thousand copies. The broadcast
variable makes it one copy per executor, held until you call unpersist().
Use it for lookup tables, configuration and models that every task needs. Do not use it for anything large enough to strain the driver, which has to hold and serialize it first, and do not use it for data that changes: a broadcast variable is immutable once created.
10. File format and compression
The same 4,000,000 rows, written three ways:
| Format | Bytes on disk |
|---|---|
| Parquet (snappy) | 64,674,200 |
| CSV | 199,139,461 |
| JSON | 487,139,201 |
JSON is 7.5× the size of Parquet for identical data, because it repeats every field name on every row and stores numbers as text. CSV is 3.1×.
Compression codec, on the same Parquet data:
| Codec | Bytes |
|---|---|
| snappy (default) | 64,674,200 |
| gzip | 34,661,093 |
| zstd | 26,399,459 |
zstd produced a file 59% smaller than the default. The trade is CPU: snappy is the default because it decompresses fast, which matters when a query is scan-bound rather than storage-bound. zstd has become a reasonable default for cold data where storage and network dominate.
One result here corrects a common claim. PushedFilters appeared in the plan
for all three formats, not just Parquet:
parquet PushedFilters in plan=True
json PushedFilters in plan=True
csv PushedFilters in plan=True
So “only columnar formats support pushdown” is wrong as stated. What differs is what the pushdown can skip. Parquet stores per-row-group statistics, so a pushed filter lets the reader skip whole row groups without decoding them, and column pruning lets it skip columns entirely. CSV and JSON have no such structure: the reader still parses every record and the pushed filter only avoids materializing rows that do not match. The plan line looks identical and the saving is not.
11. Partitioning and bucketing
Partitioning splits a table into directories by column value, which is what technique 1 exploits. Bucketing splits it into a fixed number of files by a hash of a column, and the payoff is on joins.
(orders.write.mode("overwrite")
.bucketBy(8, "customer_id").sortBy("customer_id").saveAsTable("ob"))
(customers.write.mode("overwrite")
.bucketBy(8, "customer_id").sortBy("customer_id").saveAsTable("cb"))
Joining the two bucketed tables on the bucketing column, with broadcast disabled:
bucketed join Exchange count = 0
unbucketed join Exchange count = 2
Both shuffles disappear. Rows sharing a customer_id are already in
corresponding buckets on both sides, so the join reads them locally. Both
versions returned 4,000,000 rows.
That is the strongest structural optimization available for a join that runs repeatedly, and it comes with real constraints: both sides must be bucketed on the same column into the same number of buckets, the bucket count is fixed at write time, and changing it means rewriting the table. Bucketing is a decision about a table, not about a query.
Choose partition columns with low cardinality and high filter selectivity — a date, a region. Partitioning on a high-cardinality column produces a directory per value and a small-file problem worse than the one you started with.
12. Which API: RDD, DataFrame or Dataset?
The choice determines how much Spark can do for you.
DataFrames go through Catalyst. Every optimization in this post that shows up
in a plan — pushdown, pruning, join selection, adaptive re-planning — happens
because the engine understands the operations. Rows live in UnsafeRow, a binary
format the engine manipulates without deserializing into JVM objects.
RDDs are opaque. A map containing a lambda is a black box: Catalyst cannot
see inside it, so nothing is pushed down, no column is pruned, and records are
JVM objects with the memory cost that implies. The shuffle-internals measurement
is the sharpest illustration — the same logical data occupied roughly ten times
more as boxed objects than as serialized bytes.
Datasets sit between them. Typed operations give compile-time safety, and
relational operations still optimize. A typed lambda in map or filter,
however, is as opaque as an RDD’s, and additionally pays to convert UnsafeRow
into your case class and back.
The practical ordering: DataFrame or SQL by default; Dataset where type safety earns its cost; RDD only for what the higher APIs cannot express.
13. Serialization
Serialization decides how much data crosses the wire and how much space cached records occupy.
Java serialization writes class metadata and stream state with every object and is slow. Kryo writes a compact representation and is substantially smaller.
The subtlety, established in
Spark shuffle internals,
is that Spark already chooses Kryo automatically for shuffles whose key and
value are primitives, primitive arrays or String, regardless of
spark.serializer. Setting spark.serializer to KryoSerializer changes the
fallback used for everything else, and it is worth setting for RDD pipelines
carrying case classes or tuples.
Register your classes when you do:
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.kryo.registrationRequired", "false")
Registration lets Kryo write a small class id instead of the full class name on
every record. registrationRequired=true turns a missed registration into an
error rather than a silent size regression, which is worth doing once to find
what you missed.
None of this affects Spark SQL, which serializes UnsafeRow through its own
serializer and never consults spark.serializer.
14. Caching, and the form you cache in
Caching is worth doing when a DataFrame is read more than once, and it is a straight loss when it is read once.
The storage level decides the representation. The same RDD, cached two ways:
| Storage level | Memory used |
|---|---|
MEMORY_ONLY |
677,375,144 |
MEMORY_ONLY_SER |
214,715,517 |
Three times smaller serialized, for the same records. MEMORY_ONLY keeps
records as JVM objects, which is fast to read and expensive to hold;
MEMORY_ONLY_SER keeps them as bytes, trading CPU on every read for memory.
Two practical notes. Cached data competes with execution for the same unified
memory pool, so caching something large can push the work that reads it into
spilling — the interaction is covered in
Spark memory management.
And in PySpark this table does not apply the way it looks: Python RDDs are always
stored serialized, so MEMORY_ONLY is already the serialized form and
MEMORY_ONLY_SER does not exist in the Python API.
Call unpersist() when done. A cached DataFrame nobody unpersists holds memory
for the life of the application.
15. Memory and garbage collection
Most “out of memory” reports are not a shortage of memory but a distribution problem: one task received far more data than the others, which is skew, or the representation is far larger than the data, which is technique 14.
Garbage collection is worth looking at when GC time is a visible fraction of task time, which the executor page reports directly. The causes worth checking first are the ones already covered here: caching deserialized objects, RDD pipelines holding boxed records, and collecting large results to the driver. Each produces long-lived objects that are promoted and then collected expensively.
The region-by-region arithmetic of executor memory, and which pool a spill comes from, is its own subject in Spark memory management.
16. Resources, locality and speculation
Resource allocation. Prefer several medium executors over a few very large ones. A large heap makes garbage collection pauses longer, and an executor with many cores contends on the same memory pool. Executors in the range of four to five cores are a common starting point, with memory sized so that cores × per-task working set fits.
Data locality. Spark prefers to run a task where its data already is, and
spark.locality.wait controls how long the scheduler waits for a local slot
before giving up and running the task remotely. Raising it helps when the network
is the constraint and hurts when the cluster is busy, because tasks sit idle
waiting for a preferred node. On object storage, locality is largely moot — every
read is remote.
Speculative execution. With spark.speculation enabled, Spark launches a
duplicate copy of tasks that run far slower than their peers and takes whichever
finishes first. This is a real fix for a straggler caused by a degraded node, and
a waste for a straggler caused by skew, because the duplicate task has the same
oversized partition and will be equally slow. Diagnose which you have before
enabling it; the distinction is in
data skew in Apache Spark.
17. Regular expressions, more precisely
“Avoid regex” is common advice and too blunt to act on. What the implementation actually does:
Spark compiles the pattern once when the pattern expression is foldable — a literal — and caches it. When the pattern is not foldable, such as a pattern coming from another column, it is compiled per row.
df.filter(F.col("status").rlike("^CANC")) # literal: compiled once
df.filter(F.col("status").rlike(F.col("pat"))) # column: compiled per row
So a literal regex is not the disaster the advice implies. Two rules that do
hold: never put a column-valued pattern in a hot filter, and prefer a
simpler predicate when one expresses the same thing — startswith,
contains and like are cheaper than a regex and, on a Parquet source, a
simple equality or range predicate can additionally skip row groups, which no
regex can.
In what order should you try these?
Techniques are worth ranking, because they are not equally likely to matter.
- Read less. Partition pruning, column pruning and a columnar format change how many bytes enter the job at all, and they are usually a one-line change. JSON to Parquet alone was 7.5× on the measured data.
- Shuffle less. A broadcast that replaces a sort-merge join removes two exchanges; bucketing removes them permanently;
reduceByKeyinstead ofgroupByKeywas 11× less shuffle traffic. Nothing you tune inside a shuffle beats not having it. - Let AQE size the shuffles you keep. It is on by default and it is better at choosing partition counts after the fact than you are before it.
- Use the API the optimizer can see. A DataFrame gets every optimization above; an RDD lambda gets none of them.
- Remove boundary crossings. Replace Python UDFs with built-ins where they exist.
- Then representation. Serialization, storage level and what you cache decide how much memory the same data occupies — measured at 3× between two cache levels.
- Only then cluster settings. Executor sizing, locality and speculation are real, and they are the last place to look, not the first.
The ordering is not arbitrary: each step reduces the amount of data the later steps have to be clever about. Skew is the exception that jumps the queue — a single oversized partition defeats every item on this list, so measure it early.
Key takeaways
Optimization in Spark is mostly the art of not doing work: not opening partitions, not decoding columns, not shuffling rows, not leaving the JVM. Every technique here is one of those four, and each one leaves a specific line in the query plan when it applies.
So the habit worth building is not a list of settings. It is opening
explain("formatted") before and after a change and reading the five attributes
in the table above. A change that does not alter the plan did not do anything,
however much the wall clock moved.
References
- Spark SQL performance tuning for the configuration reference behind these techniques
- Spark SQL configuration for
autoBroadcastJoinThreshold,shuffle.partitionsand the adaptive settings pyspark.sql.functionsfor the built-ins worth checking before writing a UDF- Spark joins in depth for how a join strategy is chosen
- Adaptive query execution for what AQE re-plans and what it leaves alone
- Spark shuffle internals for what the exchanges you remove were costing
Trademarks
Apache Spark, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.
Found this useful?
These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.
Comments
Questions, corrections and reviews are welcome. Sign in with GitHub to post; comments are stored in this site's public GitHub Discussions.