Apache Spark: the complete cheat sheet

• 1. Architecture: the execution model
• 2. spark-submit
• 3. Memory model
• 4. Reading and writing
• 5. Joins
• 6. Shuffle and partitioning
• 7. Adaptive query execution
• 8. Caching
• 9. Spark SQL and ANSI mode
• 10. Structured Streaming
• 11. Useful functions and patterns
• 12. Diagnosing a slow job
• 13. What changed in Spark 4
• Conclusion
• References

TL;DR

• A job splits into stages at shuffle boundaries, and a stage runs one task per partition. Almost every Spark performance question is really a question about partitions.
• ANSI mode is on by default from Spark 4.0. spark.sql.ansi.enabled defaults to true unless the SPARK_ANSI_SQL_MODE environment variable is set to false, and it turns silent nulls into runtime errors.
• Adaptive query execution has been on by default since 3.2, and it coalesces partitions, splits skew and re-picks join strategies at runtime using real statistics.
• Defaults worth memorising: spark.executor.memory and spark.driver.memory are 1g, spark.executor.cores is 1, spark.sql.shuffle.partitions is 200, and broadcast joins kick in under 10 MB.
• The advisory shuffle partition size AQE aims for is 64 MB, not the 200-partition figure, so spark.sql.shuffle.partitions matters much less than it used to.

This is the Spark companion to the Hudi and Iceberg cheat sheets, in the same shape: every row gives you the concept, a literal example, and the sentence that tells you whether you want it.

Written against the latest Spark release, 4.2.0 as of now. Every config key, default and since-version below was read from the v4.2.0 tag rather than recalled. Diagrams captioned Source: Apache Spark documentation come from docs/img in the Apache Spark source tree at that tag, © The Apache Software Foundation, used under the Apache License 2.0; the rest are my own. Where a default changed in Spark 4, it is called out, because those are the ones that break a job on upgrade.

1. Architecture: the execution model

Source: Apache Spark documentation.

flowchart LR
  subgraph ST1["stage 1: narrow, no shuffle"]
    direction LR
    R1["read trips"] --> F["filter, select"]
  end

  subgraph ST2["stage 2: after the shuffle"]
    direction LR
    AG["aggregate"] --> W["write, the action"]
  end

  F -->|"shuffle<br/>groupBy city_id"| AG

Everything between two shuffles is one stage, and a stage runs one task per partition. That is the whole scheduling model, and it is why “how many partitions” is the question behind most tuning.

Transformations are lazy and actions are eager. filter, select, join and withColumn add to a plan; count, collect, show and write execute it. That is why a typo in a filter often surfaces at the write, several lines later.

2. spark-submit

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --name daily_trips_rollup \
  --class com.example.TripsRollup \
  --num-executors 20 \
  --executor-cores 4 \
  --executor-memory 16g \
  --driver-memory 8g \
  --conf spark.sql.shuffle.partitions=400 \
  --conf spark.dynamicAllocation.enabled=true \
  --jars /opt/jars/postgresql-42.7.3.jar \
  --files /etc/app/log4j2.properties \
  /opt/apps/trips-rollup-2.4.1.jar --run-date 2026-09-15

On Kubernetes the same roles map onto pods, with the driver pod creating and owning the executor pods:

Source: Apache Spark documentation.

Note: Arguments after the application jar go to your main, not to Spark. Anything Spark-facing must come before the jar, which is the most common spark-submit mistake.

3. Memory model

flowchart TB
  C["container the cluster manager allocates"] --> H["spark.executor.memory<br/>JVM heap"]
  C --> O["spark.executor.memoryOverhead<br/>off-heap, 384m or 10 percent"]
  H --> RES["reserved<br/>300 MB"]
  H --> UP["unified pool<br/>memory.fraction 0.6"]
  H --> USR["user memory<br/>the remaining 0.4"]
  UP --> EX["execution<br/>shuffles, joins, sorts"]
  UP --> ST["storage<br/>cached blocks<br/>floor at storageFraction 0.5"]
  %% execution can evict storage down to the floor; storage can never evict execution
  EX -.->|"evicts"| ST

The container your cluster manager sees is spark.executor.memory plus overhead, so a 16g executor with the default factor asks for about 17.6g. When a job is killed for exceeding its container limit and the heap looked fine, the overhead is the first thing to check.

Execution memory (shuffles, joins, sorts) and storage memory (cached blocks) share one pool and borrow from each other. Execution can evict cached blocks down to the storage floor; storage can never evict execution. That is why caching a large DataFrame can quietly make a shuffle-heavy stage spill.

4. Reading and writing

# The single most useful read option for JDBC: parallelism
trips = (spark.read.format("jdbc")
    .option("url", "jdbc:postgresql://db.internal:5432/rides")
    .option("dbtable", "public.trips")
    .option("user", "etl")
    .option("partitionColumn", "trip_id")
    .option("lowerBound", "1")
    .option("upperBound", "40000000")
    .option("numPartitions", "16")
    .load())

Note: spark.sql.sources.partitionOverwriteMode defaults to STATIC, so INSERT OVERWRITE on a partitioned table replaces every partition, not just the ones in your DataFrame. Set it to dynamic for partition-scoped overwrites.

5. Joins

SELECT /*+ BROADCAST(c) */ t.trip_id, t.fare_amount, c.city_name
FROM trips t JOIN cities c ON t.city_id = c.city_id;

The join that hurts is the one where both sides are large and the key is skewed. AQE handles the common case automatically; see the skew settings below.

6. Shuffle and partitioning

The coalesce trap is worth spelling out: df.repartition(1000).coalesce(1) does not give you 1000-way parallelism then one file. Because coalesce avoids the shuffle, it pushes the narrow partition count up the DAG, and the upstream work runs with one task. Use repartition(1) when you genuinely want the shuffle.

7. Adaptive query execution

AQE re-optimises the plan mid-flight using statistics from completed stages, which is why it beats anything you can set by hand ahead of time.

flowchart LR
  P["logical plan"] --> ST1["run stage 1"]
  ST1 --> STATS["real statistics<br/>partition sizes, row counts"]
  STATS --> RE["re-optimise"]
  RE --> C1["coalesce small<br/>partitions"]
  RE --> C2["split skewed<br/>partitions"]
  RE --> C3["switch sort merge<br/>to broadcast"]
  C1 --> ST2["run stage 2"]
  C2 --> ST2
  C3 --> ST2

Both skew conditions must hold: a partition is treated as skewed only when it is larger than 5 times the median and larger than 256 MB. On a job whose partitions are all under 256 MB, skew handling never fires no matter how uneven they are, which is the usual reason “AQE skew join is on but nothing happened”.

8. Caching

Cache when a DataFrame is used more than once and producing it was expensive. Caching something read once makes the job slower, because you pay the write and give up memory that execution wanted.

9. Spark SQL and ANSI mode

The biggest behavioural change in the Spark 4 line is ANSI mode. spark.sql.ansi.enabled now defaults to true, and the default is computed as “true unless the SPARK_ANSI_SQL_MODE environment variable is set to false”.

Treat an ANSI failure on upgrade as a finding rather than a blocker: the error is usually pointing at data that was silently becoming NULL before. Reach for try_cast and try_divide at the specific expression before turning the flag off everywhere.

10. Structured Streaming

The model to hold in your head is that a stream is an unbounded table, with each arriving batch appended as new rows:

Source: Apache Spark documentation.

A query over that table produces a result table, recomputed incrementally each trigger, and the output mode decides which of its rows get written out:

Source: Apache Spark documentation.

(spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "kafka-broker1:9092")
    .option("subscribe", "trips")
    .option("startingOffsets", "latest")
    .load()
    .selectExpr("CAST(value AS STRING) AS payload", "timestamp AS event_time")
    .withWatermark("event_time", "10 minutes")
    .writeStream
    .format("parquet")
    .option("checkpointLocation", "s3a://lakehouse-prod/checkpoints/trips")
    .option("path", "s3a://lakehouse-prod/warehouse/trips_raw")
    .trigger(processingTime="1 minute")
    .outputMode("append")
    .start())

Event-time windows come in three shapes, and the choice changes how many windows a single record lands in:

Source: Apache Spark documentation.

The watermark is what lets Spark finalise a window and drop its state. Anything arriving behind the watermark is too late to be counted:

Source: Apache Spark documentation.

Without a watermark, a streaming aggregation keeps state forever and the job degrades over days rather than failing outright. The watermark is what lets Spark drop state it will never need again.

11. Useful functions and patterns

from pyspark.sql import Window
from pyspark.sql.functions import row_number, desc, col

# Latest row per trip, which is the shape most CDC dedup takes
latest = (trips
    .withColumn("rn", row_number().over(
        Window.partitionBy("trip_id").orderBy(desc("updated_at"))))
    .where(col("rn") == 1)
    .drop("rn"))

12. Diagnosing a slow job

The stages page is where skew and spill show up, because it gives you the task duration percentiles rather than just an average:

Source: Apache Spark documentation.

And the SQL tab lists each query with its duration and plan:

Source: Apache Spark documentation.

Opening one gives you the physical plan annotated with row counts per node, which answers “did my filter push down” and “which side got broadcast” directly:

Source: Apache Spark documentation.

The Spark UI SQL tab is the most useful page and the least used. It shows the physical plan with row counts per node, which answers “did my filter push down” and “which side got broadcast” directly, rather than by inference.

13. What changed in Spark 4

When upgrading from the 3.x line, test with spark.sql.ansi.enabled=true under your old runtime first. That isolates ANSI failures from everything else in the upgrade and turns one large change into two small ones.

Conclusion

Most Spark tuning reduces to three questions: how many partitions, how large, and is the data evenly spread across them. Executor sizing and memory fractions matter, but they matter because they decide how much work a single task can hold before it spills.

Adaptive query execution has absorbed most of what used to be manual tuning. It coalesces small partitions, splits skew and re-picks join strategies from real statistics, which means spark.sql.shuffle.partitions is far less load-bearing than the tuning guides of a few years ago suggest. Set it roughly and let AQE adjust.

The one thing to plan for deliberately on the 4.x line is ANSI mode. It is a good default and it will surface data problems that the 3.x line hid, so give it a dedicated test pass rather than meeting it during an upgrade.

References

• Spark configuration reference for every property, its default and the version it appeared in
• SQL performance tuning for adaptive query execution, join hints and partition coalescing
• Structured Streaming programming guide for triggers, watermarks and output modes
• ANSI compliance for the full list of behaviours the Spark 4 default changes
• Apache Hudi on Spark: the complete cheat sheet for the lakehouse layer on top of this