Every Apache Spark release, and the problem each one was built to solve

• Architecture: the four eras, and what moved between them
• The 1.x line: structure arrives
  • Spark 1.0, May 2014
  • Spark 1.1, September 2014
  • Spark 1.2, December 2014
  • Spark 1.3, March 2015
  • Spark 1.4, June 2015
  • Spark 1.5, September 2015
  • Spark 1.6, January 2016
• The 2.x line: unification and codegen
  • Spark 2.0, July 2016
  • Spark 2.1, December 2016
  • Spark 2.2, July 2017
  • Spark 2.3, February 2018
  • Spark 2.4, November 2018
• The 3.x line: the planner stops guessing
  • Spark 3.0, June 2020
  • Spark 3.1, March 2021
  • Spark 3.2, October 2021
  • Spark 3.3, June 2022
  • Spark 3.4, April 2023
  • Spark 3.5, September 2023
• The 4.x line: SQL becomes the surface again
  • Spark 4.0, May 2025
  • Spark 4.1, December 2025
  • Spark 4.2, July 2026
• Defaults that changed underneath you
• What was removed, and when
• Common misconceptions
• Choosing where to sit
• Frequently asked questions
• The mental model
• References
• Trademarks

TL;DR

• Spark’s centre of gravity has moved four times: from the RDD, to the DataFrame, to the query plan, to a protocol between client and driver. Every release makes more sense once you know which of those four eras it belongs to.
• The biggest upgrades are rarely the headline features. They are the default changes: sort shuffle, a 10 MB broadcast threshold, adaptive execution, bloom-filter joins, and ANSI mode all flipped on in different releases and silently changed what your existing jobs do.
• Some 1.x defaults have never moved. The broadcast-join threshold set in 1.2 is still 10485760b, and spark.sql.shuffle.partitions is still 200, both read from a 4.1.3 session.
• A single explain output on a current cluster shows three eras at once: whole-stage codegen markers from 2.0, the adaptive wrapper from 3.0, and coalescing from 3.2. Adaptive execution itself is older than any of them, first shipping in 1.6.
• The newest release ships a SQL surface ahead of its engine in places. The change-data-capture clause parses and analyses, then stops at the catalog, because the built-in one does not implement it.

You have inherited a Spark job. It is pinned to some version because that is what the platform team installed in 2021, and someone is now asking whether moving it forward is worth a quarter of engineering time. The release notes are no help: each one is a list of between 1,100 and 5,100 resolved tickets, sorted by component, with no indication of which three entries will change your job’s behaviour and which 3,000 you can ignore.

That is the question this post answers. Not “what is in each release”, which the project already documents, but which changes altered the execution model, which changed a default under you, and which deleted something you might still be using.

Everything below was checked against a running cluster rather than recalled. The version under test is 4.1.3 unless the text says otherwise, with a second cluster on 4.2.0 for the newest features. Five claims I was confident about turned out to be wrong when I ran them, and I have kept those corrections in the text where they are useful.

Architecture: the four eras, and what moved between them

Spark has had one durable architectural habit: each era moves the thing you program against one level further from the machine, and one level closer to a description of intent.

flowchart LR
  E1["<b>1.x</b><br/>the RDD era<br/><i>you describe<br/>the computation</i>"]
  E2["<b>2.x</b><br/>the DataFrame era<br/><i>you describe<br/>the data</i>"]
  E3["<b>3.x</b><br/>the plan era<br/><i>the engine revises<br/>its own decisions</i>"]
  E4["<b>4.x</b><br/>the protocol era<br/><i>the client is<br/>detached from the driver</i>"]
  E1 --> E2 --> E3 --> E4

Each move buys something and costs something:

The version-by-version sections follow that arc. If you only read one section, read Defaults that changed underneath you, because default changes cause more upgrade surprises than features do.

The 1.x line: structure arrives

The first line’s job was to turn a fast research engine into something an operations team would accept, and then to discover that the RDD was the wrong abstraction for most of the work people were doing with it.

Spark 1.0, May 2014

The release that made Spark deployable rather than merely usable.

Spark SQL arrived with SchemaRDD, not DataFrame. The name survived exactly three releases. Catalyst, introduced in the same release to choose execution plans and push predicates into Parquet, is still the optimizer in 4.2 and is covered in depth in Inside Spark’s Catalyst optimizer.

Spark 1.1, September 2014

Two defaults changed here. spark.io.compression.codec became snappy, though it has moved on since: the default in the 4.1.3 build I ran is lz4. And spark.broadcast.factory became TorrentBroadcastFactory, which is why broadcasting a large variable no longer saturates the driver’s network link.

The dynamic bytecode generation in this release is the seed of what becomes whole-stage codegen in 2.0. It compiled expressions; 2.0 compiled whole operator chains.

Spark 1.2, December 2014

The release that changed the most defaults in the shortest note.

The hash-to-sort shuffle change is the single most consequential default flip in Spark’s history. Hash shuffle opened one file per reduce partition per map task; at a few thousand partitions on each side that is millions of open files. Sort shuffle writes one file plus an index per map task. Everything about shuffle tuning after 1.2 assumes the sort writer, which is described in Apache Spark architecture.

This release also set spark.sql.autoBroadcastJoinThreshold to 10485760, raising it from 10000. That value has not changed since. On the 4.1.3 cluster I ran for this post:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("defaults").getOrCreate()
print(spark.conf.get("spark.sql.autoBroadcastJoinThreshold"))
# 10485760b
print(spark.conf.get("spark.sql.shuffle.partitions"))
# 200
spark.stop()

Eleven years and three major versions later, the broadcast threshold a 2014 release chose is still deciding whether your join shuffles. That is worth knowing before you conclude that a join strategy was picked by something sophisticated.

Spark 1.3, March 2015

The release where Spark stopped being an RDD engine.

The rename looks cosmetic and was not. A SchemaRDD was an RDD that happened to carry a schema. A DataFrame is a description of a computation over named, typed columns, which the engine is free to execute however it likes. That freedom is what every later optimisation spends.

The direct Kafka API deserves its own note. The receiver-based approach ran a long-lived receiver task that wrote to a write-ahead log for durability. The direct approach treats Kafka offsets as the checkpoint, so the log becomes unnecessary and delivery becomes exactly-once instead of at-least-once. This is the same reasoning Structured Streaming later generalises to every source.

This is the oldest API in the post that still runs unchanged today:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("df13").getOrCreate()

orders = spark.createDataFrame(
    [(1, "pune", 1200), (2, "hyderabad", 800), (3, "pune", 450)],
    "order_id INT, city STRING, amount INT",
)
orders.groupBy("city").sum("amount").orderBy("city").show()
# +---------+-----------+
# |     city|sum(amount)|
# +---------+-----------+
# |hyderabad|        800|
# |     pune|       1650|
# +---------+-----------+
spark.stop()

Spark 1.4, June 2015

Sort-merge join is the important entry. Until it existed, a shuffle join that did not fit in memory failed. After it, join size is bounded by disk instead of memory. The trade-off is explicit: sort-merge join pays a sort on both sides to remove the memory ceiling. Spark still makes exactly this choice today, and the decision procedure is walked through in Spark joins in depth.

Spark 1.5, September 2015

Tungsten’s main release. The theme is that the JVM’s object model is the bottleneck, so Spark should stop using it for data.

“Execution memory is explicitly accounted for, without relying on JVM GC” is the sentence that matters. Before Tungsten, a Spark executor’s memory behaviour was the JVM’s behaviour, and tuning meant tuning a garbage collector. After it, Spark tracks its own execution memory and spills deliberately. The memory model that grows out of this is covered in Spark memory management.

Spark 1.6, January 2016

Two things are worth flagging.

Unified memory management (SPARK-10000) replaced a fixed split between execution and cache memory with one pool and a borrowing rule. The eviction is deliberately asymmetric: execution can evict cached blocks, storage can never evict execution. A query that needs memory to finish beats a cache that is only an optimisation.

Adaptive query execution (SPARK-9858) is dated 1.6, not 3.0. The 1.6 version only picked the number of reducers for joins and aggregations. It was rebuilt from scratch for 3.0 and enabled by default in 3.2. Anyone who says AQE is new in 3.0 is describing the rewrite, not the idea.

The 2.x line: unification and codegen

If 1.x discovered that the DataFrame was the right abstraction, 2.x committed to it: one entry point, one type, one execution strategy, and streaming redefined as a query over an unbounded table.

Spark 2.0, July 2016

Whole-stage codegen is the defining change. Before it, each operator was a separate object pulling rows from the next through an iterator, which costs a virtual call per row per operator. After it, Spark compiles an entire chain of operators into one Java method with the loop fused, which the JIT can then treat like hand-written code. The release notes claim speedups of two to ten times for common operators.

You can see it in any plan today. The * markers and the number after them are codegen stage identifiers:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("codegen").getOrCreate()
df = (spark.range(0, 1_000_000)
        .selectExpr("id", "id % 7 AS bucket")
        .groupBy("bucket").count())
df.collect()
print(df._jdf.queryExecution().executedPlan().toString())
spark.stop()

On 4.1.3 that prints:

AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   ResultQueryStage 1
   +- *(2) HashAggregate(keys=[bucket#1L], functions=[count(1)], output=[bucket#1L, count#2L])
      +- AQEShuffleRead coalesced
         +- ShuffleQueryStage 0
            +- Exchange hashpartitioning(bucket#1L, 200), ENSURE_REQUIREMENTS, [plan_id=25]
               +- *(1) HashAggregate(keys=[bucket#1L], functions=[partial_count(1)], output=[bucket#1L, count#7L])
                  +- *(1) Project [(id#0L % 7) AS bucket#1L]
                     +- *(1) Range (0, 1000000, step=1, splits=11)
+- == Initial Plan ==
   HashAggregate(keys=[bucket#1L], functions=[count(1)], output=[bucket#1L, count#2L])
   +- Exchange hashpartitioning(bucket#1L, 200), ENSURE_REQUIREMENTS, [plan_id=15]
      +- HashAggregate(keys=[bucket#1L], functions=[partial_count(1)], output=[bucket#1L, count#7L])
         +- Project [(id#0L % 7) AS bucket#1L]
            +- Range (0, 1000000, step=1, splits=11)

Three eras are visible in that one output. *(1) and *(2) are 2.0’s whole-stage codegen: the range scan, the projection and the partial aggregate were compiled into a single generated method, and the final aggregate into another. AdaptiveSparkPlan with its Initial Plan and Final Plan sections is 3.0. AQEShuffleRead coalesced is 3.2 deciding at runtime that 200 shuffle partitions were too many for seven groups and reading them as fewer.

Note also what did not happen: the Exchange still says 200, because spark.sql.shuffle.partitions is still the 1.x default. AQE coalesced after the fact rather than planning a better number in the first place.

Spark 2.1, December 2016

Watermarks are the release’s real content. A streaming aggregation over event time has to keep state for every open window, and without a rule for closing them that state grows without bound. A watermark is that rule: a declaration that data more than a stated interval behind the maximum seen event time will not be waited for. It buys bounded state at the cost of dropping genuinely late records, and you choose where on that trade-off to sit.

Spark 2.2, July 2017

The cost-based optimizer is the one to be realistic about. It requires statistics you have to compute yourself with ANALYZE TABLE, and without them the rule-based path does the work. Nine years later that is still true, and the Catalyst post shows a join reordering where the rule-based ReorderJoin fired and CostBasedJoinReorder ran and changed nothing.

pip install pyspark belongs in the same list as the engine features. It is the moment PySpark stopped requiring a cluster distribution to try.

Spark 2.3, February 2018

A dense release.

Two entries deserve emphasis for opposite reasons.

Pandas UDFs (SPARK-22216) changed the economics of Python on Spark. A plain Python UDF serialises one row at a time to a Python worker and back. A pandas UDF ships an Arrow batch, runs vectorized pandas code over it, and ships a batch back. Same language, different order of magnitude of overhead. Everything Spark has done for Python since, through to the Arrow UDFs of 4.1, follows this line.

SPARK-23207 is a correctness fix, not a feature: certain combinations of shuffle and repartition could return wrong results. The DataFrame case was fixed here in 2.3; the RDD case followed in 2.4 as SPARK-23243. If you are on anything older and you repartition after a shuffle, that is your reason to move, and it outranks every feature in this post.

Spark 2.4, November 2018

The last 2.x feature release, and the long-term home of an enormous amount of production Spark.

Higher-order functions are the feature most 2.4 users never adopted and should have. Before them, transforming an array column meant either exploding and regrouping, which is a shuffle, or writing a UDF, which leaves the engine. After them it is an expression the optimizer understands:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("hof").getOrCreate()
spark.sql("SELECT transform(array(1, 2, 3), x -> x * 2) AS doubled").show()
# +---------+
# |  doubled|
# +---------+
# |[2, 4, 6]|
# +---------+
spark.stop()

The 2 GB block limit (SPARK-24296) is the other entry worth knowing, because it explains a class of failure that simply stops happening after 2.4. Spark’s network layer used a single ByteBuffer per block, and ByteBuffer is indexed by int. Any shuffle block or cached partition over 2 GB failed with an overflow. The fix is unglamorous and removes a hard ceiling that used to dictate partition counts.

The 3.x line: the planner stops guessing

The 2.x engine made one plan from estimates and executed it. The estimates were frequently wrong, because nobody knows the selectivity of a filter over data nobody has read yet. The 3.x line’s theme is letting the engine revise itself, and separately, admitting that Python is the primary language.

Note the gap: 2.4.0 shipped in November 2018 and 3.0.0 in June 2020, nineteen months later. Major versions are where the project spends its compatibility budget, and it does not spend it often.

Spark 3.0, June 2020

Adaptive query execution re-plans after each shuffle stage completes, using the measured sizes of what was just written. It coalesces partitions that turned out small, splits ones that turned out skewed, converts a sort-merge join to a broadcast when the build side turns out to fit, and reads shuffle data locally when the join no longer needs it distributed. It is off by default in this release. What it does and does not revisit is the subject of Adaptive query execution.

Dynamic partition pruning solves a different problem. A star-schema query filtering a small dimension table cannot prune fact-table partitions at plan time, because the filter is on the other side of a join. DPP plans a subquery that computes the surviving dimension keys first, then uses them to prune the fact scan. A common query shape goes from reading every partition to reading a handful.

The calendar change (SPARK-26651) is the sleeper. Spark moved from a hybrid Julian-Gregorian calendar to the proleptic Gregorian one. Dates before 1582 can shift. If you store historical dates, test before upgrading rather than after.

The release notes state that Spark 3.0 is roughly twice as fast as 2.4 on a 30 TB TPC-DS benchmark. That is the project’s own measurement on its own hardware, quoted here as a claim rather than asserted as a result you will see. Your workload is not TPC-DS.

Spark 3.1, March 2021

There is no Spark 3.1.0. The line starts at 3.1.1; the release notes for 3.1.1 describe it as the second release of the 3.x line.

Kubernetes going GA three years after its experimental introduction in 2.3 is the clearest example of the project’s pace on deployment features.

SPARK-33275 opens an argument that runs across the rest of the 3.x line and is settled in 4.0. Silently returning null on overflow or a bad cast is convenient and hides data corruption. ANSI mode raises instead. In 3.1 it is opt-in.

Spark 3.2, October 2021

AQE on by default (SPARK-33679) is the most important line in the 3.x table. If you upgrade from 3.1 to 3.2 with no config changes, your query plans change. They usually improve. They do change.

The RocksDB state store is the fix for the other half of the streaming state problem. The default state store keeps state in the JVM heap, so a large stateful stream becomes a garbage collection problem. RocksDB moves it off-heap and onto local disk, trading some read latency for state that scales past what the heap can hold.

pandas API on Spark deserves an honest note on packaging: it needs pandas installed on the cluster, and the error if it is missing is explicit about the minimum version.

from pyspark.sql import SparkSession
import pyspark.pandas as ps

spark = SparkSession.builder.appName("pandas-api").getOrCreate()
psdf = ps.DataFrame({"city": ["pune", "pune", "hyderabad"], "amount": [10, 20, 30]})
print(psdf.groupby("city").amount.sum().sort_index().to_dict())
# {'hyderabad': 30, 'pune': 30}
spark.stop()

On a clean apache/spark image that fails with [PACKAGE_NOT_INSTALLED] Pandas >= 2.2.0 must be installed, which is the correct error and still a surprise the first time.

Spark 3.3, June 2022

Row-level runtime filtering extends the dynamic pruning idea from partitions to rows. Spark builds a bloom filter from the small side of a join and pushes it into the large-side scan, so rows that cannot possibly match are discarded before they are shuffled. It buys a smaller shuffle at the cost of building and broadcasting the filter, which is why it is gated on size thresholds.

The error class work (SPARK-38781) is the unglamorous change that improved daily life most. Spark errors acquired stable identifiers such as DIVIDE_BY_ZERO instead of free-text messages, which means you can match on them in code and search for them without pasting a stack trace into a search box.

Spark 3.4, April 2023

Spark Connect is the architectural change of the 3.x line, even though it arrives near the end of it. Until 3.4, a Spark application meant a driver JVM in your process: PySpark launched one and talked to it over a local socket, and your application’s lifetime was the driver’s lifetime. Connect replaces that with a gRPC protocol carrying unresolved logical plans. The client becomes thin and the driver becomes a server that many clients can share.

What it costs is a second API surface. For several releases, some things worked in a classic session and not in a Connect one, which produces confusing symptoms; two posts here work through that in practice, for Iceberg and for Hudi.

Lateral column aliases are the small feature people notice immediately, because every other SQL engine already had it:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("lca").getOrCreate()
spark.sql("SELECT 1 AS a, a + 1 AS b").show()
# +---+---+
# |  a|  b|
# +---+---+
# |  1|  2|
# +---+---+
spark.stop()

Before 3.4, referring to a in the same SELECT list that defined it was an error, and you repeated the expression or nested a subquery.

Spark 3.5, September 2023

3.5 is the long-term support release of the 3.x line in practice, and it is where most organisations that have not moved to 4.x are sitting. Its maintenance line is still active: 3.5.9 shipped in July 2026, after 4.2.0.

The 4.x line: SQL becomes the surface again

The 4.x theme is that after a decade of enriching the DataFrame API, the project turned back to SQL and started adding things that SQL engines have and Spark did not: variant data, collations, user-defined functions written in SQL, procedural scripting, and a pipeline syntax.

Spark 4.0, May 2025

The break. Read the removals first.

And the additions:

ANSI mode by default is the change that will break jobs, and it is the right change. On 4.1.3:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ansi").getOrCreate()
print(spark.conf.get("spark.sql.ansi.enabled"))
# true

try:
    spark.sql("SELECT 1/0").collect()
except Exception as e:
    print(type(e).__name__, str(e).split("\n")[0])
# ArithmeticException [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate ...

spark.sql("SELECT try_divide(1, 0) AS v").show()
# +----+
# |   v|
# +----+
# |NULL|
# +----+
spark.stop()

The same applies to casts: CAST('abc' AS INT) now raises CAST_INVALID_INPUT rather than returning null, and try_cast gives you the old behaviour where you actually want it. The migration is mechanical and the benefit is that silent null-poisoning of a column stops being possible by accident.

VARIANT is the answer to a question every lakehouse team has asked: how do you store JSON whose schema you do not control, without either flattening it or storing a string and paying to parse it on every read? VARIANT stores a parsed binary encoding with its own metadata dictionary.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("variant").getOrCreate()
spark.sql("""
  SELECT variant_get(parse_json('{"user":{"id":7,"city":"pune"}}'), '$.user.id', 'int') AS user_id
""").show()
# +-------+
# |user_id|
# +-------+
# |      7|
# +-------+
spark.stop()

Collations let a string column declare its own comparison rules, so case-insensitive matching stops requiring lower() on both sides and the optimizer keeps its ability to use the column:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("collation").getOrCreate()
spark.sql("SELECT 'Spark' COLLATE UTF8_LCASE = 'spark' AS eq").show()
# +----+
# |  eq|
# +----+
# |true|
# +----+
spark.stop()

SQL pipe syntax rewrites a query as a sequence of steps in execution order instead of a SELECT whose clauses run in an order that does not match how they are written. This is where my first guess was wrong. I wrote this:

FROM VALUES (1), (2), (3) AS t(x) |> WHERE x > 1 |> SELECT sum(x) AS s

and got [PIPE_OPERATOR_CONTAINS_AGGREGATE_FUNCTION]. Aggregation has its own pipe operator, and |> SELECT refuses to take an aggregate function. The correct form:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("pipe").getOrCreate()
spark.sql("FROM VALUES (1), (2), (3) AS t(x) |> WHERE x > 1 |> AGGREGATE sum(x) AS s").show()
# +---+
# |  s|
# +---+
# |  5|
# +---+
spark.stop()

SQL user-defined functions and session variables close a long-standing gap for teams whose logic lives in SQL rather than in Scala or Python:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("sqludf").getOrCreate()
spark.sql("DECLARE OR REPLACE cutoff INT DEFAULT 100")
spark.sql("SET VAR cutoff = 250")
spark.sql("""
  CREATE OR REPLACE TEMPORARY FUNCTION over_cutoff(amount INT)
  RETURNS BOOLEAN RETURN amount > cutoff
""")
spark.sql("SELECT over_cutoff(300) AS big, over_cutoff(100) AS small").show()
# +----+-----+
# | big|small|
# +----+-----+
# |true|false|
# +----+-----+
spark.stop()

Spark 4.1, December 2025

Declarative Pipelines is the largest addition. You declare datasets and the queries that produce them; Spark derives the execution graph, the dependency order, the parallelism, the checkpoints and the retries. It ships with a spark-pipelines CLI.

One practical note that cost me a few minutes, and is not in the release notes: the CLI is a Spark Connect client. On a stock apache/spark:4.1.3-python3 image it fails twice before it runs, first on PyYAML and then on grpcio:

ModuleNotFoundError: No module named 'yaml'
...
PySparkImportError: [PACKAGE_NOT_INSTALLED] grpcio >= 1.48.1 must be installed; however, it was not found.

pip install pyyaml grpcio fixes both. It is worth knowing that adopting Declarative Pipelines means adopting Spark Connect’s client dependencies, because that is an architectural commitment, not a packaging detail.

SQL scripting being on by default turns Spark SQL into a procedural language. Local variables, control flow and multi-statement blocks run inside the engine rather than in a Python wrapper around it:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("scripting").getOrCreate()
print(spark.conf.get("spark.sql.scripting.enabled"))
# true

spark.sql("""
BEGIN
  DECLARE total INT DEFAULT 0;
  SET total = (SELECT sum(x) FROM VALUES (1), (2), (3) AS t(x));
  SELECT total AS grand_total;
END
""").show()
# +-----------+
# |grand_total|
# +-----------+
# |          6|
# +-----------+
spark.stop()

Recursive CTEs arrived in the same release, which matters for anyone who has been faking a hierarchy traversal with a loop in Python:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("cte").getOrCreate()
spark.sql("""
  WITH RECURSIVE nums(n) AS (
    SELECT 1
    UNION ALL
    SELECT n + 1 FROM nums WHERE n < 5
  )
  SELECT collect_list(n) AS ns FROM nums
""").show()
# +---------------+
# |             ns|
# +---------------+
# |[1, 2, 3, 4, 5]|
# +---------------+
spark.stop()

The Arrow UDF decorator is the endpoint of the line that started with pandas UDFs in 2.3. It takes and returns PyArrow arrays directly, with no pandas conversion in the middle:

import pyarrow as pa
from pyspark.sql import SparkSession
from pyspark.sql.functions import arrow_udf

spark = SparkSession.builder.appName("arrowudf").getOrCreate()

@arrow_udf("long")
def plus_one(v: pa.Array) -> pa.Array:
    return pa.compute.add(v, 1)

spark.range(3).select(plus_one("id").alias("p")).show()
# +---+
# |  p|
# +---+
# |  1|
# |  2|
# |  3|
# +---+
spark.stop()

Spark 4.2, July 2026

The current release.

The Arrow default (SPARK-54555) is the change that affects everyone without being asked for. On 4.1.3, spark.sql.execution.arrow.pyspark.enabled is false; on 4.2.0 it is true, and spark.sql.execution.pythonUDF.arrow.enabled is true as well. Ordinary Python UDFs get the vectorized path without being rewritten.

Geospatial support is where I have to be precise, because the release note summary and the shipped function registry do not quite agree. The note says “ST_* functions, WKB/WKT and Parquet read/write”. What is actually registered as a built-in in 4.2.0 is five functions:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("geo").getOrCreate()
fns = [r[0] for r in spark.sql("SHOW FUNCTIONS").collect()]
print(sorted(f for f in fns if f.startswith("st_")))
# ['st_asbinary', 'st_geogfromwkb', 'st_geomfromwkb', 'st_setsrid', 'st_srid']
spark.stop()

There is no ST_Point, no ST_AsText, and no ST_Distance in the built-in registry, and there is no cast from a WKT string either: CAST('POINT(1 2)' AS GEOMETRY(4326)) fails with DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION. The entry point in 4.2.0 is well-known binary. What does work, and is genuinely new, is that the spatial reference identifier is part of the type rather than a column beside it:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("geo2").getOrCreate()

# WKB for POINT(73.8567 18.5204), little-endian
wkb = "0101000000ed9e3c2cd4765240a1d634ef38853240"

spark.sql(f"""
  SELECT typeof(st_geomfromwkb(unhex('{wkb}')))                  AS geom_type,
         typeof(st_geogfromwkb(unhex('{wkb}')))                  AS geog_type,
         st_srid(st_setsrid(st_geomfromwkb(unhex('{wkb}')), 4326)) AS srid
""").show(truncate=False)
# +-----------+---------------+----+
# |geom_type  |geog_type      |srid|
# +-----------+---------------+----+
# |geometry(0)|geography(4326)|4326|
# +-----------+---------------+----+
spark.stop()

geometry(0) means SRID 0, an unspecified reference system. GEOGRAPHY defaults to 4326, which is WGS 84. Note also that the bare type name does not parse: CAST(NULL AS GEOMETRY) is a syntax error while CAST(NULL AS GEOMETRY(4326)) is fine, because the SRID is part of the type’s grammar.

Change data capture is the other feature where running it is more informative than reading about it. The syntax is not the one I guessed; the grammar takes a version or timestamp range:

SELECT * FROM sales CHANGES FROM VERSION 0 TO VERSION 1

On a built-in Parquet table that parses and analyses, then stops:

[UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE] The feature is not supported:
Catalog spark_catalog does not support Change Data Capture (CDC). SQLSTATE: 0A000

That is the correct behaviour and it tells you exactly what the feature is. Spark 4.2 ships the SQL surface, the parser and the analyzer rules for CDC; the capability itself is delegated to the Data Source V2 catalog. You get change feeds when your table format implements them, which is the same division of labour Spark used for time travel.

NEAREST BY does work on built-in tables, and its grammar is worth stating because it is easy to guess wrong:

SELECT * FROM sales a JOIN sales b APPROX NEAREST 5 BY DISTANCE a.amount - b.amount

The shape is (APPROX | EXACT) NEAREST <k> BY (DISTANCE | SIMILARITY) <expr>, attached where a join condition would go. It gives nearest-neighbour search a planner-visible operator instead of a cross join with a sort, which is how people have been doing vector search on Spark.

Defaults that changed underneath you

Features are opt-in. Defaults are not. This table is the one to check before an upgrade, and every current value in it was read from a live 4.1.3 or 4.2.0 session rather than from documentation.

Three of these change results rather than performance. ANSI mode turns silent nulls into errors. The proleptic Gregorian calendar in 3.0 shifts pre-1582 dates. And the 2.3 fix for shuffle plus repartition (SPARK-23207) changed wrong answers into right ones, which is also a behaviour change if something downstream was tuned to the wrong ones.

What was removed, and when

Upgrades fail on removals more often than they fail on features.

SparkR was deprecated in 4.0 (SPARK-49347) and has not been removed: 4.2 still lists SparkR changes, including Java 25 support. Deprecated is not gone, but it is a clear signal about where to put new R work.

The Scala 2.12 removal in 4.0 is the one that breaks builds hardest, because it is not a Spark-only change. Every JVM dependency you bring, including every connector, needs a 2.13 artifact.

Common misconceptions

“Adaptive query execution is a Spark 3 feature.” The idea shipped in 1.6 (SPARK-9858) and only chose reducer counts. The implementation you know was written for 3.0 (SPARK-31412) and switched on by default in 3.2 (SPARK-33679). The distinction matters when reading old tuning advice, which may be about the 1.6 version.

“Structured Streaming arrived in 2.0.” It arrived as an experimental API in 2.0 and was declared generally available in 2.2 (SPARK-20844), two releases and a year later. Production advice written between those points is about a moving target.

“Kubernetes has been supported since 2.3.” Experimentally. The release note for 2.3 says configurations, container images and entrypoints were expected to change. It went GA in 3.1 (SPARK-33005).

“Spark 4 enables ANSI mode, so my job will fail loudly on bad data.” Only for the operations ANSI mode governs, which are arithmetic overflow, division by zero and invalid casts. A string that parses as a number but means something else still passes silently, as it should.

“spark.sql.shuffle.partitions does not matter any more, because AQE coalesces.” AQE coalesces partitions down after the shuffle has been written. The value still decides how many partitions get written in the first place, and it is still 200. The plan output earlier in this post shows exactly that: Exchange hashpartitioning(bucket#1L, 200) followed by AQEShuffleRead coalesced.

“Spark 4.2 ships geospatial functions, so I can port my PostGIS queries.” Five st_* functions are registered as built-ins in 4.2.0, and WKT is not among the entry points. Check SHOW FUNCTIONS against the queries you intend to port before planning the work.

Choosing where to sit

A short version of the advice, for the three positions most teams are actually in.

On 2.4. The correctness fix in 2.3 (SPARK-23207) is behind you, which is the good news. Everything else argues for moving: no adaptive execution, no dynamic partition pruning, no Kubernetes GA, and a Scala version no current connector ships for. The jump to 3.5 is large but well trodden.

On 3.1 to 3.3. The single biggest gain available to you is AQE by default in 3.2, and you can have most of it today by setting spark.sql.adaptive.enabled=true and measuring. Do that first; it tells you what the upgrade is worth before you spend it.

On 3.5. You are on the practical long-term-support release, and it is still being maintained: 3.5.9 shipped on 16 July 2026, two days after 4.2.0. Moving to 4.x costs you a Scala 2.13 rebuild, a JDK 17 or later runtime, and an ANSI-mode audit. Budget the ANSI audit seriously and treat it as a data-quality exercise rather than a compatibility chore, because every error it raises is a place your pipeline was silently producing nulls.

On 4.0 or 4.1. Moving within the 4.x line is ordinary. Watch for the Arrow defaults in 4.2 changing Python UDF behaviour, which is usually an improvement and is still a change worth a test run.

Frequently asked questions

Which Spark version should I move to from 2.4? 3.5 if you need a staged migration, because it keeps Scala 2.12 as an option and gets you adaptive execution, dynamic partition pruning and Kubernetes GA. 4.2 if you are willing to do the Scala 2.13 and JDK 17 work once rather than twice.

Is Spark 3.5 still maintained? Yes. 3.5.9 was released on 16 July 2026, two days after 4.2.0. The 3.5 line is the practical long-term-support release for organisations that have not yet moved to 4.x.

What actually breaks when I move to Spark 4? Four things, in descending order of how often they bite: Scala 2.12 artifacts that have no 2.13 build, a JDK older than 17, ANSI mode turning silent nulls into raised errors, and Mesos deployments having nowhere to go. The first two are build failures you find immediately. The third is a runtime failure you find in production unless you audit for it.

Do I have to use Spark Connect on Spark 4? No. The classic in-process driver is still the default, and spark-submit works as it always has. Connect is opt-in, with one caveat: some newer components are Connect clients themselves, including the spark-pipelines CLI for Declarative Pipelines, which needs grpcio installed even when your queries do not.

When did adaptive query execution actually become the default? 3.2, via SPARK-33679. It existed but was off by default in 3.0 and 3.1, and a much smaller version of the idea shipped in 1.6 as SPARK-9858. Tuning advice that predates 3.2 usually assumes it is off.

Has SparkR been removed? Not yet. It was deprecated in 4.0 (SPARK-49347) and is still present in 4.2, which added Java 25 support to it. Support for R 3.x was dropped in 4.2 (SPARK-57767), so an old R runtime will stop you before the deprecation does.

Why does the SQL pipe operator reject my aggregate function? Because |> SELECT does not accept aggregates. Use |> AGGREGATE instead. The error class is PIPE_OPERATOR_CONTAINS_AGGREGATE_FUNCTION and it names the fix.

The mental model

If you remember one thing, make it the shape rather than the list.

Spark spent its first line learning that the RDD hid too much from the engine, its second line replacing it with a schema the engine could compile, its third line admitting that plans made from estimates are wrong and building a mechanism to revise them, and its fourth line detaching the client from the driver and pouring the accumulated capability back into SQL.

Every feature in this post is an instance of one of those four moves. When the next release lands, that is the question to ask of each headline: which move is this, and does it change what my plans do or only what I am able to write?

References

• Spark release archive, the index every release note in this post was read from
• Spark 3.0.0 release notes for adaptive query execution, dynamic partition pruning and the TPC-DS claim quoted above
• Spark 4.0.0 release notes for ANSI by default, VARIANT, collations and the Scala, JDK and Mesos removals
• Spark 4.1.0 release notes for Declarative Pipelines, real-time mode and SQL scripting
• Spark 4.2.0 release notes for geospatial types, change data capture and the Arrow defaults
• SqlBaseParser.g4, the grammar that settled the CHANGES and NEAREST BY syntax when my first guesses failed to parse
• Spark SQL migration guide for the behaviour changes each version introduced
• Apache Spark architecture for the runtime the shuffle and scheduling changes act on
• Adaptive query execution for what the 3.x re-planning does and does not revisit

Trademarks

Apache Spark, Apache Hudi, Apache Iceberg, Apache Parquet, Apache Avro, Apache ORC, Apache Arrow, Apache Kafka, Apache Hadoop, Apache Mesos, Apache YuniKorn and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.