All posts

What's new in Apache Spark 4.2: geospatial types, vector functions and a clearer SQL

Spark 4.2 new features, tested on 4.2.1: geospatial types, vector functions, QUALIFY, metric views, cursors and more. Each feature has a status, a source ticket, a sample with its printed output, and a plain note on what needs a flag or a connector.

29 min read Spark

Spark 4.2 puts three things that analytics and machine learning code used to hand-roll into the engine: spatial types, vector distance functions and a join that returns the K best matches. It also adds SQL for the clauses people kept rewriting, filtering on a window function, choosing where unqualified names resolve, and defining a metric once. This post is for analytics engineers, ML engineers and teams that plan change data capture. You will learn what nine features do, the syntax, what a real run printed, and which ones need a flag or a connector before they work. All four releases side by side are in the Spark 4.x features-by-release post.

Spark 4.2 on 14 Jul 2026: GEOMETRY and GEOGRAPHY, NEAREST BY, vector functions and tuple sketches; QUALIFY, SET PATH, metric views, time_bucket and cursors; the CHANGES clause, Auto CDC SCD Type 1, schema-evolving INSERT and DSv2 transactions; Arrow UDFs on by default, the rebuilt Web UI, Kubernetes executor resizing and Java 25 Everything in Spark 4.2.

Verified execution. Every sample in this post ran on Spark 4.2.1, on the apache/spark:4.2.1-python3 image, in November 2026. Each feature carries a Status line: Ran means the printed output is real, behind a flag names the setting, and needs a connector means the engine defines the syntax and a table format has to implement it. Where I could not run something, it is labelled listed and its limits are in Known limitations. The runnable files are in ranga-spark4-examples under spark-4.2/.

Before you start

What you need

Need Detail
Java 17. The runs used the image’s Java 17
Python, for PySpark 3.10 or newer for the 4.2 tarball. Python 3.9 fails with a TypeError in a Python worker
Spark The apache/spark:4.2.1-python3 image, or the 4.2.1 release tarball (529 MB)
A catalog with change data capture Only to see feature 8 succeed. Built-in tables do not support it

The smallest check that you are on the release you think you are on is 00_hello_world.sql. Run it first. It printed:

4.2.1 617618396087429b5627a86ce933bb0f5d6645cb

If it prints another version, an older SPARK_HOME in your shell is winning. Start with env -u SPARK_HOME.

Glossary

Term Meaning
SRID Spatial reference identifier: the number that names a coordinate system, such as 4326
WKB Well-known binary: the standard byte encoding of a geometry
Embedding A list of numbers that represents text, an image or a user, so similar things have similar lists
Window function A function such as row_number() computed across a group of rows without collapsing them
SQL path The ordered list of schemas Spark searches to resolve an unqualified name
Metric view A view that defines dimensions and measures once, in YAML
CDC Change data capture: a feed of inserts, updates and deletes from a table
Cursor A way for a SQL script to walk a query result one row at a time

Try this first

A three-step path if you are new to these features. None of it needs a connector.

  1. Confirm the version. Run the hello-world file above.
  2. Run one clause. Try QUALIFY on a table with a window function.
  3. Run one function. Compare two vectors with vector_cosine_similarity.

1. Geospatial types: GEOMETRY and GEOGRAPHY [Intermediate]

Since: 4.2.0 · Status: Ran · Source: SPARK-51658, SPARK-55261, SPARK-55260, SPARK-55790, SPARK-56771

Lineage: the same five spatial functions are listed on 4.2.1 and 4.3.0, so 4.3 adds nothing to the function list.

What it is

Two new column types hold spatial values (SPARK-51658). A GEOMETRY describes shapes in flat coordinates, and a GEOGRAPHY describes positions on the Earth. Each carries an SRID, a spatial reference identifier that names its coordinate system, so a value knows where it lives instead of being an anonymous BINARY. You build them from WKB, the standard well-known binary encoding, and they read and write Parquet (SPARK-55261, SPARK-55260).

Why it matters

This means you can keep spatial data typed in a table and in Parquet, instead of as opaque bytes that every reader has to interpret.

Under the hood, the SRID registry is built from PROJ 9.7.1 data (SPARK-55790), and the types are on by default in 4.2 (SPARK-56771); the same call on 4.1.3 fails with GEOSPATIAL_DISABLED. A GEOMETRY defaults to SRID 0 (unknown) and a GEOGRAPHY to 4326. Be clear about the limit: on 4.2.1 SHOW FUNCTIONS lists five spatial functions, st_asbinary, st_geogfromwkb, st_geomfromwkb, st_setsrid and st_srid, and 4.3.0 lists the same five. There are no distance or containment functions yet, so these types are a storage and interchange foundation, not a spatial query engine.

Sample code

Runnable file: 01_geospatial_types.sql.

-- a point as WKB: x = 1.0, y = 2.0
CREATE TEMP VIEW pt AS SELECT X'0101000000000000000000F03F0000000000000040' AS wkb;

SELECT typeof(ST_GeomFromWKB(wkb)) FROM pt;     -- geometry(0)
SELECT typeof(ST_GeogFromWKB(wkb)) FROM pt;     -- geography(4326)

-- give the geometry a coordinate system
SELECT ST_SRID(ST_SetSRID(ST_GeomFromWKB(wkb), 4326)) FROM pt;   -- 4326

When to use it

Use it when you store spatial values and want them typed in tables and in Parquet. Skip it when you need distance or containment; only five functions exist. Common beginner mistake: assuming a distance function exists, or leaving the SRID at 0 when the data has a coordinate system.

Visual

WKB bytes become a geometry with ST_GeomFromWKB or a geography with ST_GeogFromWKB, and ST_SetSRID gives the geometry SRID 4326 The same bytes become a geometry with SRID 0 or a geography with SRID 4326.

2. NEAREST BY top-K join [Intermediate]

Since: 4.2.0 · Status: Ran · Source: SPARK-56395

What it is

For each row on the left, a NEAREST BY join returns the K best rows on the right under a ranking expression (SPARK-56395). It is the join shape behind “closest store”, “most similar item” and fuzzy matching. Before it, you wrote a cross join, a window function that ranked the matches, and a filter on the rank. The syntax is { APPROX | EXACT } NEAREST [ k ] BY { DISTANCE | SIMILARITY } expr, and k defaults to 1.

Why it matters

This means you can ask “give me the two closest” in one clause that states its own intent.

Under the hood, evaluation is brute force today: the 4.2.1 documentation states that the implementation evaluates the full cross product of the two sides, so a large right side should be filtered first. APPROX lets the optimiser use a faster strategy once one exists. INNER and LEFT OUTER are the supported join types, and k ranges from 1 to 100000.

Sample code

Runnable file: 03_nearest_by_join.sql.

-- pts:  (q1, 1.0), (q2, 9.0)       base: (a, 0.0), (b, 2.0), (c, 8.0), (d, 20.0)
SELECT q, x, name, y
FROM pts JOIN base EXACT NEAREST 2 BY DISTANCE abs(x - y)
ORDER BY q, abs(x - y);
-- q1 1.0 a 0.0
-- q1 1.0 b 2.0
-- q2 9.0 c 8.0
-- q2 9.0 b 2.0

Before 4.2, the K nearest needed a cross join, a window to rank, and a filter:

SELECT q, x, name, y FROM (
  SELECT q, x, name, y, row_number() OVER (PARTITION BY q ORDER BY abs(x - y)) AS rn
  FROM pts CROSS JOIN base) WHERE rn <= 2 ORDER BY q, abs(x - y);
-- q1 1.0 a 0.0,  q1 1.0 b 2.0,  q2 9.0 c 8.0,  q2 9.0 b 2.0

It returns the same four rows, and it does the same cross-product work that the current NEAREST BY does.

When to use it

Use it when you want the K best matches per row without writing a cross join and a window. Skip it when the right side is large; evaluation is brute force, so filter it first. Common beginner mistake: forgetting to pre-filter the right side, which makes the work grow with left rows times right rows.

Visual

Two query points on a number line each joined to their two nearest base points, q1 to a and b, q2 to c and b Each query point keeps its two nearest base points.

3. Vector functions [Beginner]

Since: 4.2.0 · Status: Ran · Source: SPARK-54713, SPARK-55030, SPARK-55031

Lineage: 4.3 adds Scala and Python bindings for the vector functions (SPARK-57284).

What it is

An embedding is a list of numbers that represents a piece of text, an image or a user, so that similar things have similar lists. 4.2 adds SQL functions that compare these lists when they are stored as ARRAY<FLOAT> (SPARK-54713, SPARK-55030, SPARK-55031): vector_cosine_similarity, vector_l2_distance, vector_inner_product, vector_norm, vector_normalize, and the aggregates vector_avg and vector_sum.

Why it matters

This means you can rank documents against a query inside Spark SQL, with no UDF and no external service.

Under the hood, the functions compute exact results row by row, and they are not an approximate-nearest-neighbour index, so they suit moderate candidate sets. The arguments are strict about type: vector_norm(..., 2.0D) is rejected with DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE because the degree must be a FLOAT, so write 2.0F. Paired with the NEAREST BY join above, they give a top-K similarity search in plain SQL. vector_avg over [1,2] and [3,4] returned [2.0,3.0], and the join above matched q1 to d1 and d3, the two documents whose cosine similarity to [1,0] is highest.

Sample code

Runnable files: 08_vector_functions.sql and 11_vector_similarity_join.sql.

SELECT vector_cosine_similarity(ARRAY(1.0F, 0.0F), ARRAY(1.0F, 1.0F));   -- 0.70710677
SELECT vector_l2_distance(ARRAY(1.0F, 2.0F), ARRAY(4.0F, 6.0F));         -- 5.0

-- the two closest documents to each query, ranked by cosine similarity
-- queries(q, qe) and docs(doc, de) hold ARRAY<FLOAT> embeddings
SELECT q, doc
FROM queries JOIN docs EXACT NEAREST 2 BY SIMILARITY vector_cosine_similarity(qe, de)
ORDER BY q, doc;
-- q1 d1,  q1 d3,  q2 d2,  q2 d3

When to use it

Use it when you rank embeddings against a query inside SQL for a moderate candidate set. Skip it when you have millions of candidates and need an approximate-nearest-neighbour index. Common beginner mistake: passing 2.0D to vector_norm, which fails with DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE; the degree must be a FLOAT.

Visual

Four vector functions each take two small arrays and return a number: cosine similarity 0.7071, L2 distance 5.0, inner product 11.0 and norm 5.0 Each function takes ARRAY<FLOAT> arguments and returns a single number.

4. QUALIFY [Beginner]

Since: 4.2.0 · Status: Ran · Source: SPARK-31561

What it is

QUALIFY filters rows on the result of a window function (SPARK-31561). A window function, such as row_number() or rank() with an OVER clause, computes a value across a group of rows without collapsing them. Before QUALIFY, keeping “the top row per group” meant computing the window in a subquery and filtering it in the outer query.

Why it matters

This means you can write the top-N-per-group query as one statement, and queries from engines that already have QUALIFY port more easily.

Under the hood, the 4.2.1 documentation says the clause filters rows after the window functions have been evaluated, and that it can refer to a window function in the SELECT list by alias or define one directly in its condition. Aggregates are not allowed in the condition. Ties behave as the window function defines them: with rank(), every tied row qualifies, so a “top one” can return several rows. In the sample below, rank() = 1 returns ids 2, 3 and 5, because region eu has a tie at 30.0 between ids 3 and 5. Version note: QUALIFY is new in 4.2.0.

Sample code

Runnable file: 02_qualify.sql.

-- the highest-amount order per region
SELECT id, region, amount FROM orders_t
QUALIFY row_number() OVER (PARTITION BY region ORDER BY amount DESC, id) = 1
ORDER BY region;
-- 3  eu  30.0
-- 2  us  20.0

-- an alias works; with ties, every row ranked 1 qualifies
SELECT id, region, rank() OVER (PARTITION BY region ORDER BY amount DESC) AS r
FROM orders_t QUALIFY r = 1 ORDER BY id;
-- 2 us 1,  3 eu 1,  5 eu 1

Before 4.2, the top row per group needed a subquery around the window:

SELECT id, region, amount FROM (
  SELECT id, region, amount, row_number() OVER (PARTITION BY region ORDER BY amount DESC, id) AS rn
  FROM orders_t) WHERE rn = 1 ORDER BY region;
-- 3  eu  30.0
-- 2  us  20.0

It returns the same two rows. All three before/after pairs are in 12_before_after.sql.

When to use it

Use it when you want the top row per group in one statement. Skip it when the condition has no window function; an aggregate is not allowed there. Common beginner mistake: expecting rank() = 1 to return one row per group. Ties all qualify, so ids 3 and 5 both came back for eu.

Visual

Order of evaluation from FROM to LIMIT with QUALIFY after the window functions, and a row_number QUALIFY query returning the top row per region QUALIFY runs after the window functions, so it can filter on them directly.

5. SET PATH [Advanced]

Since: 4.2.0 · Status: Ran, behind a flag · Source: SPARK-54806

What it is

The SQL path is the ordered list of schemas Spark searches to resolve an unqualified table, view, function or variable name (SPARK-54806). SET PATH changes it for the session, current_path() returns it, and persistent views and SQL functions capture the path when they are created, so they resolve names the same way later. The default path is the system schemas followed by the current schema.

Why it matters

This means you can point a session at a reference schema or a sales schema without prefixing every table, and switch with one statement.

Under the hood, the feature is behind spark.sql.path.enabled, default false: with the flag off, SET PATH fails with UNSUPPORTED_FEATURE.SET_PATH_WHEN_DISABLED. Two schemas can each hold a table named facts, and the one that wins is decided by order in the path. DDL does not use the path: CREATE TABLE t always creates t in the current schema. Built-in names can also be qualified: system.builtin.upper('spark') returns SPARK on 4.2.1.

Sample code

Runnable file: 04_set_path.sql.

SET spark.sql.path.enabled = true;
-- ref.facts holds (2,'ref'); sales.facts holds (1,'sales')

SET PATH = spark_catalog.ref, DEFAULT_PATH;
SELECT current_path();  -- spark_catalog.ref,system.builtin,...
SELECT * FROM facts;    -- (2, 'ref')

SET PATH = spark_catalog.sales, DEFAULT_PATH;
SELECT * FROM facts;    -- (1, 'sales')

When to use it

Use it when a session should resolve unqualified names against a reference or sales schema. Skip it when explicit qualified names are clearer for the reader. Common beginner mistake: running SET PATH without spark.sql.path.enabled=true, which fails with SET_PATH_WHEN_DISABLED.

Visual

Two SET PATH settings put ref or sales first in the lookup order, so the unqualified name facts resolves to ref.facts or sales.facts The first schema in the path that holds facts wins.

6. Metric views [Intermediate]

Since: 4.2.0 · Status: Ran · Source: SPARK-54119

What it is

A metric view defines dimensions and measures once, in YAML, so every report computes the same number (SPARK-54119). A dimension is something you group by, such as a region. A measure is an aggregation, such as total amount or order count. You create the view with WITH METRICS LANGUAGE YAML, then query a measure with MEASURE(...) and group by a dimension.

Why it matters

This means two dashboards cannot disagree about what “revenue” means, because the definition lives in one object, not in two copies of a query.

Under the hood, the view stores the YAML, and a query refers to its measures by name through MEASURE(...). It works on the built-in catalog in 4.2. The 4.2.1 documentation has no page for this syntax, so the statement form here comes from the project’s own test suite, and the YAML layout shown is the one that ran. The YAML carries version: "0.1", a sign that the format is young, so expect it to change. Version note: metric views are new in 4.2.0.

Sample code

Runnable file: 05_metric_views.sql.

CREATE OR REPLACE VIEW sales.revenue_mv WITH METRICS LANGUAGE YAML AS $$
version: "0.1"
source: spark_catalog.sales.orders
dimensions:
  - name: region
    expr: region
measures:
  - name: total_amount
    expr: sum(amount)
  - name: order_count
    expr: count(1)
$$;
SELECT region, MEASURE(total_amount) AS total, MEASURE(order_count) AS n
FROM sales.revenue_mv GROUP BY region ORDER BY region;
-- eu  70.0  3
-- us  25.0  2

When to use it

Use it when one definition of a measure must serve many reports. Skip it when the query is a one-off. Common beginner mistake: writing the YAML from memory. The 4.2.1 documentation has no page for the syntax, so copy the layout shown.

Visual

A YAML definition with a region dimension and two measures becomes sales.revenue_mv, and a MEASURE query returns eu 70.0 over 3 orders and us 25.0 over 2 One YAML definition, queried with MEASURE.

7. time_bucket and IGNORE NULLS [Beginner]

Since: 4.2.0 · Status: Ran · Source: SPARK-56594, SPARK-55256, SPARK-55533

Lineage: 4.3 adds ASOF JOIN and BIN BY for more time-series SQL.

What it is

time_bucket(size, ts[, origin]) returns the start of the fixed-width bucket a timestamp falls into (SPARK-56594). That replaces the date_trunc or epoch-division arithmetic that everyone writes for fifteen-minute or six-hour windows. In the sample, 11:27 falls in the fifteen-minute bucket that starts at 11:15. A second change is smaller: collect_list, array_agg and collect_set now accept IGNORE NULLS and RESPECT NULLS (SPARK-55256, SPARK-55533), so you choose whether nulls appear in the collected array.

Why it matters

This means a group-by-window query reads as what it is, and null handling is stated instead of implied by a FILTER clause.

Under the hood, time_bucket takes an interval for the size and an optional origin that aligns the buckets, and it returns the bucket start as a timestamp, so you can group by it and join on it. Plain collect_list skips nulls, which I confirmed on 4.2.1 returns [1,3] for 1, NULL and 3, so IGNORE NULLS states the existing behaviour and RESPECT NULLS is the new capability: it returns [1,null,3].

Sample code

Runnable file: 06_time_bucket_ignore_nulls.sql.

SELECT time_bucket(INTERVAL '15' MINUTE, TIMESTAMP_NTZ '2026-11-01 11:27:00');
-- 2026-11-01 11:15:00

-- hourly counts and totals
SELECT time_bucket(INTERVAL '1' HOUR, ts) AS hour, count(*) AS n, sum(v) AS total
FROM (VALUES (TIMESTAMP_NTZ '2026-11-01 10:05:00', 1),
             (TIMESTAMP_NTZ '2026-11-01 10:40:00', 2),
             (TIMESTAMP_NTZ '2026-11-01 11:10:00', 5)) AS t(ts, v)
GROUP BY 1 ORDER BY 1;
-- 10:00  2  3
-- 11:00  1  5

Before 4.2, date_trunc covered calendar units only, so a 15-minute bucket needed epoch arithmetic. For an hour the two agree:

SELECT date_trunc('HOUR', TIMESTAMP_NTZ '2026-11-01 10:40:00');   -- 2026-11-01 10:00:00
SELECT time_bucket(INTERVAL '1' HOUR, TIMESTAMP_NTZ '2026-11-01 10:40:00');  -- 2026-11-01 10:00:00

When to use it

Use it when you group by fixed-width windows such as 15 minutes. Skip it when a calendar unit is enough; date_trunc already covers it. Common beginner mistake: assuming collect_list keeps nulls. Plain collect_list returns [1,3]; only RESPECT NULLS keeps them.

Visual

Three timestamps, 10:05 and 10:40 in the 10:00 bucket and 11:10 in the 11:00 bucket, with the count and total of each bucket Each timestamp maps to the start of its hour; the group-by then needs no arithmetic.

8. Change data capture: the CHANGES clause [Advanced]

Since: 4.2.0 · Status: Ran to the error only; needs a connector · Source: SPARK-55668, SPARK-55948, SPARK-55949, SPARK-55950, SPARK-56249

Lineage: Auto CDC with SCD Type 1 in 4.2, SCD Type 2 in 4.3 (SPARK-58321).

What it is

Change data capture (CDC) exposes what changed in a table between two versions, as rows describing each change. 4.2 adds a CHANGES clause to SQL (SPARK-55948) and matching DataFrame and PySpark APIs (SPARK-55949, SPARK-55950) under the umbrella SPARK-55668, plus Auto CDC in Declarative Pipelines for SCD Type 1, which keeps only the latest row per key (SPARK-56249). SCD means slowly changing dimension, the pattern for tracking how a dimension row evolves.

Why it matters

This means you can read a change feed with a query and feed it into a pipeline, without hand-written diff logic.

Under the hood, Spark defines the syntax and plans the query, and the catalog connector has to implement the change-feed API. A built-in table does not, so the clause fails cleanly: on a Parquet table in spark_catalog the statement returned UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE, “Catalog spark_catalog does not support Change Data Capture”. Treat CDC as a capability to ask of your table format. I did not run it against a lakehouse connector, so no success output is shown.

Sample code

Runnable file: 10_changes_clause.sql.

CREATE TABLE ev (id INT, v STRING) USING parquet;
INSERT INTO ev VALUES (1, 'a');

-- needs a catalog that implements change data capture
SELECT * FROM ev CHANGES FROM VERSION 1 TO VERSION 2;
-- [UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE] Catalog spark_catalog does not
-- support Change Data Capture (CDC).

When to use it

Use it when your table format implements change data capture and you want a change feed as a query. Skip it when the table is plain Parquet in spark_catalog, which does not support it. Common beginner mistake: running CHANGES on Parquet and expecting rows; it fails with UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE.

Visual

The CHANGES clause goes to Spark, which parses and plans it, then asks the catalog; a catalog connector that implements the CDC API supports it, and a built-in Parquet table in spark_catalog returns UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE Spark owns the syntax; the catalog owns the change feed.

9. Cursors in SQL scripts [Advanced]

Since: 4.2.0 · Status: Ran, behind two internal flags · Source: SPARK-54759

Lineage: CONTINUE HANDLER support arrived in 4.1 (SPARK-53621) behind an internal flag that is still off in 4.3.0.

What it is

A cursor lets a SQL script walk a query result one row at a time, with DECLARE ... CURSOR, OPEN, FETCH and CLOSE (SPARK-54759). SQL scripting, which Spark 4.1 enabled by default, lets a BEGIN ... END block declare variables, loop and handle errors. A handler is code that runs when a condition occurs. A CONTINUE handler for NOT FOUND runs when FETCH has no more rows and lets the script carry on, which is how a cursor loop knows to stop.

Why it matters

This means you can port row-by-row logic from another database’s procedural SQL.

Under the hood, both pieces are behind internal flags that default to false: spark.sql.scripting.cursorEnabled (since 4.2.0) and spark.sql.scripting.continueHandlerEnabled (since 4.1.0). Each missing flag fails differently, on 4.2.1 as on 4.3.0: with neither, UNSUPPORTED_FEATURE.SQL_CURSOR; with only the cursor flag, UNSUPPORTED_FEATURE.CONTINUE_EXCEPTION_HANDLER; with both, the script runs and returned 60. Because both flags are internal, treat cursors as unfinished and keep them out of production scripts. A set-based query is almost always the better answer.

Sample code

Runnable file: 09_sql_cursors.sql.

-- set spark.sql.scripting.cursorEnabled and ...continueHandlerEnabled to true first
BEGIN
  DECLARE total INT DEFAULT 0; DECLARE v INT; DECLARE done BOOLEAN DEFAULT false;
  DECLARE c CURSOR FOR SELECT col1 FROM VALUES (10), (20), (30);
  DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = true;
  OPEN c;
  FETCH c INTO v;
  WHILE NOT done DO
    SET total = total + v;
    FETCH c INTO v;
  END WHILE;
  CLOSE c;
  SELECT total AS cursor_total;
END;
-- 60

On 4.2.1 run the script through spark.sql(...) in Scala or Python. The 4.2.1 spark-sql -f splits a file at each ;, so a BEGIN ... END script fails with PARSE_SYNTAX_ERROR on its first line. The 4.3.0 CLI ran the same file.

When to use it

Use it when row-by-row logic cannot be expressed set-based and you are porting a procedure. Skip it when a set-based query will do, which is almost always. Common beginner mistake: setting only the cursor flag; the CONTINUE handler needs its own, and without it the script fails with CONTINUE_EXCEPTION_HANDLER.

Visual

A BEGIN END block declares a cursor over the values 10, 20 and 30, opens it, fetches in a WHILE loop with a CONTINUE handler for NOT FOUND, and returns the total 60 A loop over a cursor: the same BEGIN ... END shape as the 4.1 scripting examples, with a handler that ends it.

Smaller additions worth knowing

Feature What it does Status
Arrow-optimized Python UDFs on by default (SPARK-54555) spark.sql.execution.pythonUDF.arrow.enabled reads true on 4.2.1, so Python UDFs exchange data with the JVM through Arrow Default changed; test your own UDFs
SYSTEM.BUILTIN qualification (SPARK-54807) system.builtin.upper('spark') returned SPARK Ran
Top-K max_by and min_by (SPARK-55322) Return the K best values instead of one Ran
Schema-evolving INSERT and REPLACE ON (SPARK-54971, SPARK-56001) Engine syntax that a connector must implement Listed, needs a connector
Data Source V2 transactions (SPARK-55855) Commit several operations atomically Listed, needs a connector
Auto CDC, SCD Type 1 (SPARK-56249) Apply a change feed in a pipeline Listed, needs a connector
Tuple sketches (SPARK-54179) Native Apache Tuple sketches, with set operations (SPARK-55558) Listed
Web UI rebuilt on Bootstrap 5 (SPARK-55760) Dark mode, searchable and zoomable SQL plans Listed
Kubernetes executor resizing (SPARK-55555) Heterogeneous executors with resize plugins Listed
Java 25 (SPARK-51167) Build and runtime support for Java 25 Listed

Version lineage

Several 4.2 features start an arc that later releases continue. Read the 4.2 column as what this release added.

Feature 4.0 4.1 4.2 4.3
Geospatial types none none (GEOSPATIAL_DISABLED on 4.1.3) Introduced, on by default (SPARK-51658, SPARK-56771) Same five functions listed
Top-K and vector search none none NEAREST BY, vector functions (SPARK-56395, SPARK-54713) Scala and Python bindings for the vector functions (SPARK-57284)
Procedural SQL Session variables, SQL UDFs (SPARK-42849, SPARK-46057) SQL scripting generally available (SPARK-54499) Cursors, behind flags (SPARK-54759) none in the headlines
Window and time-series SQL none none QUALIFY, time_bucket (SPARK-31561, SPARK-56594) ASOF JOIN, BIN BY (SPARK-58092, SPARK-57133)
Names and views none none SET PATH, metric views (SPARK-54806, SPARK-54119) none
Change data and history none Declarative Pipelines (SPARK-51727) CHANGES clause, Auto CDC SCD Type 1 (SPARK-55668, SPARK-56249) Auto CDC SCD Type 2 (SPARK-58321)
Python UDFs Python UDTFs (SPARK-43797) Arrow UDF decorators (SPARK-52214) Arrow-optimized UDFs on by default (SPARK-54555) Experimental transpiling (SPARK-55206)

Flags at a glance

A feature that parses is not a feature that is on. Every feature in this post that is off by default:

Feature Config key Default What it enables
SET PATH spark.sql.path.enabled false The SET PATH statement
Cursors spark.sql.scripting.cursorEnabled false, internal, since 4.2.0 DECLARE CURSOR, OPEN, FETCH, CLOSE
CONTINUE handlers spark.sql.scripting.continueHandlerEnabled false, internal, since 4.1.0 The handler a cursor loop needs

Geospatial types, NEAREST BY, the vector functions, QUALIFY, metric views and time_bucket have no flag in 4.2.

Settings and costs

Setting Default What it does
spark.sql.execution.pythonUDF.arrow.enabled true (read on 4.2.1) Python UDFs exchange data through Arrow
Feature Cost to plan for
NEAREST BY The current implementation evaluates the full cross product of the two sides, so its work grows with left rows times right rows times the log of k. Pre-filter the right side
Vector functions Exact results row by row. They are not an approximate-nearest-neighbour index
Arrow-optimized Python UDFs Types, nulls and timestamps are where the new exchange path can differ from the old. Test your own UDFs
Metric views The 4.2.1 documentation has no page for the syntax, so you depend on the project’s tests and release notes
Change data capture The connector decides whether the feature exists for your table
Cursors Internal flags, so the behaviour can change without notice

When these features bite

Each item is a failure hit during the runs, with the text it produced, so you can grep for it. Every item is also an anchor, so you can link to one error.

[UNSUPPORTED_FEATURE.GEOSPATIAL_DISABLED] on 4.1.3. Geospatial types are off before 4.2. Fix: On by default from 4.2.

[UNSUPPORTED_FEATURE.SET_PATH_WHEN_DISABLED]. SET PATH is behind a flag. Fix: spark.sql.path.enabled=true.

[UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE]. The engine defines CHANGES, but the catalog, here spark_catalog, does not implement change data capture. The same shape applies to INSERT ... WITH SCHEMA EVOLUTION and Data Source V2 transactions. Fix: Check that your connector implements the feature before you plan around it.

[UNSUPPORTED_FEATURE.SQL_CURSOR], then [UNSUPPORTED_FEATURE.CONTINUE_EXCEPTION_HANDLER]. Cursors and CONTINUE handlers each have an internal flag in 4.2.1. Fix: Set spark.sql.scripting.cursorEnabled and spark.sql.scripting.continueHandlerEnabled, and keep cursors out of production.

[PARSE_SYNTAX_ERROR] on the first line of a BEGIN ... END script run with spark-sql -f. The 4.2.1 CLI splits the file at each ;, so a script is cut into pieces. Fix: Run the script through spark.sql(...) in Scala or Python. The 4.3.0 CLI ran the same file.

[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] on vector_norm(..., 2.0D). The degree must be a FLOAT. Fix: Write 2.0F.

[UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_NO_SCAN]. TABLESAMPLE SYSTEM needs a direct reference to a source that supports block-level sampling. It failed on a view and on a Parquet table, checked on 4.2.1. Fix: Use a source that supports it.

TypeError: unsupported operand type(s) ... 'type' and 'type' from a Python worker. The 4.2 tarball declares Python 3.10 or newer, and Python 3.9 cannot run it. Fix: Use a newer interpreter.

Run it yourself

The samples ran on apache/spark:4.2.1-python3 and on the 4.2.1 release tarball (529 MB, Scala 2.13.18) with Java 17. The tarball declares Python 3.10 or newer. The vector, cursor and CHANGES examples also ran on the 4.3.0 tarball with the same results. If you run a tarball locally, start it with env -u SPARK_HOME, because an older SPARK_HOME in the shell wins over the tarball’s own directory and silently runs the wrong Spark. The download script and the SQL runner are in RUNNING_ON_TARBALLS.md.

Hello world. 00_hello_world.sql printed 4.2.1 and the build hash. Before and after. 12_before_after.sql runs QUALIFY, NEAREST BY and time_bucket next to their older forms.

SQL. smoke/smoke42.sql printed this on 4.2.1:

geospatial         geometry(0)
QUALIFY            3 eu 30.0   /   2 us 20.0
NEAREST BY         q1 a   /   q2 c
SET PATH           current_path() = system.builtin,system.session,spark_catalog.default
time_bucket        2026-11-01T11:15
metric view        eu 70.0   /   us 25.0

Upgrade notes for teams moving to 4.2

Behaviour changes to test before you upgrade. The first rows come from the 4.2 release notes; the rest come from the official SQL and PySpark migration guides, which list more.

Change What to do Source
Python UDFs and UDTFs use Arrow by default, and spark.sql.execution.arrow.pyspark.enabled is true Run your UDFs. spark.sql.execution.pythonUDF.arrow.enabled=false and pythonUDTF.arrow.enabled=false restore the old path SPARK-54555
The minimum PyArrow is 18.0.0, and PyPy is no longer officially supported Check the Python environment PySpark migration guide
Order-independent shuffle checksums are on, and a mismatch rolls back and re-runs dependent stages spark.sql.shuffle.orderIndependentChecksum.enabled=false and ...enableFullRetryOnMismatch=false restore SQL migration guide
Pandas UDFs receive nullable integer columns as Int8 to Int64, not float64 Retest UDFs that handle nulls PySpark migration guide
option(None) is treated as unset, and Python data sources whose Arrow types miss the declared schema fail with DATA_SOURCE_RETURN_SCHEMA_MISMATCH Pass "" to set an empty string; fix the schema PySpark migration guide
NOT NULL is enforced on inserts into the built-in file sources A pipeline that wrote nulls into a NOT NULL column can now fail SPARK-55716
DROP TABLE can drop a view A script that relied on DROP TABLE failing for a view behaves differently SPARK-55019
The Postgres JDBC source has a default fetchSize Recheck large reads SPARK-56251
NATURAL JOIN honours spark.sql.caseSensitive; duplicate CTE names are rejected case-insensitively with DUPLICATED_CTE_NAMES Recheck join keys; rename CTEs SQL migration guide
A SQL UDF parameter no longer shadows a parameterless built-in such as current_date; SET CATALOG name resolves a session variable first spark.sql.legacy.allowUdfParameterToShadowParameterlessFunction=true, or rename; use SET CATALOG 'name' SQL migration guide
Observation.get raises the underlying error instead of returning an empty result Add error handling SQL migration guide
builtin.func() and session.func() try the system namespace first spark.sql.legacy.persistentCatalogFirst=true restores SQL migration guide
R 3.x is dropped, and per-write options take precedence over session configuration in file source writes Upgrade R; recheck write options SPARK-57767, SPARK-56414

The migration guide also lists GROUP BY GROUPING SETS (()) returning one row under 4.1 to 4.2, while the ticket (SPARK-57775) has fix version 4.3.0. See the 4.3 post and test on your version.

Adopt now or wait?

4.2.0 shipped on 14 July 2026 and 4.2.1 followed on 27 October 2026, so the first patch release is out. The versioning policy states that the 18-month maintenance window for 4.2.x is unchanged, which makes 4.2 the 4.x line with the longest stated window so far. As of 18 November 2026, Apache Iceberg and Apache Hudi publish no Spark 4.2 artifact, and the latest Delta Lake, 4.4.1, is built against Spark 4.2.0. My recommendation, a judgement and not a measurement: if your connectors allow it, adopt 4.2.1, not 4.2.0, and test your own Python UDFs first, because the Arrow default is the change most likely to surprise a pipeline.

Known limitations in this release

  • Change data capture. I did not run CHANGES or Auto CDC against a catalog that implements change data capture, so only the UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE failure on Parquet is shown.
  • Geospatial. Five functions exist, and none measures distance or containment, so the types are a storage foundation.
  • Search. NEAREST BY evaluates the full cross product and the vector functions are exact, not approximate, so neither replaces a vector index.
  • Metric views and cursors. The metric-view syntax comes from the project’s tests, not the documentation, and its YAML is version 0.1. Cursors need two internal flags and cannot be run through the 4.2.1 spark-sql -f.
  • Listed only. Schema-evolving INSERT, Data Source V2 transactions, Auto CDC, the Web UI rebuild and Java 25 are from the release notes, not run.

Cheat sheet

Feature Syntax Needs
Geospatial ST_SetSRID(ST_GeomFromWKB(wkb), 4326) Nothing
NEAREST BY a JOIN b EXACT NEAREST 2 BY DISTANCE abs(a.x - b.y) Nothing
Vector functions vector_cosine_similarity(ARRAY(1.0F, 0.0F), ARRAY(1.0F, 1.0F)) Nothing
QUALIFY ... QUALIFY row_number() OVER (PARTITION BY k ORDER BY v DESC) = 1 Nothing
SET PATH SET PATH = spark_catalog.ref, DEFAULT_PATH A flag
Metric view CREATE VIEW v WITH METRICS LANGUAGE YAML AS $$ ... $$, then MEASURE(m) Nothing
time_bucket time_bucket(INTERVAL '15' MINUTE, ts) Nothing
CHANGES SELECT * FROM t CHANGES FROM VERSION 1 TO VERSION 2 A connector with change data capture
Cursors DECLARE c CURSOR FOR ...; OPEN c; FETCH c INTO v; CLOSE c; Two internal flags

Key takeaways

  • Types: GEOMETRY and GEOGRAPHY are a storage foundation today, with five functions and no distance or containment yet.
  • Matching: NEAREST BY plus the vector functions give a top-K similarity search in SQL, with exact brute-force evaluation, so filter the right side first.
  • Everyday SQL: QUALIFY and time_bucket remove a subquery and some arithmetic from queries you already write.
  • Governance: SET PATH is off until you set its flag, and metric views keep a measure’s definition in one object.
  • Connectors: change data capture, schema-evolving inserts and transactions are engine syntax. Check your table format supports them before you plan around them.

Further reading

Trademarks

Apache Spark, Apache Parquet, Apache Arrow and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. The Apache Software Foundation does not endorse this post.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee

Comments

Questions, corrections and reviews are welcome. Sign in with GitHub to post; comments are stored in this site's public GitHub Discussions.