All posts

What's new in Apache Spark 4.0: ANSI mode by default, VARIANT and a far more expressive SQL

Spark 4.0 new features, tested on 4.0.5: ANSI mode by default, VARIANT, pipe syntax, SQL UDFs, collation and more. Each feature has a status, a source ticket, a sample with its printed output, and a plain note on what to check before you upgrade from 3.x.

35 min read Spark

Spark 4.0 is a break with the past and a step forward in the same release. The baseline moves to Scala 2.13 and Java 17, ANSI mode becomes the default so bad data stops a job instead of turning into NULL, and SQL grows the pieces it was missing: a type for JSON, user-defined functions, variables, collations and a top-to-bottom query syntax. This post is for engineers who are meeting Spark 4 for the first time and for teams planning a migration from 3.x. You will learn what twelve features do, the syntax, what a real run printed, and what to check before you upgrade. All four releases side by side are in the Spark 4.x features-by-release post.

Spark 4.0 on 23 May 2025: Scala 2.13 and Java 17, ANSI mode, structured logging and the Kubernetes operator; VARIANT, pipe syntax, SQL UDFs, session variables, collation, GROUP BY on MAP and DESCRIBE AS JSON; the XML source, Python Data Source API, polymorphic UDTFs, parameter markers, Plotting API and EXECUTE IMMEDIATE; transformWithState, the state data source, the lightweight Python client and ML on Connect Everything in Spark 4.0.

Verified execution. Every sample in this post ran on Spark 4.0.5, on the apache/spark:4.0.5-python3 image, in November 2026. Each feature carries a Status line: Ran means the printed output is real, behind a flag names the setting, and needs a connector means the engine defines the syntax and a table format has to implement it. Where I could not run something, it is labelled listed and its limits are in Known limitations. The runnable files are in ranga-spark4-examples under spark-4.0/.

Scala 2.13 and Java 17 are themselves the baseline change: Scala 2.12 and JDK 8 and 11 are gone, covered in the release history post.

Before you start

What you need

Need Detail
Java 17, the baseline for 4.0. JDK 8 and 11 are no longer supported
Scala 2.13. Every jar your jobs depend on needs a Scala 2.13 build
Python, for PySpark 3.9 or newer for the 4.0 tarball, which is the lowest floor of the four releases
Spark The apache/spark:4.0.5-python3 image, or the 4.0.5 release tarball (523 MB)
pyarrow and pandas For features 8 and 10. The official image does not include them
protobuf and the gRPC packages For feature 10: pip install protobuf grpcio grpcio-status googleapis-common-protos
The RocksDB state store For feature 10: set spark.sql.streaming.stateStore.providerClass to RocksDBStateStoreProvider
plotly Only to try the plotting API in the smaller additions

The smallest check that you are on the release you think you are on is 00_hello_world.sql. Run it first. It printed:

4.0.5 b472a3aa3c4b71b5a6ff068ef396f757bd662e59

If it prints another version, an older SPARK_HOME in your shell is winning. Start with env -u SPARK_HOME.

Glossary

Term Meaning
ANSI mode Spark follows the SQL standard for errors: a bad cast or a divide by zero raises instead of returning NULL
try_ function A variant such as try_cast that returns NULL on bad input instead of raising
UDF User-defined function: a function you write yourself
UDTF User-defined table function: returns several rows per call
Collation A rule for how strings compare and sort, such as case-insensitive
Session variable A typed value that lives for the session, set with DECLARE and SET VAR
VARIANT The Spark type for semi-structured data such as JSON, encoded once in binary
rowTag The XML element that the XML reader treats as one row

Try this first

A three-step path if you are new to 4.0. None of it needs pyarrow.

  1. Confirm the version. Run the hello-world file above.
  2. See the default that changed. Run a bad CAST, then the same with try_cast.
  3. Run one query. Write a query top to bottom with pipe syntax.

1. ANSI mode on by default [Beginner]

Since: 4.0.0 · Status: Ran · Source: SPARK-44444

Lineage: ANSI mode is also on by default for the pandas API on Spark in 4.1 (SPARK-53295).

What it is

ANSI mode makes Spark follow the SQL standard for errors. Before 4.0, an invalid cast and a divide by zero both returned NULL, silently, and the job stayed green. With ANSI mode on, both fail loudly (SPARK-44444), and the error message names the function that tolerates the bad value. Data you used to lose without noticing now stops the job.

Why it matters

This means a typo in a numeric column becomes an error you can see, not a NULL that flows into a report.

Under the hood, the setting is spark.sql.ansi.enabled, now true. The error classes are CAST_INVALID_INPUT and DIVIDE_BY_ZERO, and each message points at the try_ variant, try_cast or try_divide, which returns NULL on purpose. This is the change most likely to turn a green pipeline red. Run your own data through it before the upgrade, and use the try_* functions where NULL was the intended answer. Setting spark.sql.ansi.enabled to false brings the old behaviour back: I confirmed on 4.0.5 that the bad cast and the divide by zero both return NULL again.

Sample code

Runnable file: 01_ansi_mode.sql.

SET spark.sql.ansi.enabled;          -- true

SELECT CAST('abc' AS INT);           -- CAST_INVALID_INPUT: use `try_cast` to tolerate it
SELECT try_cast('abc' AS INT);       -- NULL
SELECT 1/0;                          -- DIVIDE_BY_ZERO: use `try_divide` to tolerate it
SELECT try_divide(1, 0);             -- NULL

Before 4.0, with ANSI off, both mistakes returned NULL and the job stayed green. Setting the flag to false shows it:

SET spark.sql.ansi.enabled=false;
SELECT CAST('abc' AS INT), 1/0;   -- NULL  NULL
SET spark.sql.ansi.enabled=true;

Both forms are in 11_before_after.sql.

When to use it

Use it when you want bad data to stop a job instead of turning into NULL. Skip it when a pipeline depends on NULL for bad values; use try_cast and try_divide there, and spark.sql.ansi.enabled=false only as a stopgap. Common beginner mistake: upgrading without testing real data. This is the change most likely to turn a green pipeline red.

Visual

With ANSI off, a bad cast and a divide by zero both give NULL and the job stays green; with ANSI on they raise CAST_INVALID_INPUT and DIVIDE_BY_ZERO, and try_cast and try_divide return NULL on purpose Off: silent NULL. On: an error, with a try_ function for when you want NULL.

2. VARIANT [Intermediate]

Since: 4.0.0 · Status: Ran · Source: SPARK-45827, SPARK-49451

Lineage: VARIANT in 4.0, generally available with colon paths in 4.1 (SPARK-54454, SPARK-52494), editing functions in 4.3 (SPARK-57804).

What it is

VARIANT (SPARK-45827) holds a semi-structured value, such as JSON, in an encoded binary form. A query reads a path out of it without re-parsing text and without a fixed schema up front. The alternative was a STRING column and get_json_object on every read, which parses the whole text each time. You create one with parse_json, read it with variant_get, and inspect it with schema_of_variant.

Why it matters

This means you can land JSON whose shape changes and still query it by path, without writing a schema first.

Under the hood, parse_json encodes the text once, and every later read walks the binary structure. schema_of_variant reports the shape it found, such as OBJECT<a: BIGINT, b: ARRAY<BIGINT>>. Version note: this is the 4.0 form of the type. The v:field colon shorthand is not in 4.0: the same statement fails on 4.0.5 with PARSE_SYNTAX_ERROR ... at or near ':', and it works from 4.1, where VARIANT is also generally available. Editing functions arrive in 4.3. The 4.0 release also lets parse_json accept duplicate keys (SPARK-49451).

Sample code

Runnable file: 02_variant.sql.

-- encode the JSON text once
SELECT parse_json('{"a":1,"b":[1,2,3],"c":{"d":"x"}}') AS v;
-- {"a":1,"b":[1,2,3],"c":{"d":"x"}}

-- read a path and cast the result
SELECT variant_get(parse_json('{"a":1,"b":[1,2,3]}'), '$.b[1]', 'int');   -- 2

SELECT schema_of_variant(parse_json('{"a":1,"b":[1,2,3]}'));
-- OBJECT<a: BIGINT, b: ARRAY<BIGINT>>

Before 4.0, JSON lived in a STRING column and get_json_object returned a string every time:

SELECT typeof(get_json_object('{"a":1,"b":[1,2,3]}', '$.b[1]'));                       -- string
SELECT typeof(variant_get(parse_json('{"a":1,"b":[1,2,3]}'), '$.b[1]', 'int'));        -- int

When to use it

Use it when JSON arrives with a changing shape and you read fields by path. Skip it when the shape is fixed and typed columns would be simpler to filter and document. Common beginner mistake: expecting the v:field colon shorthand in 4.0. It fails with PARSE_SYNTAX_ERROR; use variant_get, or move to 4.1.

Visual

A STRING column re-parses the JSON text on every get_json_object read, while parse_json encodes it once into a VARIANT that variant_get reads by path A string is re-parsed on every read; a variant is encoded once.

3. SQL pipe syntax [Beginner]

Since: 4.0.0 · Status: Ran · Source: SPARK-49555

Lineage: 4.2 adds aggregate functions and GROUP BY in a pipe SELECT (SPARK-54292) and | as an alternative to |> (SPARK-51518).

What it is

Pipe syntax (SPARK-49555) lets a query read top to bottom, one step per |>, instead of inside-out. A normal query starts with SELECT, which names the result before you have said where it comes from. A piped query starts with FROM and then transforms the rows one step at a time: WHERE, AGGREGATE, EXTEND, SELECT, ORDER BY, LIMIT. It is the same logical plan, written in the order you think it.

Why it matters

This means a long query is easier to read, review and extend, because adding a step is adding a line, not wrapping what you had in a subquery.

Under the hood, it is a different way to write a query, not a different engine: for the first sample, EXPLAIN showed the same physical operators in the same order as the conventional SELECT ... GROUP BY ... ORDER BY form (a scan, a two-stage hash aggregate with an exchange, then an exchange and a sort), checked on 4.0.5. AGGREGATE carries the GROUP BY inside the step, and EXTEND adds a computed column while keeping the existing ones. The two samples below return the total per region above a threshold, and the first two rows with a doubled amount.

Sample code

Runnable file: 03_pipe_syntax.sql.

FROM orders
|> WHERE amount > 5
|> AGGREGATE sum(amount) AS total GROUP BY region
|> ORDER BY total DESC;
-- eu  40.0
-- us  20.0

FROM orders |> EXTEND amount * 2 AS doubled |> SELECT id, doubled |> LIMIT 2;
-- 1  20.0
-- 2  40.0

Before 4.0, the same query read inside-out, with the aggregation named before the filter that precedes it:

SELECT region, sum(amount) AS total FROM orders WHERE amount > 5
GROUP BY region ORDER BY total DESC;
-- eu  40.0
-- us  20.0

When to use it

Use it when a long query is easier to read top to bottom, one step per line. Skip it when the query is short and already clear. Common beginner mistake: expecting a performance change. For the first sample, EXPLAIN showed the same physical operators as the conventional form.

Visual

The same query written inside-out with a subquery and top to bottom with FROM, WHERE, AGGREGATE and ORDER BY steps joined by the pipe operator The same query, inside-out on the left and top to bottom on the right.

4. SQL user-defined functions [Intermediate]

Since: 4.0.0 · Status: Ran · Source: SPARK-46057

Lineage: 4.2 adds DESCRIBE FUNCTION for SQL UDFs (SPARK-56883) and a default collation for them (SPARK-55528).

What it is

A user-defined function (UDF) is a function you write yourself. In 4.0 you can write one in SQL (SPARK-46057), either scalar, returning one value, or table-valued, returning rows. You declare it with CREATE FUNCTION, give it a return type, and put the body after RETURN. Before this, a custom function meant Python, Scala or Java code and a deployed artifact.

Why it matters

This means analysts can package a calculation, such as a tax rule, without leaving SQL. Under the hood, the body is SQL, so the optimiser can see through it and inline it into the calling query, which a Python or Scala UDF does not allow, because its code is opaque to Spark. A temporary function lives for the session, and a persistent one is stored in a catalog. The table-valued form returns a TABLE (...) and is used in the FROM clause. Version note: 4.2 adds DESCRIBE FUNCTION for SQL UDFs and a default collation for them, which the 4.2 post lists.

Sample code

Runnable file: 04_sql_udf.sql.

-- a scalar function
CREATE TEMPORARY FUNCTION add_tax(x DOUBLE) RETURNS DOUBLE RETURN x * 1.18;
SELECT id, add_tax(amount) AS gross FROM orders;        -- 1  11.799999999999999 ...

-- a table-valued function
CREATE TEMPORARY FUNCTION big_orders(min_amt DOUBLE)
  RETURNS TABLE (id INT, amount DOUBLE)
  RETURN SELECT id, amount FROM orders WHERE amount >= min_amt;
SELECT * FROM big_orders(20.0);                          -- (2, 20.0), (3, 30.0)

When to use it

Use it when a reusable calculation can be written in SQL, so the optimiser can see through it. Skip it when the logic needs Python or Scala libraries. Common beginner mistake: forgetting RETURNS TABLE (...) for a table-valued function, and calling it in the FROM clause as if it were scalar.

Visual

A SQL UDF add_tax and a table-valued big_orders on the left, and on the right a session variable threshold that filters orders to ids 2 and 3 and then to id 3 after SET VAR A scalar and a table-valued SQL function, beside the session variable of the next section.

5. Session variables [Beginner]

Since: 4.0.0 · Status: Ran · Source: SPARK-42849

Lineage: SQL scripting, which builds on variables, is enabled by default in 4.1 (SPARK-54499); spark.sql.scripting.enabled read false on 4.0.5.

What it is

A session variable is a typed value that lives for the session (SPARK-42849). You create it with DECLARE, change it with SET VAR, and use it anywhere an expression is allowed. It is the typed alternative to splicing a value into SQL text, which is how most injection bugs and most quoting bugs begin.

Why it matters

This means you can state a threshold once and reuse it across several queries, then change it in one place.

Under the hood, the variable has a declared type, and an assignment is cast to it: SET VAR threshold = 'abc' failed with CAST_INVALID_INPUT on 4.0.5 because 'abc' cannot become a DOUBLE. DECLARE OR REPLACE makes a script safe to rerun. In the sample, amount > threshold returns ids 2 and 3 at 15.0 and only id 3 after SET VAR threshold = 25.0. Session variables are also the first building block of SQL scripting, which Spark 4.1 turns on by default, and of the parameter markers that 4.1 extends. See the 4.1 post for both.

Sample code

Runnable file: 05_session_variables.sql.

-- a typed, session-scoped variable
DECLARE OR REPLACE VARIABLE threshold DOUBLE DEFAULT 15.0;
SELECT id FROM orders WHERE amount > threshold;          -- 2, 3

-- change it once, and every later query sees the new value
SET VAR threshold = 25.0;
SELECT id FROM orders WHERE amount > threshold;          -- 3

When to use it

Use it when a threshold or setting is reused across several queries in one session. Skip it when the value must outlive the session. Common beginner mistake: assigning a value of the wrong type. An assignment is cast to the declared type, so 'abc' into a DOUBLE fails with CAST_INVALID_INPUT.

Visual

A SQL UDF add_tax and a table-valued big_orders on the left, and on the right a session variable threshold that filters orders to ids 2 and 3 and then to id 3 after SET VAR On the right: the variable threshold filters to ids 2 and 3, then to id 3 after SET VAR.

6. String collation [Intermediate]

Since: 4.0.0 · Status: Ran · Source: SPARK-46830

Lineage: schema-level collation in 4.1 (SPARK-52219), SHOW COLLATIONS in 4.2 (SPARK-49543).

What it is

A collation is a rule for how strings compare and sort: case-insensitive, accent-aware, or by a locale (SPARK-46830). Before collations, case-insensitive matching meant wrapping both sides in lower(), and has to be remembered on every query. With a collation, the comparison itself changes. UTF8_LCASE ignores case, and UNICODE_CI is a case-insensitive Unicode comparison.

Why it matters

This means 'Apple' = 'apple' can be true when you say so, and a count(DISTINCT ...) over names that differ only in case counts them once.

Under the hood, a column or an expression carries its collation as part of its type, and collation(expr) returns its name: x COLLATE UNICODE_CI reports SYSTEM.BUILTIN.UNICODE_CI. The default is still binary comparison, so nothing changes until you ask. In the sample, four names, Ana, ana, ANA and Bob, drop from four distinct values to two under UTF8_LCASE. Later releases extend this: 4.1 adds schema-level collation, and 4.2 adds a SHOW COLLATIONS command.

Sample code

Runnable file: 06_string_collation.sql.

SELECT 'Apple' = 'apple';                                -- false
SELECT 'Apple' COLLATE UTF8_LCASE = 'apple';             -- true
SELECT collation('x' COLLATE UNICODE_CI);                -- SYSTEM.BUILTIN.UNICODE_CI

-- values: 'Ana', 'ana', 'ANA', 'Bob'
SELECT count(DISTINCT n COLLATE UTF8_LCASE) FROM names;  -- 2

Before 4.0, case-insensitive matching meant wrapping both sides in lower():

SELECT lower('Apple') = lower('apple');         -- true
SELECT 'Apple' COLLATE UTF8_LCASE = 'apple';    -- true

When to use it

Use it when you compare or count strings case-insensitively. Skip it when binary comparison is what you want, which is the default. Common beginner mistake: assuming the default changed. Nothing changes until you add COLLATE.

Visual

Apple equals apple is false by default and true with UTF8_LCASE, and count distinct over Ana, ana, ANA and Bob drops from 4 to 2 The default compares bytes; UTF8_LCASE ignores case.

7. The built-in XML data source [Beginner]

Since: 4.0.0 · Status: Ran · Source: SPARK-44265, SPARK-44753

What it is

Spark 4.0 reads and writes XML without the old external package (SPARK-44265). Two things come with it. A data source, spark.read.format("xml"), with a rowTag option that names the element to treat as one row. And functions for XML held in a string column: from_xml parses it into a struct, to_xml writes one back, and schema_of_xml infers the schema.

Why it matters

This means XML feeds, such as vendor files or a legacy message queue, load as a DataFrame with no extra dependency to manage in every cluster.

Under the hood, the source infers a schema from the elements it finds, and the functions let you do the same inside a query on a column. In the samples, <p><name>Ana</name><age>30</age></p> becomes {name: Ana, age: 30}, and the inferred schema is STRUCT<age: BIGINT, name: STRING>, with the fields in alphabetical order in this run. A file with two <person> elements read with rowTag person returned the rows (30, 'Ana') and (41, 'Bob'). The 4.0 release notes list the XML reader and writer for PySpark and Spark Connect as well.

Sample code

Runnable file: 07_xml_functions.sql.

-- parse an XML string into a struct
SELECT from_xml('<p><name>Ana</name><age>30</age></p>', 'name STRING, age INT');
-- {name: Ana, age: 30}

SELECT schema_of_xml('<p><name>Ana</name><age>30</age></p>');
-- STRUCT<age: BIGINT, name: STRING>
# read files, one <person> element per row
df = spark.read.format("xml").option("rowTag", "person").load("/tmp/xmlin")
# age: long, name: string  ->  (30, 'Ana'), (41, 'Bob')

When to use it

Use it when you load XML feeds or parse XML held in a string column. Skip it when the data is already JSON or Parquet. Common beginner mistake: forgetting the rowTag option, which names the element that becomes one row.

Visual

from_xml turns an XML string into a struct, and spark.read.format xml with a rowTag reads XML files into a DataFrame An XML string becomes a struct; XML files become a DataFrame.

8. The Python Data Source API and Python UDTFs [Advanced]

Since: 4.0.0 · Status: Ran; needs pyarrow · Source: SPARK-44076, SPARK-43797

Lineage: filter pushdown for Python data sources in 4.1 (SPARK-51271), admission control for Python streaming readers in 4.2 (SPARK-55304).

What it is

You can write a data source in plain Python, register it, and read it by name, including from SQL (SPARK-44076). A Python user-defined table function (UDTF, SPARK-43797) returns several rows per input row. A data source is two small classes: a DataSource that names it and gives its schema, and a DataSourceReader whose read method yields rows. A UDTF is a class with an eval method that yields tuples.

Why it matters

This means you can read from an API or a custom format in a few lines of Python, with no Scala connector to build and ship.

Under the hood, Python workers run the reader, and data moves through Arrow, so the pyarrow package must be installed. On the official image both features fail until it is: the data source worker raises ModuleNotFoundError: No module named 'pyarrow'. The sample source named counter produced the rows n=1..4 with square=1,4,9,16, and the split_words UDTF turned spark four point zero into four rows with their lengths. The 4.0 release also adds polymorphic UDTFs, whose output schema is decided at call time, which I did not run.

Sample code

Runnable file: 09_python_data_source.py.

from pyspark.sql.datasource import DataSource, DataSourceReader

class CounterDataSource(DataSource):
    @classmethod
    def name(cls): return "counter"
    def schema(self): return "n INT, square INT"
    def reader(self, schema): return CounterReader(self.options)

class CounterReader(DataSourceReader):
    def __init__(self, options): self.limit = int(options.get("limit", 3))
    def read(self, partition):
        for i in range(1, self.limit + 1):
            yield (i, i * i)

spark.dataSource.register(CounterDataSource)   # needs pyarrow
spark.read.format("counter").option("limit", 4).load().show()   # n=1..4
from pyspark.sql.functions import udtf

@udtf(returnType="word: string, length: int")
class SplitWords:
    def eval(self, text: str):
        for w in text.split():
            yield (w, len(w))

spark.udtf.register("split_words", SplitWords)
spark.sql("SELECT * FROM split_words('spark four point zero')").show()  # spark 5, four 4, ...

When to use it

Use it when a custom source can be a few lines of Python, with no Scala connector to build. Skip it when a JVM connector already exists, or the source is performance-critical. Common beginner mistake: running without pyarrow; the worker raises ModuleNotFoundError: No module named 'pyarrow'.

Visual

A DataSource class and a DataSourceReader class are registered with spark.dataSource.register and read by name, producing n and square rows Two classes, one registration call, and the source is readable by name.

9. GROUP BY on maps and DESCRIBE … AS JSON [Intermediate]

Since: 4.0.0 · Status: Ran · Source: SPARK-47430, SPARK-50541

What it is

Before 4.0 a MAP column could not be a grouping key. In 4.0 it groups directly (SPARK-47430), so a query can count rows by the whole map. A MAP holds key and value pairs in one column. A second 4.0 addition is DESCRIBE ... AS JSON (SPARK-50541), which returns a table description as one JSON document instead of a text layout, so a script can parse it instead of scraping the output.

Why it matters

This means you can group on a map without first turning it into a string, and read table metadata from code.

Under the hood, grouping works but comparing two maps with = does not: map(1,'a') = map(1,'a') fails with DATATYPE_MISMATCH.INVALID_ORDERING_TYPE, so a join on a map column is still not available. The JSON description carries the table name, the catalog, the namespace, the schema and the columns with their types, including the collation of a string column, and more fields beyond what the sample shows. Both statements ran on 4.0.5 and gave the same output as on 4.3.0.

Sample code

Runnable file: 10_group_by_map_describe_json.sql.

-- a MAP column can be a grouping key
SELECT m, count(*) AS n
FROM VALUES (map('k', 1)), (map('k', 1)), (map('k', 2)) AS t(m)
GROUP BY m ORDER BY n;
-- {"k":2}  1
-- {"k":1}  2

CREATE TABLE people (id INT, name STRING) USING parquet;
-- the table description as one JSON document
DESCRIBE EXTENDED people AS JSON;
-- {"table_name":"people","catalog_name":"spark_catalog","namespace":["default"], ...}

When to use it

Use it when you count or aggregate by a map column, or a script needs table metadata as JSON. Skip it when you need to compare or join on maps, which is still not available. Common beginner mistake: using = on two maps; it fails with INVALID_ORDERING_TYPE. Group on the map instead.

Visual

A MAP column grouped by its whole value gives the map with k equal to 2 once and the map with k equal to 1 twice, and DESCRIBE EXTENDED people AS JSON returns one JSON document with the table name, catalog, namespace and columns Maps group as whole values, and the table description comes back as one JSON document.

10. transformWithState and the state data source [Advanced]

Since: 4.0.0 · Status: Ran; needs the RocksDB state store and protobuf · Source: SPARK-46815, SPARK-45511, SPARK-46865, SPARK-47553

Lineage: a row-based Python API in 4.1 (SPARK-51814), stateful queries under the real-time trigger in 4.3, for Scala and Java only (SPARK-57228).

What it is

Structured Streaming remembers information per key between micro-batches, such as a running count. That memory is called state. transformWithState (SPARK-46815) lets you write the logic yourself as a stateful processor: a class with an init method that declares typed state variables, a handleInputRows method that receives the rows for one key, and a close method. It exists in Scala, Java and Python, where the Python form is transformWithStateInPandas. The state data source (SPARK-45511) reads that state back as a table with spark.read.format("statestore").

Why it matters

This means you can keep custom per-key logic and inspect its state with a query instead of guessing.

Under the hood, a processor’s state variables are stored as separate column families, which only the RocksDB state store supports. On the default provider the query fails with UNSUPPORTED_FEATURE.STATE_STORE_MULTIPLE_COLUMN_FAMILIES. The Python runner also needs protobuf and the gRPC packages: without protobuf the query fails with STREAMING_PYTHON_RUNNER_INITIALIZATION_FAILURE and an ImportError for descriptor. In the run, a rate source produced the keys 0, 1 and 2, and after 40 seconds each key held a count of 78, in the output and in the state data source. The count depends on how long the query ran. Reading state before any batch has committed fails with STDS_COMMITTED_BATCH_UNAVAILABLE.

Sample code

Runnable file: 12_transform_with_state.py.

# needs RocksDBStateStoreProvider as the state store, and pip install protobuf grpcio
class CountProcessor(StatefulProcessor):
    def init(self, handle):
        schema = StructType([StructField("value", IntegerType())])
        self.count = handle.getValueState("count", schema)

    def handleInputRows(self, key, rows, timerValues):
        total = self.count.get()[0] if self.count.exists() else 0
        for pdf in rows:
            total += len(pdf)
        self.count.update((total,))
        yield pd.DataFrame({"id": [key[0]], "count": [total]})

    def close(self): pass
counts = stream.groupBy("id").transformWithStateInPandas(statefulProcessor=CountProcessor(),
    outputStructType="id STRING, count INT", outputMode="Update", timeMode="None")
state = spark.read.format("statestore").option("stateVarName", "count").load(checkpoint)
# [('0', 78), ('1', 78), ('2', 78)] from both the output and the state data source

When to use it

Use it when a streaming query needs custom per-key logic, such as running counts, sessions or timers, and you want to inspect the state. Skip it when a plain groupBy(...).count() does the job. Common beginner mistake: running it on the default state store, which fails with STATE_STORE_MULTIPLE_COLUMN_FAMILIES.

Visual

A rate source feeds groupBy and transformWithStateInPandas, which reads and updates one count per key in the RocksDB state store and emits output rows 0, 1 and 2 each with 78; the state data source reads the same counts back from the state store The same per-key counts appear in the output and in the state data source.

11. Polymorphic Python UDTFs [Intermediate]

Since: 4.0.0 · Status: Ran · Source: SPARK-43797

Lineage: Arrow-optimized UDTFs in 4.1 (SPARK-52979), and regular Python UDTFs are Arrow-optimized by default from 4.2, per the PySpark migration guide.

What it is

A user-defined table function (UDTF) returns several rows per call. Normally you declare its output columns up front with returnType, so the schema is fixed when you define the function. A polymorphic UDTF instead defines a static analyze method that receives the argument types and returns the output schema, so one function can serve different inputs. Python UDTFs arrived in 4.0 (SPARK-43797), and the 4.0 announcement names the analyze form polymorphic UDTFs.

Why it matters

This means you can write one generic table function, such as an echo or an unpivot, instead of one per type.

Under the hood, analyze runs when Spark plans the query. It receives an AnalyzeArgument that carries the argument’s dataType, and it returns an AnalyzeResult holding the schema. In the run, one class returned struct<c1:int> for echo(5) and struct<c1:string> for echo('x'). Without analyze, the schema is fixed at definition time, so a function that works over many types needs one UDTF per type. You call the function from SQL after spark.udtf.register, the same way as any other table function.

Sample code

Runnable file: 13_pyspark_extras.py.

@udtf
class Echo:
    @staticmethod
    def analyze(a: AnalyzeArgument) -> AnalyzeResult:
        # the output column takes the type of the argument
        return AnalyzeResult(StructType().add("c1", a.dataType))

    def eval(self, a):
        yield (a,)

spark.udtf.register("echo", Echo)
spark.sql("SELECT * FROM echo(5)")     # struct<c1:int>,    row (5)
spark.sql("SELECT * FROM echo('x')")   # struct<c1:string>

When to use it

Use it when one table function should work over several input types. Skip it when the output schema is fixed, where returnType is simpler. Common beginner mistake: assuming the schema is fixed: the same class returned c1 int for echo(5) and c1 string for echo('x').

Visual

The queries SELECT star FROM echo of 5 and echo of x both call analyze, which returns an output column typed from the argument: struct c1 int with row 5 for the integer, and struct c1 string with row x for the string analyze turns the argument type into the output schema.

12. Named and unnamed parameter markers [Beginner]

Since: 4.0.0 · Status: Ran · Source: SPARK-41666

Lineage: markers are allowed nearly anywhere, including identifiers, from 4.1 (SPARK-53573).

What it is

A parameter marker is a placeholder in a SQL string whose value you pass separately, so application code never splices a value into the text. Spark 4.0 supports two forms through spark.sql(..., args=...) in PySpark (SPARK-41666). A named marker is written :name and bound from a dictionary. An unnamed marker is written ? and bound from a list, in order. Both exist to replace string formatting, which is the usual source of SQL injection and quoting bugs.

Why it matters

This means you can build a query from user input without concatenating strings.

Under the hood, the values are bound separately from the statement, so a value never becomes SQL text. In the run, SELECT :a + :b with {a: 1, b: 2} and SELECT ? + ? with [1, 2] both returned 3. A value that looks like an attack, x'; DROP TABLE t; --, came back unchanged as a plain string. In 4.0 a marker stands where a literal value would. Spark 4.1 widens that to identifiers and other positions, which the 4.1 post covers.

Sample code

Runnable file: 13_pyspark_extras.py.

# named: a dictionary, matched by key
spark.sql("SELECT :a + :b AS s", args={"a": 1, "b": 2})        # 3

# unnamed: a list, matched by position
spark.sql("SELECT ? + ? AS s", args=[1, 2])                    # 3

# a value that looks like SQL stays a value
spark.sql("SELECT :s AS s", args={"s": "x'; DROP TABLE t; --"})   # x'; DROP TABLE t; --

When to use it

Use it when a value comes from user input or application code. Skip it when the value is a constant of the query itself. Common beginner mistake: building the SQL string with an f-string anyway, which puts the value into the text. Markers keep it out.

Visual

SELECT :a + :b with the dictionary a 1, b 2 and SELECT ? + ? with the list 1, 2 are bound by key and by position, and both return 3 Named markers bind by key, unnamed markers bind by position.

Smaller additions worth knowing

Feature What it does Status
EXECUTE IMMEDIATE (SPARK-46246) Runs a SQL string with bound parameters: EXECUTE IMMEDIATE 'SELECT count(*) FROM orders WHERE amount > ?' USING 8 returned 3 Ran on 4.0.5
Lightweight Python client and spark.api.mode The pyspark-client package, whose 4.0.0 source archive is 1.5 MB on PyPI, and a setting to turn Spark Connect on Listed
Structured JSON logging (SPARK-47240) Machine-readable driver and executor logs, enabled with spark.log.structuredLogging.enabled per the 4.0 announcement. The official image logs plain text, so I did not see JSON output Listed
Spark Kubernetes operator (SPARK-45923), Java 21 support Run Spark applications as Kubernetes custom resources Listed
PySpark plotting API (SPARK-49530) df.plot.bar(x="k", y="v") returned a Plotly Figure. It needs plotly installed, and I did not render the figure Ran
Unified PySpark UDF profiling One profiler for Python UDFs Listed
Spark ML on Connect pyspark.ml over a Connect server Listed

Version lineage

Several 4.0 features start an arc that later releases continue. Read the 4.0 column as what this release added.

Feature 4.0 4.1 4.2 4.3
VARIANT Introduced (SPARK-45827) Generally available, colon paths (SPARK-54454, SPARK-52494) none in the headlines Editing functions (SPARK-57804)
Procedural SQL Session variables, SQL UDFs (SPARK-42849, SPARK-46057); scripting flag off SQL scripting generally available (SPARK-54499) Cursors, behind flags (SPARK-54759) none in the headlines
Pipe syntax Introduced (SPARK-49555) none Aggregates and GROUP BY in a pipe SELECT, | alias (SPARK-54292, SPARK-51518) none
String collation Introduced (SPARK-46830) Schema-level collation (SPARK-52219) SHOW COLLATIONS (SPARK-49543) none
Streaming state transformWithState, state data source (SPARK-46815, SPARK-45511) Python row-based transformWithState (SPARK-51814) Stable source names, state snapshot repair (SPARK-54909, SPARK-54121) Stateful real-time mode, Scala and Java (SPARK-57228)
Python extensibility Data Source API, UDTFs (SPARK-44076, SPARK-43797) Filter pushdown for Python data sources, Arrow UDFs (SPARK-51271, SPARK-52214) Arrow-optimized UDFs on by default (SPARK-54555) Experimental transpiling (SPARK-55206)
ANSI mode On by default (SPARK-44444) On for the pandas API on Spark (SPARK-53295) none none

Flags at a glance

A feature that parses is not a feature that is on. In 4.0 the gates that matter are three:

Feature Config key Default What it controls
ANSI mode spark.sql.ansi.enabled true (read on 4.0.5) Invalid casts and divide by zero raise. Set false to restore the old NULL results
SQL scripting spark.sql.scripting.enabled false (read on 4.0.5) BEGIN ... END scripts. A script fails to parse until it is on; 4.1 turns it on by default
transformWithState spark.sql.streaming.stateStore.providerClass The HDFS-backed provider; set RocksDBStateStoreProvider The multiple column families that transformWithState needs

VARIANT, pipe syntax, SQL UDFs, session variables, collation, the XML source and the Python Data Source API have no flag in 4.0.

Settings and costs

Defaults the 4.0 release notes list as changed. I read the first row in the runs; the rest are from the notes, not read.

Setting Default in 4.0 Source
spark.sql.ansi.enabled true Read on 4.0.5
spark.eventLog.rolling.enabled true Release notes, SPARK-45771
spark.ui.prometheus.enabled true Release notes, SPARK-46886
spark.shuffle.service.db.backend ROCKSDB Release notes, SPARK-45351
Feature Cost to plan for
ANSI mode The change most likely to turn a green pipeline red. Test your real data before the upgrade
Python data source and UDTFs pyarrow must be installed on every node that runs a Python worker
Scala 2.13 and Java 17 Every jar the jobs depend on needs a Scala 2.13 build

When these features bite

Each item is a failure hit during the runs, with the text it produced, so you can grep for it. Every item is also an anchor, so you can link to one error.

[UNSUPPORTED_FEATURE.STATE_STORE_MULTIPLE_COLUMN_FAMILIES]. transformWithState stores each state variable as its own column family, and the default HDFS-backed state store does not support that. Fix: Set spark.sql.streaming.stateStore.providerClass to org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider.

[STREAMING_PYTHON_RUNNER_INITIALIZATION_FAILURE] with ImportError: cannot import name 'descriptor' from 'google.protobuf'. The Python runner for transformWithStateInPandas needs protobuf and the gRPC packages, which the official image lacks. Fix: pip install protobuf grpcio grpcio-status googleapis-common-protos.

[STDS_COMMITTED_BATCH_UNAVAILABLE]. The state data source was read before the query committed a batch. Fix: Let the query run and commit at least one micro-batch before you read its checkpoint.

[CAST_INVALID_INPUT], [DIVIDE_BY_ZERO]. ANSI mode is on by default, so work that silently produced NULL now stops. The message names the fix. Fix: Use try_cast and try_divide where NULL was the intended answer. spark.sql.ansi.enabled=false restores the old behaviour for the session.

[PARSE_SYNTAX_ERROR] ... at or near ':'. VARIANT colon paths do not exist in 4.0, so a job written on 4.1 breaks on a 4.0 cluster at the first statement. Fix: Not a flag: use 4.1 or later, or variant_get.

ModuleNotFoundError: No module named 'pyarrow' from the Python data source worker. The official image has no pyarrow. Fix: Install pyarrow and pandas.

[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] on map(...) = map(...). A MAP can be a grouping key in 4.0, but two maps cannot be compared with =. Fix: Group on the map; do not join on it.

Run it yourself

The samples ran on apache/spark:4.0.5-python3 and on the plain 4.0.5 release tarball (523 MB, Scala 2.13.16) with Java 17. This is the one release whose tarball runs the PySpark examples on a stock Python 3.9, because its Requires-Python is 3.9 or newer; the later releases need 3.10 or 3.11. The download and checksum script, and the runner for SQL files, are in RUNNING_ON_TARBALLS.md.

Hello world. 00_hello_world.sql printed 4.0.5 and the build hash. Before and after. 11_before_after.sql runs pipe syntax, collation, VARIANT and ANSI mode next to their older forms.

Streaming and Python. 12_transform_with_state.py needs the RocksDB state store and the protobuf and gRPC packages. 13_pyspark_extras.py covers the parameter markers, the polymorphic UDTF and the plotting API, and needs plotly. Both ran on apache/spark:4.0.5-python3.

SQL. smoke/smoke40.sql runs the headline features in one pass. On 4.0.5 it printed:

try_cast           NULL
variant_get        2
pipe syntax        eu 40.0, us 20.0
SQL UDF            1 11.799999999999999 ... 4 5.8999999999999995
session variable   2, 3
collation          true
from_xml           [Ana,30]

Python data source and UDTF. Create a virtual environment with pyarrow and pandas, point PYTHONPATH at the tarball’s python folder and its py4j zip, and run tarball/datasource_udtf.py. It ran with pyarrow 21.0.0 and pandas 2.3.3, and printed the four counter rows, the two rows with n > 1, and the four words from split_words.

python3 -m venv venv && venv/bin/pip install pyarrow pandas
SH=~/spark-4.x/spark-4.0.5-bin-hadoop3
env -i HOME="$HOME" PATH=/usr/bin:/bin JAVA_HOME="$JAVA" SPARK_HOME="$SH" \
  PYTHONPATH="$SH/python:$(ls $SH/python/lib/py4j-*.zip)" SPARK_LOCAL_IP=127.0.0.1 \
  PYSPARK_PYTHON=venv/bin/python venv/bin/python datasource_udtf.py

Upgrade notes for teams moving from 3.x to 4.0

Behaviour changes to test before you upgrade. The first rows are the ones I verified; the rest come from the official SQL and PySpark migration guides, which list many more.

Change What to do Source
Scala 2.13 and JDK 17 are the baseline; Scala 2.12 and JDK 8 and 11 are gone Rebuild every jar for Scala 2.13 SPARK-45314, SPARK-45315
ANSI mode is on by default Test your data; use try_*; spark.sql.ansi.enabled=false restores NULL (verified) SPARK-44444
CREATE TABLE without USING uses spark.sql.sources.default, not Hive spark.sql.legacy.createHiveTableByDefault=true SQL migration guide
ORC default compression is zstd, not snappy spark.sql.orc.compression.codec=snappy SQL migration guide
Inner CTEs take precedence (ctePrecedencePolicy is CORRECTED) spark.sql.legacy.ctePrecedencePolicy=EXCEPTION or LEGACY SQL migration guide
Timestamp parse failures raise CANNOT_PARSE_TIMESTAMP in ANSI mode, NULL otherwise spark.sql.legacy.timeParserPolicy=EXCEPTION SQL migration guide
! is no longer accepted as NOT in expr ! IN (...) and similar spark.sql.legacy.bangEqualsNot=true, or write NOT SQL migration guide
JDBC type mappings changed for Postgres, MySQL, Oracle, SQL Server and DB2 Per-database spark.sql.legacy.*Mapping.enabled settings, listed in the guide SQL migration guide
Hive metastore versions before 2.0.0 are unsupported Upgrade the metastore SQL migration guide
Python 3.8 dropped; pandas 2.0.0, NumPy 1.21 and PyArrow 11.0.0 are the minimums Upgrade the Python environment SPARK-47993, SPARK-50657
Many pandas API on Spark removals (iteritems, append, mad, Int64Index) and compute.ops_on_diff_frames on by default Run your pandas-on-Spark code first PySpark migration guide
from pyspark.sql.functions import * no longer exports DataFrame, Column or StructType Import them from pyspark.sql or pyspark.sql.types PySpark migration guide
A map column’s schema is inferred from all key-value pairs, not the first spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled=true PySpark migration guide
spark.eventLog.rolling.enabled, spark.ui.prometheus.enabled and a RocksDB shuffle-service store are on by default Recheck log and metrics setups SPARK-45771, SPARK-46886, SPARK-45351

Adopt now or wait?

4.0.0 shipped on 23 May 2025 and has had five patch releases, 4.0.1 to 4.0.5, the latest on 27 October 2026. The versioning policy gives a major release a 6-month maintenance window and plans 4.5.0 as the 4.x long-term-support release with 18 months, yet 4.0.5 shipped about 17 months after 4.0.0, so read the window as a minimum, not as an end date. My recommendation, a judgement and not a measurement: if you are leaving 3.x, move to the latest patch, 4.0.5, or go straight to a later line if your connectors allow it. The Scala 2.13, Java 17 and ANSI changes arrive here, and every later release builds on them.

Known limitations in this release

  • XML. I ran from_xml, schema_of_xml and a file read with rowTag. Writing XML was not run.
  • Python data source and UDTFs. I ran a batch reader, a plain UDTF and a polymorphic one. The streaming and writer interfaces were not run.
  • transformWithState. I ran the Python form with one value state per key. Timers, TTL, list and map state and the Scala and Java forms were not run, and the counts in the sample depend on how long the query ran.
  • ANSI mode. I tested invalid casts and divide by zero. Other ANSI behaviours, such as overflow, were not run.
  • Listed only. Unified UDF profiling, the lightweight Python client, structured JSON logging, ML on Connect and the Kubernetes operator are from the release notes, not run. The plotting API was run only as far as getting a figure back.

Cheat sheet

Feature Syntax Needs
ANSI mode try_cast(x AS INT), try_divide(a, b) Nothing; spark.sql.ansi.enabled=false restores NULL
VARIANT parse_json(s), variant_get(v, '$.b[1]', 'int') Nothing
Pipe syntax FROM t |> WHERE x > 5 |> AGGREGATE sum(x) AS s GROUP BY k Nothing
SQL UDF CREATE FUNCTION f(x DOUBLE) RETURNS DOUBLE RETURN x * 1.18 Nothing
Session variable DECLARE OR REPLACE VARIABLE v DOUBLE DEFAULT 15.0; SET VAR v = 25.0 Nothing
Collation 'Apple' COLLATE UTF8_LCASE = 'apple' Nothing
XML from_xml(s, 'name STRING'), spark.read.format("xml").option("rowTag", "person") Nothing
Python data source spark.dataSource.register(MySource) pyarrow
Stateful streaming groupBy(k).transformWithStateInPandas(statefulProcessor=P(), outputStructType=..., outputMode="Update", timeMode="None") RocksDB state store, protobuf
State data source spark.read.format("statestore").option("stateVarName", "count").load(checkpoint) A committed batch
Polymorphic UDTF @staticmethod def analyze(a): return AnalyzeResult(schema) Nothing
Parameter markers spark.sql("SELECT :a + ?", args=...): a dictionary for :a, a list for ? Nothing
GROUP BY on a map SELECT m, count(*) FROM t GROUP BY m Nothing
Table description as JSON DESCRIBE EXTENDED t AS JSON Nothing

Key takeaways

  • Baseline: Scala 2.13 and Java 17 are required. Start with your dependency tree.
  • Errors: ANSI mode turns silent NULLs into errors. It is the change most likely to break a working pipeline, so test it first.
  • SQL: pipe syntax, SQL functions, session variables and collations let you write more in SQL and less in driver code.
  • Semi-structured data: VARIANT and the built-in XML source read nested data by path, with no schema declared first.
  • Python: the Data Source API and UDTFs let you extend Spark in Python, and both need pyarrow installed. Polymorphic UDTFs and parameter markers make table functions and queries safer to reuse.
  • Streaming: transformWithState gives you per-key state you can read back, but it needs the RocksDB state store and protobuf, and each missing piece fails with its own error.

Further reading

Trademarks

Apache Spark, Apache Kafka, Apache Parquet, Apache Arrow and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. The Apache Software Foundation does not endorse this post.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee

Comments

Questions, corrections and reviews are welcome. Sign in with GitHub to post; comments are stored in this site's public GitHub Discussions.