All posts

What's new in Apache Spark 4.3: stateful real-time streaming, JSON_TABLE and history tracking

Spark 4.3 new features, tested: stateful real-time streaming, Auto CDC with history, JSON_TABLE, ASOF JOIN, VARIANT editing and more. Each feature has a status, a source ticket, a sample with its printed output, and a plain note on what needs a flag or a connector.

34 min read Spark

Spark 4.3 changes what a streaming query can do and what a SQL query can say. Stateful queries, such as running counts and de-duplication, now run under the real-time trigger. SQL gains standard answers to three old workarounds: reading JSON as rows, matching each row to the closest earlier row, and keeping the history of a changing record. This post is for data engineers and streaming platform teams on Spark 4.x. You will learn what ten features do, the syntax, what a real run printed, and which ones need a flag or a connector first. All four releases side by side are in the Spark 4.x features-by-release post.

Spark 4.3 on 30 Oct 2026: JSON_TABLE, ASOF JOIN, UNNEST in FROM, BIN BY and @ time travel; VARIANT editing, TIME(p), nanosecond timestamps and new functions; stateful real-time mode, Auto CDC SCD Type 2 and async progress tracking; the archive reader, DataFrame.zip, single-task execution, null join key spreading and the Connect REST API Everything in Spark 4.3.

Verified execution. Every sample in this post ran on Spark 4.3.0, on the apache/spark:4.3.0-python3 image or the plain release tarball, in November 2026. Each feature carries a Status line: Ran means the printed output is real, behind a flag names the setting, and needs a connector means the engine defines the syntax and a table format has to implement it. Where I could not run something, it is labelled listed and its limits are in Known limitations. The runnable files are in ranga-spark4-examples under spark-4.3/.

Before you start

What you need

Need Detail
Java 17. The runs used Temurin 17.0.20
Python, for PySpark 3.11 or newer for the 4.3 tarball. Python 3.10 support was dropped in this release
Spark The apache/spark:4.3.0-python3 image, or the 4.3.0 release tarball (535 MB)
Kafka A broker, for feature 1 only
A catalog with row-level operations For feature 2 only, for example an Apache Iceberg table

The smallest check that you are on the release you think you are on is 00_hello_world.sql. Run it first. It printed:

4.3.0 98bc0212c94705434471627068f41e374ccab143

If it prints another version, an older SPARK_HOME in your shell is winning. Start with env -u SPARK_HOME.

Glossary

Term Meaning
Micro-batch Structured Streaming’s default: the stream is run as a series of small batch jobs
Real-time mode A streaming mode where long-lived tasks process records as they arrive, with no batch boundary
Shuffle Moving data between stages so that rows with the same key meet on the same task
Task slot One unit of parallelism on an executor. Real-time mode needs a free slot for every task
CDC Change data capture: a feed of inserts, updates and deletes from a table
SCD Slowly changing dimension. Type 1 overwrites old values; Type 2 keeps every version as a row
VARIANT The Spark type for semi-structured data such as JSON, encoded once in binary
As-of join A join that matches each row to the closest earlier row on the other side

Try this first

A three-step path if you are new to these features. Each step is small and none needs Kafka or a connector.

  1. Confirm the version. Run the hello-world file above.
  2. Run one function. Try UNNEST on an array, or variant_set in feature 6.
  3. Flip one flag and run one join. Set spark.sql.join.asofJoin.enabled and run the ASOF JOIN sample.

1. Stateful real-time mode [Advanced]

Since: 4.3.0 · Status: Ran (needs a Kafka broker) · Source: SPARK-58635, SPARK-58508, SPARK-57228, SPARK-57141, SPARK-58263

Lineage: stateless queries in 4.1.0, stateful queries in 4.3.0.

What it is

Structured Streaming normally cuts a stream into small batches, called micro-batches, and runs one job per batch. Real-time mode replaces that with long-lived tasks that process records as they arrive. Spark 4.1 shipped it for stateless queries. 4.3 adds streaming aggregation (SPARK-58635), dropDuplicates (SPARK-58508) and Scala or Java transformWithState (SPARK-57228), the operators that keep state between records, and removes the @Experimental marker (SPARK-57281).

Why it matters

This means you can keep a running count or de-duplicate a Kafka topic and see each update as the record lands, instead of waiting for the end of a batch.

Under the hood, the stages of a query run at the same time, joined by a pipelined shuffle (StreamingShuffleManager, SPARK-57141) in which a downstream task reads while the upstream task is still writing, and the scheduler admits the stage group only when slots are free for all of its tasks, failing fast otherwise, and fails the group as one unit (SPARK-58263). That is why the cluster needs a free task slot for every task in every stage.

Sample code

Runnable file: 07_stateful_real_time_mode.scala.

// needs a Kafka broker
// set spark.sql.shuffle.partitions to a number of free slots
val counts = spark.readStream.format("kafka")
  .option("kafka.bootstrap.servers", "kafka:29092")
  .option("subscribe", "rtm43_demo3").option("startingOffsets", "latest").load()
  .select(col("value").cast("string").alias("word"))
  .groupBy("word").count()

// the real-time trigger replaces the micro-batch trigger
counts.writeStream.format("console").outputMode("update")
  .option("checkpointLocation", "/tmp/rtm43_ckpt3")
  .trigger(Trigger.RealTime("5 seconds")).start()

Producing a, b, a, a printed a 1, a 2, a 3 and b 1 in one batch, so each update is emitted as it happens, not once per key at the end. The default 200 shuffle partitions on a two-core session failed with CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT: it needed 201 slots and 2 were free.

When to use it

Use it when stateful queries on Kafka where a result should appear as each record lands, and you can reserve a task slot per task. Skip it when micro-batch latency is fine, the query needs Python or an ASOF JOIN, or the cluster is small. Common beginner mistake: leaving spark.sql.shuffle.partitions at 200 on a small cluster, which fails with CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT.

Constraints from the documentation

These are stated in the 4.3 real-time mode guide; I did not run them.

  • Output and checkpoint. The output mode must be update, or the query fails with STREAMING_REAL_TIME_MODE.OUTPUT_MODE_NOT_SUPPORTED. A checkpointLocation is required.
  • Sources and sinks. Kafka is the only source; the built-in rate source is not supported. Sinks are Kafka, foreach, console and memory, and foreachBatch is not supported.
  • Not supported. Stream-stream joins, flatMapGroupsWithState, session windows, PySpark transformWithState, range partitioning and Adaptive Query Execution.
  • Sizing. Count the source’s reader tasks plus the stateful operator’s shuffle tasks. A 10-partition Kafka topic feeding an aggregation with 5 shuffle partitions needs at least 15 cores, and the guide advises one real-time query per cluster.
  • The trigger duration is a checkpoint interval, not a latency target, and it must be at least spark.sql.streaming.realTimeMode.minBatchDuration, 5,000 ms by default.
  • Delivery. Processing is exactly-once, but the built-in Kafka sink is at-least-once.
  • Reported latency. The vendor’s 4.3 announcement reports P99 end-to-end latencies of 136 ms for dropDuplicates, 139 ms for a window aggregation and 141 ms for transformWithState, from benchmarks tracked in SPARK-58884, which has fix version 4.4.0. It gives no hardware or method, and I did not measure latency.

Visual

Kafka records a, b, a, a flow through a groupBy count under the real-time trigger and print running counts a 1, a 2, a 3, b 1 Records on the left, running counts on the right, with no batch boundary in between.

2. Auto CDC with SCD Type 2 [Advanced]

Since: 4.3.0 · Status: Ran to the error only; needs a connector · Source: SPARK-58321, SPARK-58261

Lineage: Auto CDC with SCD Type 1 in 4.2 (SPARK-56249), SCD Type 2 and its SQL syntax in 4.3.

What it is

Change data capture (CDC) is a feed of inserts, updates and deletes from a source table. Auto CDC applies that feed to a target table inside a Declarative Pipeline, the 4.1 framework in which you declare datasets and Spark derives the run order. A slowly changing dimension (SCD) says what happens to old values. Type 1 overwrites them, so only the latest row per key survives. Type 2, new in 4.3 (SPARK-58321), keeps every version as a row bounded by __START_AT and __END_AT.

Why it matters

This means you can keep the full history of a customer or product record with one declaration instead of hand-written MERGE logic.

Under the hood, Spark applies each micro-batch to the target as a MERGE, so the target must be a table whose connector implements Data Source V2 row-level operations. On a default Parquet table the syntax parses and the flow then fails with AUTOCDC_TARGET_DOES_NOT_SUPPORT_MERGE. The 4.3 documentation points to lakehouse connectors such as Apache Iceberg. I did not run a pipeline against an Iceberg catalog, so the success path is not shown here. Version note: SCD Type 1 came in 4.2 (SPARK-56249), Type 2 and its SQL syntax in 4.3 (SPARK-58261).

Sample code

Runnable file: 08_auto_cdc_scd2.sql.

-- target must support row-level operations, such as an Iceberg table
CREATE STREAMING TABLE customers_history
FLOW AUTO CDC
FROM STREAM(cdc_events)
KEYS (id)
APPLY AS DELETE WHEN op = 'DELETE'
SEQUENCE BY version
COLUMNS * EXCEPT (op)
STORED AS SCD TYPE 2;

When to use it

Use it when you need the history of a dimension record and your table format supports row-level operations. Skip it when the target is a plain Parquet table, or you only need the latest row, which is SCD Type 1. Common beginner mistake: pointing the flow at a Parquet table and expecting it to run; it fails with AUTOCDC_TARGET_DOES_NOT_SUPPORT_MERGE.

Visual

Three change events for one key flow through AUTO CDC; Type 1 keeps only the latest row and Type 2 keeps every version with the version range it was current Three events for one key: Type 1 keeps the last, Type 2 keeps all three with their ranges.

3. JSON_TABLE [Intermediate]

Since: 4.3.0 · Status: Ran; flat form only · Source: SPARK-58366

What it is

JSON_TABLE is the SQL-standard function that turns a JSON document into rows and typed columns (SPARK-58366). You give it a path that selects the repeating part of the document, then a COLUMNS list that maps paths to column names and types. It is the construct that Oracle, DB2 and MySQL queries use, so it is the one a migrated query already contains.

Why it matters

This means you can read a JSON string with one declarative query instead of chaining from_json with a schema, then explode.

Under the hood, FOR ORDINALITY adds a 1-based row number, EXISTS PATH returns a boolean for whether a path is present, and the function takes a column through LATERAL, so it works over a whole table of documents. With rows {"id":10,"opt":1} and {"id":20}, a hasOpt BOOLEAN EXISTS PATH '$.opt' column returned true and then false. Only the flat form is supported in 4.3.0: NESTED PATH columns are not, so a document nested more than one array deep still needs two steps.

Sample code

Runnable file: 01_json_table.sql.

-- seq counts the rows; id and name are typed columns read by path
SELECT t.* FROM JSON_TABLE(
  '{"items":[{"id":1,"n":"a"},{"id":2,"n":"b"}]}',
  '$.items[*]'
  COLUMNS (
    seq  FOR ORDINALITY,
    id   INT    PATH '$.id',
    name STRING PATH '$.n'
  )
) AS t;
-- 1  1  a
-- 2  2  b

*

When to use it

Use it when you read JSON arrays into rows, or you are migrating a query that already uses JSON_TABLE. Skip it when the document nests arrays more than one level deep; NESTED PATH is not supported, so split it into two steps. Common beginner mistake: expecting NESTED PATH columns to work.

Visual

A JSON document with an items array passes through JSON_TABLE with an ordinality column, an INT path and a STRING path and becomes two typed rows One document on the left, two typed rows on the right.

4. UNNEST in the FROM clause [Beginner]

Since: 4.3.0 · Status: Ran · Source: SPARK-58419

What it is

An array column holds several values in one row. UNNEST expands it into one row per value, and in 4.3 it works directly in the FROM clause (SPARK-58419). Before this, the same result came from explode or LATERAL VIEW, which are Spark-specific. UNNEST is the ANSI form, so it reads the same as in other engines. explode and LATERAL VIEW still work, so nothing has to be migrated; the gain is portability. A typical use is flattening an order’s line items into one row per item before you aggregate them.

Why it matters

This means you can port array-flattening queries from PostgreSQL or Trino with much less rewriting.

Under the hood, listing several arrays expands them in parallel, and the shorter ones are padded with NULL to the length of the longest. WITH ORDINALITY adds the position of each value, and a LATERAL reference lets the array come from a column of the same query.

Sample code

Runnable file: 03_unnest.sql.

-- two arrays of different length expand in parallel, the shorter padded with NULL
SELECT * FROM UNNEST(array(1,2), array(10,20,30)) AS t(a, b);
-- 1     10
-- 2     20
-- NULL  30

-- with the position of each value
SELECT * FROM UNNEST(array('a','b')) WITH ORDINALITY AS t(value, pos);
-- a  1
-- b  2

*

When to use it

Use it when you want array expansion that reads the same as in other SQL engines. Skip it when existing explode code works and portability is not a goal; nothing has to be migrated. Common beginner mistake: forgetting that the shorter of several arrays is padded with NULL, and that positions start at 1, where posexplode starts at 0.

Visual

Three UNNEST forms: with ordinality, two arrays of different length padded with NULL, and a lateral unnest over an array column The three forms: with ordinality, two arrays padded to the longer, and lateral over a column.

5. ASOF JOIN [Intermediate]

Since: 4.3.0 · Status: Ran, behind a flag · Source: SPARK-58092, SPARK-58124, SPARK-56887

What it is

An as-of join matches each row on the left with the closest earlier row on the right: the quote in force when a trade happened, or the price list valid on an order date (SPARK-58092). A MATCH_CONDITION states the time comparison, and an ON clause states the key that must be equal. Without it you write a window function and a join, and you have to get the tie-breaking right yourself.

Why it matters

This means you can attach reference data to events by time with one join, which is the shape of most time-series enrichment.

Under the hood, the SQL form executes as a sort-merge join (SPARK-58124), backed by a dedicated operator (SPARK-56887). It is behind spark.sql.join.asofJoin.enabled, default false; with the flag off the statement fails at parse time with UNSUPPORTED_FEATURE.ASOF_JOIN. Only INNER and LEFT OUTER are supported, and real-time mode does not support it. In the sample, the AAPL trade at 10:00:20 takes the 10:00:10 quote, not the 10:00:00 one, because the join picks the closest earlier row, not the first match.

Sample code

Runnable file: 02_asof_join.sql.

SET spark.sql.join.asofJoin.enabled = true;
-- each trade takes the latest quote at or before its timestamp
SELECT t.id, t.sym, t.ts, q.ts AS quote_ts, q.bid
FROM trades t ASOF JOIN quotes q
  MATCH_CONDITION (t.ts >= q.ts)
  ON t.sym = q.sym
ORDER BY t.id;
-- 1  AAPL  10:00:05  10:00:00  100.0
-- 2  AAPL  10:00:20  10:00:10  101.0
-- 3  MSFT  10:00:07  10:00:06  200.0

When to use it

Use it when you enrich events with the latest reference row at or before each event time. Skip it when you need an outer join other than LEFT OUTER, or the query runs in real-time mode. Common beginner mistake: running it without the flag; the statement fails at parse time with UNSUPPORTED_FEATURE.ASOF_JOIN.

Visual

ASOF JOIN on a timeline: each AAPL and MSFT trade is matched to the latest quote at or before its timestamp Three trades matched to the quotes in force at the time: 100.0, 101.0 and 200.0.

6. VARIANT editing functions [Intermediate]

Since: 4.3.0 · Status: Ran · Source: SPARK-57804

Lineage: VARIANT in 4.0, generally available in 4.1, editing functions in 4.3.

What it is

VARIANT is the Spark type for semi-structured data such as JSON, stored in a binary encoding so it is parsed once. Before 4.3 you could read a variant but not change it without rebuilding the whole value. Five functions now return a new variant with a path set, inserted, deleted or appended to (SPARK-57804 and others).

Why it matters

This means you can patch one field in a JSON payload inside SQL, without a round trip through a struct schema.

Under the hood, every function returns a new value, because variants are immutable. variant_set and variant_insert differ on an existing key: set replaces it and insert raises VARIANT_DUPLICATE_KEY. The try_ forms (try_variant_set and others) return NULL instead of raising, which is the safer choice for dirty data. Inserting into an array shifts the later elements right: variant_insert at $[1] on ["a","b","c"] with z gives ["a","z","b","c"], and variant_array_append adds to the end. variant_strip_nulls removes null fields and null array elements, so {"a":1,"b":null,"c":[1,null]} becomes {"a":1,"c":[1]}.

Sample code

Runnable file: 04_variant_editing.sql.

-- set a new key
SELECT variant_set(parse_json('{"a":1,"tags":["x"]}'), '$.b', 3);
-- {"a":1,"b":3,"tags":["x"]}

-- delete two paths in one call
SELECT variant_delete(parse_json('{"a":1,"b":2,"items":[1,2,3]}'), '$.a', '$.items[0]');
-- {"b":2,"items":[2,3]}

-- remove null fields and null array elements
SELECT variant_strip_nulls(parse_json('{"a":1,"b":null,"c":[1,null]}'));
-- {"a":1,"c":[1]}

When to use it

Use it when you patch a field in a JSON payload that has no fixed schema. Skip it when the shape is fixed and a typed column or struct would do; typed columns are easier to filter and document. Common beginner mistake: using variant_insert on a key that already exists, which raises VARIANT_DUPLICATE_KEY. Use variant_set to replace.

Visual

Five variant editing functions each take a variant on the left and return the edited variant on the right: set, insert, delete, array append and strip nulls Five functions, each taking a variant and returning an edited copy.

7. BIN BY [Intermediate]

Since: 4.3.0 · Status: Ran, behind a flag · Source: SPARK-57133, SPARK-58003

What it is

BIN BY spreads a measure over fixed-width time bins in proportion to how much of each interval falls inside each bin (SPARK-57133). A call that runs from 10:00 to 10:25 and costs 100 becomes 40, 40 and 20 across three ten-minute bins. That is the usual step before charting usage per interval. Without it, the work is a join to a calendar table plus overlap arithmetic.

Why it matters

This means you can turn start and end timestamps into per-interval figures with one clause.

Under the hood, it is a relation operator with its own physical execution (SPARK-58003), and it is behind spark.sql.binByRelationOperator.enabled, default false. The two range columns must be TIMESTAMP or TIMESTAMP_NTZ: an INT column fails with BIN_BY_RANGE_TYPE_MISMATCH. The grammar also takes ALIGN TO to set the bin origin. Use it for a quantity that accrues over the interval, such as cost, bytes or minutes of usage. DISTRIBUTE UNIFORM names the columns to share out evenly across the bins, and BIN_START AS and BIN_END AS name the two bin-boundary columns it adds.

Sample code

Runnable file: 09_bin_by.sql.

SET spark.sql.binByRelationOperator.enabled = true;
-- cost is spread over the 10-minute bins each call overlaps
SELECT id, bin_start, bin_end, cost
FROM calls
BIN BY (RANGE started TO ended BIN WIDTH INTERVAL 10 MINUTES DISTRIBUTE UNIFORM (cost)
        BIN_START AS bin_start BIN_END AS bin_end)
ORDER BY id, bin_start;
-- 1  10:00  10:10  40.0
-- 1  10:10  10:20  40.0
-- 1  10:20  10:30  20.0
-- 2  10:00  10:10  15.0   (10:05 to 10:15 costs 30: five minutes in each bin)
-- 2  10:10  10:20  15.0

When to use it

Use it when a measure accrues over an interval, such as cost, bytes or minutes of usage, and you want it per time bin. Skip it when you are counting events rather than spreading an amount, or the range columns are not timestamps. Common beginner mistake: passing integer range columns, which fails with BIN_BY_RANGE_TYPE_MISMATCH.

Visual

Call 1, from 10:00 to 10:25 with cost 100, splits into 40, 40 and 20 across three ten-minute bins; call 2, from 10:05 to 10:15 with cost 30, splits into 15 and 15 Each bin gets the share of the interval that falls inside it.

8. TIME(p) and nanosecond timestamps [Intermediate]

Since: 4.3.0 · Status: Ran, behind flags; nanosecond timestamps are a preview · Source: SPARK-57551, SPARK-56876

What it is

TIME(p) is a time-of-day type with a precision p up to nine digits, which is nanoseconds (SPARK-57551). TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p) do the same for timestamps, with and without a time zone, for p from 7 to 9 (SPARK-56876). Spark’s default timestamp keeps microseconds, six digits, so a nanosecond value is silently shortened unless you ask for the wider type.

Why it matters

This means you can keep the precision that trading systems, sensors and some databases produce, instead of rounding it away on ingest.

Under the hood, both families are behind flags: spark.sql.timeType.enabled for TIME, and spark.sql.timestampNanosTypes.enabled, which the release notes call a preview, for the timestamps. Without the first, CAST(... AS TIME(9)) fails with UNSUPPORTED_TIME_TYPE. In this release the Python client could not materialise a TIMESTAMP_NTZ(9) result: selecting the column raised CANNOT_PARSE_DATATYPE, which is why the sample casts to a string. The release also threads TIME through the built-in sources: schema inference in CSV, JSON and XML, the JDBC source, and predicate pushdown to ORC (SPARK-57572, SPARK-57555, SPARK-57571).

Sample code

Runnable file: 05_time_and_nanosecond_types.sql.

SET spark.sql.timeType.enabled = true;
SET spark.sql.timestampNanosTypes.enabled = true;
-- the cast to STRING keeps all nine digits
SELECT CAST(CAST('2026-11-01 10:00:00.123456789' AS TIMESTAMP_NTZ(9)) AS STRING);
-- 2026-11-01 10:00:00.123456789

-- the default timestamp keeps only microseconds
SELECT CAST('2026-11-01 10:00:00.123456789' AS TIMESTAMP_NTZ);
-- 2026-11-01 10:00:00.123456

When to use it

Use it when your source produces nanosecond precision, such as trading or sensor data, and you must keep it. Skip it when microseconds are enough; the default timestamp is simpler and already supported everywhere. Common beginner mistake: casting to the default TIMESTAMP_NTZ and expecting nanoseconds, which are cut to six digits, or selecting a TIMESTAMP_NTZ(9) column from Python.

Visual

The input 2026-11-01 10:00:00.123456789 is cut to microseconds by the default TIMESTAMP_NTZ and kept in full by TIMESTAMP_NTZ(9); TIME(9) holds 10:15:30.123456789; each needs its own flag The default type cuts at six digits; the nine-digit types need their flags.

9. DataFrame.zip [Beginner]

Since: 4.3.0 · Status: Ran · Source: SPARK-55886, SPARK-57247

What it is

DataFrame.zip appends the columns of a second DataFrame to the first, row by row, with no join key (SPARK-55886). Spark Connect has it too (SPARK-57247). A DataFrame is a table of rows in Spark. The normal way to combine two of them is a join on a key. zip is for the case where there is no key and the two DataFrames line up by position, such as attaching a derived column that you computed in a separate step.

Why it matters

This means you can add a computed column from a second step without inventing a join key. Under the hood the pairing is positional, so it relies on both sides keeping the same rows in the same order. The Scaladoc says the other DataFrame must derive from the same source as the receiver, which holds in the sample, where the second DataFrame is computed from the first. Zipping with an unrelated spark.range(3) also ran here, because both sides were the same range. I did not test two sources with different partitioning, which is where a positional pairing is not safe. If you need a guarantee, join on a key.

Sample code

Runnable file: 11_dataframe_zip.scala.

val a = spark.range(3).toDF("id")
// b is computed from a, so the two line up row for row
val b = a.selectExpr("id * 10 AS tenfold")
a.zip(b).show()
// +---+-------+
// | id|tenfold|
// +---+-------+
// |  0|      0|
// |  1|     10|
// |  2|     20|
// +---+-------+

When to use it

Use it when the second DataFrame is derived from the first and you want its columns alongside. Skip it when the two come from independent sources or are partitioned differently; join on a key instead. Common beginner mistake: assuming zip matches rows by a key. It matches by position.

Visual

a is a DataFrame with id 0, 1, 2 and b is computed from it as id times 10; a.zip(b) appends the columns of b to a and returns the rows (0, 0), (1, 10) and (2, 20) The columns of b are appended to a, row by row, with no key.

10. Single-task execution for small queries [Intermediate]

Since: 4.3.0 · Status: Ran, behind a flag · Source: SPARK-57851

What it is

Spark plans every query for large data: aggregates and sorts are separated by shuffles, which move rows between tasks. For a query over a handful of rows, those shuffles cost more than the work. Single-task execution (SPARK-57851), part of the effort the 4.3 announcement calls Project Feather, lets an eligible small query run in one task and skip the shuffles. It is opt-in, behind spark.sql.optimizer.singleTaskExecution.enabled, and it changes the physical plan, not the result.

Why it matters

This means a small query, such as a lookup or a test over a tiny table, no longer pays for the full distributed plan.

Under the hood, with the flag off the sample’s plan was LocalTableScan, HashAggregate, Exchange, HashAggregate, Exchange, Sort. With the flag on it was LocalTableScan, HashAggregate, Sort: both Exchange steps were gone, the query ran as one stage with one task, and the result was the same, [eu,40.0] and [us,25.0]. The vendor’s announcement says an eligible local relation holds up to 1,000 rows and an eligible file scan reads one small non-empty file; I tested only a four-row local relation, so I cannot confirm those limits. Opt in per session, not cluster-wide, because a query that grows past the limits needs the normal plan.

Sample code

Runnable file: 13_single_task_execution.scala.

val query = "SELECT region, sum(amount) AS total FROM orders GROUP BY region ORDER BY region"
// orders is a four-row local relation: (1,'eu',10.0), (2,'us',20.0), (3,'eu',30.0), (4,'us',5.0)
// operators(sql) lists the operator names in EXPLAIN FORMATTED; see the runnable file
println("OFF " + operators(query))
spark.conf.set("spark.sql.optimizer.singleTaskExecution.enabled", "true")
println("ON  " + operators(query))
// OFF LocalTableScan > HashAggregate > Exchange > HashAggregate > Exchange > Sort
// ON  LocalTableScan > HashAggregate > Sort

When to use it

Use it when a session runs many tiny queries, such as tests or lookups over small tables. Skip it when the data can grow; the normal distributed plan is what you want then. Common beginner mistake: expecting a speed-up on a large table. The flag does nothing for data beyond the eligibility limits, and I made no timing measurement.

Visual

With the flag off the plan is LocalTableScan, HashAggregate, Exchange, HashAggregate, Exchange, Sort; with the flag on it is LocalTableScan, HashAggregate, Sort, so both Exchange steps are gone and the query runs as one task with the same result The two shuffles disappear for an eligible small query.

Smaller additions worth knowing

Feature What it does Status
Archive reader (SPARK-57135 and others) Reads the entries of a tar or zip as files, with spark.sql.files.archive.reader.enabled=true. A tar holding two CSVs returned three rows. With the flag off, count(*) on the same tar returned 4 and raised no error Ran, behind a flag
New functions (SPARK-56820, SPARK-57253, SPARK-58399, SPARK-58295) counter_diff, collect_union, jaro_winkler_similarity, to_base32 and from_base32 Ran
Arrow-native Dataset caching (SPARK-57268) Opt-in caching with compression and column statistics, set with the static spark.sql.cache.serializer Listed, behind a flag
Local Connect server reuse (SPARK-57787) Reuse a persistent local Connect server for faster PySpark startup, with SPARK_LOCAL_CONNECT_REUSE=1; the announcement calls it experimental Listed
Connect REST endpoints (SPARK-57941) GET /api/v1/applications/<appId>/connect/sessions returned a session with sessionId, userId, startTimestamp, finishTimestamp, totalExecution and totalTime. /connect/operations exists but returned an empty list in my run Ran partly
Connect operation IDs (SPARK-58547) Trace a request end to end Listed
@ time travel on table names (SPARK-57421, SPARK-58012) Read a table at a version or timestamp Listed, needs a connector
Connect JDBC driver complex types (SPARK-57380) Return ARRAY, MAP and STRUCT through the Connect JDBC driver Listed
SQLContext wrapper on Connect (SPARK-57021) Old SQLContext code runs against a Connect session Listed
Security and UI hardening (SPARK-58054, SPARK-59319, SPARK-59508) A Content-Security-Policy header by default, an opt-in restricted SQL execution mode, and a restricted unpickle serializer for the Python client Listed
Experimental transpiling of simple Python UDFs (SPARK-55206) Turns simple UDFs such as len() into Catalyst expressions, behind spark.sql.experimental.optimizer.transpilePyUDFs Listed, experimental

Version lineage

Several 4.3 features finish an arc that began earlier. Read the 4.3 column as what changed since 4.2.

Feature 4.0 4.1 4.2 4.3
VARIANT Introduced (SPARK-45827) Generally available, colon paths (SPARK-54454, SPARK-52494) none in the headlines Editing functions (SPARK-57804)
Real-time mode none Stateless, Scala, Kafka (SPARK-53736) PySpark trigger (SPARK-54660) Stateful: aggregation, dropDuplicates, transformWithState (SPARK-58635)
Change data and history none Declarative Pipelines (SPARK-51727) CHANGES clause, Auto CDC SCD Type 1 (SPARK-55668, SPARK-56249) Auto CDC SCD Type 2 (SPARK-58321)
Time-series SQL none none time_bucket (SPARK-56594) ASOF JOIN, BIN BY (SPARK-58092, SPARK-57133)
Small data none none none Single-task execution, Arrow caching (SPARK-57851, SPARK-57268)
Spark Connect Lightweight Python client JDBC driver (SPARK-53484) GetStatus API (SPARK-55606) REST endpoints (SPARK-57941)
Python UDFs Python UDTFs Arrow UDF decorators (SPARK-52214) Arrow-optimized UDFs on by default (SPARK-54555) Experimental transpiling (SPARK-55206)

Flags at a glance

A feature that parses is not a feature that is on. Every feature in this post that is off by default:

Feature Config key Default What it enables
ASOF JOIN spark.sql.join.asofJoin.enabled false The ASOF JOIN ... MATCH_CONDITION syntax
BIN BY spark.sql.binByRelationOperator.enabled false The BIN BY relation operator
TIME(p) spark.sql.timeType.enabled off The TIME type
Nanosecond timestamps spark.sql.timestampNanosTypes.enabled off, preview TIMESTAMP_NTZ(9) and TIMESTAMP_LTZ(9)
Archive reader spark.sql.files.archive.reader.enabled false Reading the entries of a tar or zip as files
Single-task execution spark.sql.optimizer.singleTaskExecution.enabled false Running eligible small queries in one task, without shuffles
Arrow cache prefetch spark.sql.execution.arrow.cache.prefetch.enabled false Prefetching the next batch of a compressed Arrow cache

Settings and costs

Setting Default What to set it to
spark.sql.streaming.realTimeMode.minBatchDuration 5000 ms (from the docs) The shortest allowed real-time trigger duration
spark.sql.shuffle.partitions 200 For real-time mode, the number of task slots the cluster can run at once. At 200 on a two-slot session the query failed, needing 201 slots with 2 free
Feature Cost to plan for
Stateful real-time mode One free task slot per task across all stages, held for the life of the query
Auto CDC Each micro-batch is a MERGE, so the target must support row-level operations
BIN BY One output row per bin an interval overlaps. In the sample, a 25-minute call over 10-minute bins produced three rows
Archive reader A silent wrong result if the flag is forgotten: a two-file tar returned 4 rows and no error
Nanosecond types Behind flags, and not yet readable by the Python client

When these features bite

Each item is a failure hit during the runs, with the text it produced, so you can grep for it. Every item is also an anchor, so you can link to one error.

[AUTOCDC_TARGET_DOES_NOT_SUPPORT_MERGE]. The engine defines STORED AS SCD TYPE 2, but a Parquet target has no row-level operations. The same shape applies to @ time travel. Fix: Check that your connector implements the feature.

[UNSUPPORTED_FEATURE.ASOF_JOIN]. ASOF JOIN is behind a flag. Fix: spark.sql.join.asofJoin.enabled=true.

[UNSUPPORTED_TIME_TYPE]. TIME(p) is behind a flag. Fix: spark.sql.timeType.enabled=true.

[FEATURE_NOT_ENABLED] for nanosecond timestamps. A preview feature, off by default. Fix: spark.sql.timestampNanosTypes.enabled=true.

CANNOT_PARSE_DATATYPE ... timestamp_ntz(9). The Python client cannot materialise a nanosecond timestamp column. Fix: CAST(... AS STRING) before collecting.

[BIN_BY_RANGE_TYPE_MISMATCH]. The BIN BY range columns must be TIMESTAMP or TIMESTAMP_NTZ. Fix: Cast the range columns.

[VARIANT_DUPLICATE_KEY]. variant_insert on a key that already exists. Fix: Use variant_set, or the try_ form that returns NULL.

count(*) is 4 on a tar of two CSVs, and no error. The archive reader is off by default, so the tar is read as plain CSV text. Fix: spark.sql.files.archive.reader.enabled=true.

STREAMING_REAL_TIME_MODE.OUTPUT_MODE_NOT_SUPPORTED, STREAMING_REAL_TIME_MODE.INPUT_STREAM_NOT_SUPPORTED and INVALID_STREAMING_REAL_TIME_MODE_TRIGGER_INTERVAL. Real-time mode needs the update output mode, a Kafka source and a trigger of at least 5,000 ms. These come from the documentation, not from a run. Fix: Use update, read from Kafka, and set the trigger to 5 seconds or more.

[CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT] ... needs 201 concurrent task slots ... only 2 are currently free. Every stage of a real-time query runs at once, so the cluster needs a free slot for each task. The default 200 shuffle partitions on a small session cannot fit. Fix: Set spark.sql.shuffle.partitions to what the cluster can run at once.

Run it yourself

The samples ran on apache/spark:4.3.0-python3 and on the 4.3.0 release tarball (535 MB, Scala 2.13.18) with Java 17. The tarball declares Python 3.11 or newer, the highest floor of the four releases. If you run it locally, start it with env -u SPARK_HOME, because an older SPARK_HOME in the shell wins and silently runs the wrong Spark. The download script and the SQL runner are in RUNNING_ON_TARBALLS.md. The BIN BY, archive reader and DataFrame.zip examples run with spark-sql -f and spark-shell -i on the tarball.

Single-task execution. 13_single_task_execution.scala prints the plan with the flag off and on. Hello world. 00_hello_world.sql printed 4.3.0 and the build hash. Before and after. 12_before_after.sql runs the from_json and posexplode forms next to JSON_TABLE and UNNEST.

SQL. smoke/smoke43.sql printed this on 4.3.0:

JSON_TABLE         1 1 a   /   2 2 b
UNNEST             1 10   /   2 20   /   NULL 30
variant_set        {"a":1,"b":3,"tags":["x"]}
ASOF JOIN          1 10:00:00 100.0   /   2 10:00:10 101.0
TIME(9)            time(9)

Stateful real-time mode against Kafka. It reads from a broker on localhost:9092, which the datalake-playground publishes with the all profile. The Kafka source is not in the tarball, so add it with --packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.3.0, run on local[6], and set spark.sql.shuffle.partitions small enough that every stage can run at once. Feed tarball/rtm43.scala on standard input. It printed the counts a 1, 2 and 3 and b 1 in one batch, and finished with exception=None.

git clone https://github.com/rangareddy/datalake-playground.git
cd datalake-playground
PROFILE=all SPARK_VERSION=3.5.9 sh run_datalake.sh start
sh run_datalake.sh status

Upgrade notes for teams moving to 4.3

Behaviour changes to test before you upgrade. The first rows come from the 4.3 release notes; the rest come from the official SQL and PySpark migration guides, which list more.

Change What to do Source
Python 3.10 support is dropped, and the minimum NumPy is 1.23.2 Check the Python version on every image SPARK-56889, SPARK-56928
mapInPandas functions must return a strict iterator A function that returned a list or another iterable can fail at run time SPARK-58601
spark.rdd.compress, the web UI Content-Security-Policy header and, on Kubernetes, useDriverPodIP and no privilege escalation are on by default Recheck RDD sizes, UI embedding and pod security settings SPARK-57723, SPARK-58054
ceil and floor raise overflow errors in ANSI mode Test numeric code near the limits SPARK-58052
GROUP BY GROUPING SETS (()) over empty input returns one row spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled=false restores no rows. The ticket’s fix version is 4.3.0, but the SQL migration guide lists the change under 4.1 to 4.2, so test on your version SPARK-57775
Whole-stage code generation for UNION is on by default Expect a plan difference if you compare plans SPARK-56805
replaceHashWithSortAgg and combineAdjacentAggregation default to true Set both spark.sql.execution.* flags to false to restore SQL migration guide
The adaptive rule DynamicJoinSelection is now DemoteBroadcastHashJoin; shuffled-hash preference moved to spark.sql.adaptive.convertSortMergeJoinToShuffledHashJoin.enabled. MergeSubplans moved packages Excluding the old rule names silently does nothing; use the new names SQL migration guide
percentile, percentile_cont and median can differ in the last ULP Loosen exact comparisons SQL migration guide
Non-deterministic filters such as rand() are no longer pushed to Data Source V2 sources None; Spark evaluates them after the scan SQL migration guide
hash() and xxhash64 include the days field of CalendarInterval Hash values over intervals change, so rebuild anything keyed on them SQL migration guide
HAVING is evaluated before window functions when the select list has a generator such as explode Recheck queries that mix them SQL migration guide
Dataset.observe counts only the last successful attempt of each task spark.sql.legacy.observeMetricsAggregateAllAttempts=true restores SQL migration guide
foreachBatch on Spark Connect receives the correct session Code that relied on the old session can break SPARK-44462
Spark Connect session errors now carry SQLSTATE 08003, not HY000 Match on 08003 or class 08 SQL migration guide
Declarative Pipelines honours spark.sql.caseSensitive, and all flows into one table must agree on it Set the same value on every flow, or spark.sql.caseSensitive=true SQL migration guide

Connector availability

Check this before the engine. As of 18 November 2026, on Maven Central: Apache Iceberg publishes iceberg-spark-runtime for Spark 4.0 and 4.1 only (1.12.0), and Apache Hudi publishes hudi-spark4.0-bundle and hudi-spark4.1-bundle only (1.2.1); neither has an artifact for Spark 4.2 or 4.3. The latest delta-spark_2.13, 4.4.1, is built against Spark 4.2.0, with none built against 4.3. Spark’s own Kafka connector for 4.3.0 exists. A lakehouse stack therefore has to stay on 4.1 or wait for those releases.

Adopt now or wait for 4.3.1?

As of 18 November 2026, 4.3.0 is the latest release and spark.apache.org lists no 4.3.1. The versioning policy makes 4.3.0 the first quarterly feature release, gives feature releases a 6-month maintenance window, and plans 4.5.0 as the 4.x long-term-support release with 18 months. It also lists a 4.4 branch cut on 16 November 2026 and a first release candidate by 7 December 2026. 4.3.0 shipped without preview builds, so it has had the least outside testing, and the first patch release is where early corrections land. My recommendation, a judgement and not a measurement: evaluate on a test cluster now, and move production to the first patch release, unless a feature in this post, such as stateful real-time mode, is the reason you are upgrading. If you run Iceberg, Hudi or Delta, the connector availability above decides first.

Known limitations in this release

  • Auto CDC SCD Type 2. I did not run it against a catalog that supports row-level operations, so only the AUTOCDC_TARGET_DOES_NOT_SUPPORT_MERGE failure on Parquet is shown, not a successful history.
  • Nanosecond timestamps. The Python client could not materialise a TIMESTAMP_NTZ(9) result (CANNOT_PARSE_DATATYPE), so results must be cast to a string.
  • Syntax gaps. JSON_TABLE has no NESTED PATH, and the vendor announcement says the JSON_VALUE, JSON_EXISTS and JSON_QUERY follow-on work is on a separate timeline. ASOF JOIN supports only INNER and LEFT OUTER, and not in real-time mode.
  • Real-time mode. The constraints listed under feature 1 come from the documentation, and I did not benchmark latency. The vendor’s reported figures are attributed, not reproduced.
  • Single-task execution. I tested one four-row local relation. The eligibility limits are from the vendor’s announcement.
  • Spark Connect. I found the REST endpoints and read a live session from /connect/sessions, but my client attempts did not complete a query, so I did not capture an operations payload. Operation IDs, the JDBC complex types, the SQLContext wrapper and @ time travel are listed from the release notes, not run.

Cheat sheet

Feature Syntax Needs
Stateful real-time mode .trigger(Trigger.RealTime("5 seconds")) Kafka; free task slots
Auto CDC SCD Type 2 CREATE STREAMING TABLE t FLOW AUTO CDC FROM STREAM(s) KEYS (id) SEQUENCE BY v STORED AS SCD TYPE 2 A connector with row-level operations
JSON_TABLE JSON_TABLE(doc, '$.items[*]' COLUMNS (id INT PATH '$.id')) Nothing
UNNEST FROM UNNEST(arr) WITH ORDINALITY AS t(value, pos) Nothing
ASOF JOIN a ASOF JOIN b MATCH_CONDITION (a.ts >= b.ts) ON a.k = b.k A flag
VARIANT editing variant_set(v, '$.b', 3), variant_delete(v, '$.a') Nothing
BIN BY BIN BY (RANGE s TO e BIN WIDTH INTERVAL 10 MINUTES DISTRIBUTE UNIFORM (m)) A flag
TIME(p), nanoseconds CAST(x AS TIME(9)), TIMESTAMP_NTZ(9) Flags
Single-task execution spark.conf.set("spark.sql.optimizer.singleTaskExecution.enabled", "true") A flag; small queries only
DataFrame.zip a.zip(b) Nothing

Key takeaways

  • Streaming: real-time mode now covers stateful queries. Budget a free task slot per task across all stages, and set the shuffle partition count to match.
  • History tracking: Auto CDC SCD Type 2 keeps every version of a record, but only on a target that supports row-level operations. Test your connector before you plan around it.
  • Nested data: JSON_TABLE, UNNEST and the VARIANT editing functions replace most from_json and explode chains with standard SQL.
  • Time series: ASOF JOIN and BIN BY put enrichment and interval allocation in one clause each, and both are off until you set their flags.
  • Small queries: single-task execution removes both shuffles for an eligible small query; it is behind a flag, and I measured the plan, not the time.
  • Read the flags: TIME, nanosecond timestamps, ASOF JOIN, BIN BY and the archive reader all default to off. A feature that parses is not a feature that is on.

Further reading

Trademarks

Apache Spark, Apache Kafka, Apache Iceberg, Apache Parquet and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. The Apache Software Foundation does not endorse this post.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee

Comments

Questions, corrections and reviews are welcome. Sign in with GitHub to post; comments are stored in this site's public GitHub Discussions.