Apache Iceberg through Spark Connect: the full surface, and the extension that breaks time travel

• Architecture: where Iceberg has to live
• Starting the server
• Connecting the client
• The write and read surface
• Time travel, and the extension that breaks it
  • The failure, and the diagnosis that found it
• What the thin client cannot do
• Production notes
• Frequently asked questions
• Conclusion
• References
• Trademarks

TL;DR

• The Iceberg runtime jar, spark.sql.extensions and the catalog configs belong to the Spark Connect server, set at startup. A client cannot add them, which makes the server config a contract rather than a per-job choice.
• The client is a pip install with no Spark, no Java and no Iceberg jar. Everything here was driven from one.
• DDL with hidden partitioning, INSERT, UPDATE, MERGE, DELETE, writeTo, the metadata tables and CALL procedures all work unchanged over gRPC.
• SQL time travel works over Connect too, in all three syntaxes, as long as Iceberg’s extension is the only one loaded.
• Add Hudi’s session extension to the same server and every SQL time-travel syntax starts failing with [INTERNAL_ERROR] Found the unresolved operator: 'TimeTravelRelation. It is one-directional: Hudi keeps working, Iceberg loses time travel, and the snapshot-id reader option keeps working either way.

Iceberg is not part of Spark. It arrives as a runtime jar, installs a Catalyst extension and registers a catalog. Those are session-level concerns, and Spark Connect moves the session behind a gRPC connection, so the first question is which half of the setup goes where.

The second question is more interesting, and it is the reason this post exists. When something fails through Spark Connect, the transport is the most attractive suspect and usually the wrong one. This post runs Iceberg’s full surface from a thin client, and the single thing that breaks turns out to be caused neither by Spark Connect nor by Iceberg, but by another format’s session extension sharing the server. Finding that took two control experiments, and the first one, run on its own, produced a confident wrong answer that this post had published.

Architecture: where Iceberg has to live

A classic session keeps the SparkSession, the analyzer and the catalog in your process, which is why your own spark-submit line can install Iceberg. Spark Connect splits that:

flowchart TB
  subgraph C["client, your laptop"]
    A["DataFrame and SQL calls"] --> B["unresolved plan<br/>as protobuf"]
  end
  B -->|"gRPC, port 15002"| S
  subgraph S["Spark Connect server, the driver"]
    D["analyzer plus SparkCatalog"] --> E["Catalyst optimiser plus<br/>IcebergSparkSessionExtensions"]
    E --> F["physical plan"]
  end
  F --> X["executors, Iceberg runtime on the classpath"]
  X -->|"Arrow batches"| A

The client cannot resolve iceberg.lake.trips: it has no catalog and no notion of an Iceberg table. It ships the operation and the server works out what it means. Three things therefore have to be on the server, and all three are fixed when it starts:

spark.conf.set from a client reaches the server for runtime SQL configs, but spark.sql.extensions is read once while the session is built, before any client connects. A client can read it, which is the quickest way to tell a misconfigured server from a bad query.

Starting the server

$SPARK_HOME/sbin/start-connect-server.sh \
  --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.iceberg.type=hadoop \
  --conf spark.sql.catalog.iceberg.warehouse=/tmp/warehouse/iceberg \
  --master "local[4]"

Swap --packages for --jars with an absolute path when the host cannot reach Maven Central; both forms were run for this post. The coordinate carries two compatibilities that fail hard rather than warn:

Spark 4 is Scala 2.13 only, so a _2.12 runtime never loads.

Two notes on the catalog. type=hadoop keeps the example self-contained by putting the pointer in the warehouse directory, and it gives no atomic commits across engines, so use hive or rest for anything shared. And the catalog name you choose becomes the first part of every table name, so iceberg here means tables are iceberg.lake.trips.

The extension class is org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions, which is where you would expect it, under the project’s own package. It can still be confirmed from the jar rather than from memory:

unzip -l iceberg-spark-runtime-4.0_2.13-1.11.0.jar | grep IcebergSparkSessionExtensions
#    9663  org/apache/iceberg/spark/extensions/IcebergSparkSessionExtensions.class

A wrong class name does not stop the server. It logs a warning and starts, and then every MERGE fails later for reasons that look unrelated, so grep the server log for Cannot use before debugging your SQL.

Connecting the client

# pin the client to the version your server runs
pip install "pyspark[connect]==4.0.2"

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
print(spark.conf.get("spark.sql.extensions"))
# org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

No Java, no SPARK_HOME, no Iceberg jar. spark.version reports the server’s version, not the client’s, so it is not evidence about your local install; use pyspark.__version__ for that.

The pin is deliberate. An unpinned install takes whatever is newest on PyPI, and a client newer than the server runs SQL happily and then fails on the first createDataFrame with SQL_CONF_NOT_FOUND for spark.sql.session.localRelationSizeLimit, which it expects the server to have. An older client is tolerated; a newer one is not.

The write and read surface

Hidden partitioning in a plain CREATE TABLE is itself proof the extension loaded, since days(started_at) is Iceberg syntax rather than Spark syntax:

spark.sql("CREATE NAMESPACE IF NOT EXISTS iceberg.lake")

spark.sql("""
  CREATE TABLE iceberg.lake.trips (
    trip_id    BIGINT,
    rider_id   BIGINT,
    city       STRING,
    fare       DECIMAL(10,2),
    started_at TIMESTAMP)
  USING iceberg
  PARTITIONED BY (days(started_at))""")

spark.sql("""
  INSERT INTO iceberg.lake.trips VALUES
    (1, 101, 'Hyderabad', 245.50, TIMESTAMP '2026-09-01 08:15:00'),
    (2, 102, 'Bengaluru', 180.00, TIMESTAMP '2026-09-01 09:30:00'),
    (3, 103, 'Hyderabad', 320.75, TIMESTAMP '2026-09-02 07:45:00')""")

Note that the positional INSERT is fine here. Iceberg keeps the declared column order because its partitioning is derived from a column rather than stored as one, which is a real difference from Hudi, where PARTITIONED BY moves the partition column to the end of the schema and breaks positional inserts.

Row-level DML, all of it extension-provided:

spark.sql("UPDATE iceberg.lake.trips SET fare = 250.00 WHERE trip_id = 1")

spark.sql("""
  MERGE INTO iceberg.lake.trips t
  USING (SELECT 4L AS trip_id, 104L AS rider_id, 'Pune' AS city,
                CAST(99.00 AS DECIMAL(10,2)) AS fare,
                TIMESTAMP '2026-09-03 10:00:00' AS started_at) s
    ON t.trip_id = s.trip_id
  WHEN MATCHED     THEN UPDATE SET t.fare = s.fare
  WHEN NOT MATCHED THEN INSERT *""")

spark.sql("DELETE FROM iceberg.lake.trips WHERE trip_id = 2")

The v2 DataFrame writer works too:

from decimal import Decimal
from pyspark.sql import functions as F

new_trip = (spark.createDataFrame(
        [(5, 105, "Chennai", Decimal("75.25"), "2026-09-04 11:00:00")],
        "trip_id bigint, rider_id bigint, city string, "
        "fare decimal(10,2), started_at string")
    .withColumn("started_at", F.to_timestamp("started_at")))

new_trip.writeTo("iceberg.lake.trips").append()

Decimal("75.25") rather than 75.25 is deliberate, and the reason is in the limitations section below.

Metadata tables and procedures, which are the parts people expect to break over gRPC:

spark.table("iceberg.lake.trips.snapshots").count()
spark.table("iceberg.lake.trips.files").count()
spark.table("iceberg.lake.trips.history").count()
spark.sql("CALL iceberg.system.rewrite_data_files(table => 'lake.trips')").collect()

From a clean warehouse the whole sequence gives:

count after INSERT                      3
fare for trip_id 1 after UPDATE         250.00
count after MERGE and DELETE            3
count after writeTo append              4
read via format("iceberg")              4
iceberg.lake.trips.snapshots            5 rows
iceberg.lake.trips.files                4 rows
iceberg.lake.trips.history              5 rows
CALL rewrite_data_files                 returns a row, 0 files rewritten

Five snapshots for five write operations, and rewrite_data_files rewriting nothing is correct on a four-row table whose files are all under the target size. A procedure that returns a row without doing work is not a failure.

Time travel, and the extension that breaks it

Start with the good news, on a server whose only extension is Iceberg’s. All three time-travel syntaxes work through the thin client. Derive the snapshot id rather than hardcoding one, and derive it by commit time, because Iceberg snapshot ids are random 64-bit values and are not ordered chronologically:

oldest = spark.sql("""
    SELECT snapshot_id FROM iceberg.lake.trips.snapshots
    ORDER BY committed_at LIMIT 1""").collect()[0][0]

spark.sql("SELECT count(*) FROM iceberg.lake.trips VERSION AS OF %d" % oldest)
spark.sql("SELECT count(*) FROM iceberg.lake.trips FOR SYSTEM_VERSION AS OF %d" % oldest)
spark.read.option("snapshot-id", oldest).format("iceberg").load("iceberg.lake.trips").count()

VERSION AS OF oldest             3
FOR SYSTEM_VERSION AS OF oldest  3
snapshot-id reader option        3
current table                    4

min(snapshot_id) is the wrong way to find the oldest snapshot and it is an easy mistake to make, because it looks like an id sequence. On one test table the minimum id was the third of five snapshots, so the query silently returned the wrong point in history rather than failing. Order by committed_at.

Passing an id that does not exist gives a clear error rather than a wrong answer, which is worth knowing when you are debugging:

IllegalArgumentException: Cannot find snapshot with ID 2136647905244113308

The failure, and the diagnosis that found it

Now the part that cost me a published mistake. On a server that loads both Hudi’s and Iceberg’s session extensions, every SQL time-travel syntax fails:

spark.sql("SELECT count(*) FROM iceberg.lake.trips VERSION AS OF %d" % oldest)

SparkException: [INTERNAL_ERROR] Found the unresolved operator: 'TimeTravelRelation
SQLSTATE: XX000

An INTERNAL_ERROR naming an unresolved operator looks exactly like a transport that cannot serialise a plan node, and Spark Connect is the newest thing in the stack, so it is the natural suspect. It is not the culprit, but proving that takes two controls rather than one, and I originally ran only the first.

Running it in a classic local session rules out Connect. That much I did, and the query failed there too, so I concluded the fault was an Iceberg and Spark 4.0 analyzer gap. That conclusion was wrong, because my classic session was configured from the same runbook as the server: it loaded both extensions. The control isolated the transport and left the real variable untouched.

Varying the extension list as well gives a 2x2 that settles it:

The transport is irrelevant. Loading Hudi’s HoodieSparkSessionExtension alongside Iceberg’s breaks Iceberg’s SQL time travel, in a classic session just as much as through Connect.

Two further facts make it easier to live with. The interference is one-directional: on that same both-extension server, Hudi’s own DML, MERGE, procedures, incremental reads and as.of.instant time travel all worked. And the Iceberg snapshot-id reader option keeps working with both extensions loaded, because it never builds a TimeTravelRelation in the first place.

So the practical advice is concrete rather than “wait for a fix”:

• If you need Iceberg SQL time travel, give Iceberg a server whose spark.sql.extensions contains only IcebergSparkSessionExtensions.
• If you must share one server with Hudi, use the snapshot-id reader option, which is unaffected.

The wider lesson is about the shape of the control experiment, not about Iceberg. Reproducing a failure without Spark Connect proved only that Connect was not the cause; it said nothing about which of the remaining variables was. A control that holds the suspect constant is not a control at all, and copying the full config into the repro is exactly how that happens.

What the thin client cannot do

spark.sparkContext

PySparkAttributeError: [JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `sparkContext` is not
supported in Spark Connect as it depends on the JVM.

spark.range(10).rdd.map(lambda r: r.id).collect()

PySparkNotImplementedError: [NOT_IMPLEMENTED] rdd is not implemented.

Neither matters for Iceberg work, which goes through SQL and DataFrames. The one that actually costs time is a Python float in a decimal column, which a classic session coerces and the Connect client rejects:

spark.createDataFrame(
    [(9, 109, "Kochi", 75.25, "2026-09-05 12:00:00")],
    "trip_id bigint, rider_id bigint, city string, "
    "fare decimal(10,2), started_at string")

AssertionError

That is the whole message. The traceback lands in convert_decimal in pyspark/sql/conversion.py, which does assert isinstance(value, decimal.Decimal). Any table with a money column hits it on the first createDataFrame, and a bare AssertionError reads like a client bug rather than a type error.

Production notes

• Treat the server config as a contract. The jar, the extension and the catalogs are fixed at startup, so adding a catalog means a restart. Plan for more than one server if teams need different catalog sets.
• Use hive or rest catalog types for shared tables. hadoop is fine for a self-contained example and gives no cross-engine commit atomicity.
• Confirm the extension from the client with spark.conf.get("spark.sql.extensions") before debugging a rejected MERGE.
• Give Iceberg a server whose only extension is Iceberg’s if you need SQL time travel. Adding HoodieSparkSessionExtension to the same session breaks it.
• Vary one thing at a time when reproducing a failure. A classic-session repro rules out Spark Connect and nothing else; if you copy the whole config into the repro, the actual cause comes along with it.
• Find the oldest snapshot by committed_at, never by min(snapshot_id). Snapshot ids are random 64-bit values, so the minimum is not the earliest and the query returns the wrong history without erroring.
• Pass Decimal for decimal columns in createDataFrame, and prefer CAST(... AS DECIMAL(10,2)) inside SQL.

Frequently asked questions

Does the client need the Iceberg jar? No. The client sends a plan naming a table and the server resolves it, so a virtualenv with pyspark is the entire client-side dependency. That also removes the jar-version skew that usually makes onboarding painful.

Can one server serve Iceberg and Hudi together? Mostly, with one measured exception. spark.sql.extensions takes a comma-separated list, Hudi takes over spark_catalog while Iceberg gets a named catalog, and both formats’ DDL and DML work. But loading Hudi’s extension breaks Iceberg’s SQL time travel, as shown above, so a shared server costs you that one feature and you fall back to the snapshot-id reader option. Hudi loses nothing in the arrangement. The Hudi side is covered in Apache Hudi through Spark Connect.

Are metadata tables really available over gRPC? Yes, and they are just tables as far as the client is concerned. spark.table("iceberg.lake.trips.snapshots") is resolved entirely on the server, so snapshots, files, history and the rest all work.

Why does rewrite_data_files report zero files? Because there was nothing worth rewriting. On a small table all files are already under the target size, so the procedure returns a row with zeroes in it. Judge maintenance procedures by the table’s file layout afterwards, not by whether the call returned.

Is SQL time travel broken because of Spark Connect? No, and it is not broken in general either. It works through Connect when Iceberg’s extension is the only one loaded. It breaks when Hudi’s extension shares the session, in a classic session just as much as through Connect, so the variable is the extension list rather than the transport.

Conclusion

Iceberg through Spark Connect is close to uneventful, which is the interesting result. Hidden partitioning, row-level MERGE, the metadata tables, the stored procedures and time travel all cross a gRPC boundary without special handling, because they are resolved server-side where the Iceberg runtime already lives, and the client stays a pip install and a URL. What Connect takes away is the per-job escape hatch: the jar, the extension and the catalog set are decided once when the server starts, so a new catalog is an operational change rather than a config line in someone’s notebook. For a shared lakehouse that is usually the behaviour you wanted anyway.

The part worth keeping is how the one failure was diagnosed, because the first attempt got it wrong. [INTERNAL_ERROR] Found the unresolved operator is about as suggestive of a broken transport as an error message gets, so the first control was a classic local session, and it reproduced the failure. That was enough to clear Spark Connect and it felt like enough to name a cause, so this post originally blamed an Iceberg and Spark 4.0 analyzer gap.

It was not that. The classic session had been configured by copying the server’s own config, which loaded Hudi’s session extension alongside Iceberg’s, and that is the actual cause. Varying the extension list rather than the transport gives a 2x2 where the two Iceberg-only cells pass and the two both-extension cells fail, whether Connect is involved or not. A control that changes one variable while silently carrying the real one along is not a control, and copying config into a repro is the ordinary way that happens. The advice changes with the diagnosis: not “wait for a fix” but “do not put Hudi’s extension on a server that owes anyone Iceberg SQL time travel”.

References

• Spark Connect overview for the architecture and the client APIs
• Iceberg Spark configuration for catalog types and the extension class
• Iceberg Spark queries for the metadata tables and the time-travel read options
• conversion.py, whose convert_decimal is the assertion behind the empty AssertionError
• Apache Iceberg architecture for the snapshot, manifest and delete-file machinery underneath all of this

Trademarks

Apache Spark, Apache Iceberg, Apache Hudi, Apache Hive, Apache Parquet and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.