All posts

Life of a write query in Iceberg: three files, one pointer swap

Every Iceberg commit writes a manifest, a manifest list and a metadata file, then swaps one pointer. Two writers hammering the same table produced a linear snapshot chain with no lost rows and no errors. Here is the sequence, and what optimistic concurrency actually does.

7 min read Iceberg

TL;DR

  • Every commit writes exactly three metadata files — a manifest, a manifest list, a metadata.json — then atomically swaps the catalog pointer. Measured across three appends: 2 → 5 → 8 → 11 metadata files.
  • The swap is a compare-and-swap: commit only if the metadata I started from is still current. That is the entire concurrency control.
  • Two threads writing 6 rows each concurrently produced 13 snapshots, 13 rows, A=6 and B=6 — no lost writes, no surfaced errors. Retries were absorbed internally.
  • The snapshot chain stayed linear: every snapshot has a parent, so history is a sequence even though writes were parallel.
  • Manifest reuse keeps this affordable — a commit adding one file reports added-data-files=1 while total-records accumulates.

The sequence

A write is four steps, and only the last one is visible to readers.

flowchart TB
    P["1. plan: read current metadata<br/>schema, spec, current snapshot"] --> W["2. executors write data files<br/>and any delete files"]
    W --> M["3. write manifest, manifest list,<br/>new metadata.json"]
    M --> CAS{"4. catalog compare-and-swap<br/>is my base still current?"}
    CAS -->|yes| OK["commit: new snapshot is live"]
    CAS -->|no| R["retry: re-plan from the new base"]
    R --> M

Nothing is visible until step 4. Data files written in step 2 exist on storage but belong to no snapshot, so no reader can see them. A job that dies between steps 2 and 4 leaves files nobody references — orphans, cleaned up later by remove_orphan_files, and harmless apart from the storage.

That is also why a failed write never corrupts a table. There is no partial state to observe.

What each commit costs

Creating a partitioned table and appending three times, counting files in the metadata directory:

after CREATE, before any data   metadata_files=2    data_files=0
after append #1                 metadata_files=5    data_files=1
after append #2                 metadata_files=8    data_files=2
after append #3                 metadata_files=11   data_files=3

Three per commit, every commit. One manifest listing the data files this commit added, one manifest list for the new snapshot, one metadata.json recording the new table state. The two before any data are the initial metadata.json and the catalog’s version hint.

This is the number that makes streaming tables accumulate metadata: a commit per minute is three metadata files per minute, regardless of how many rows moved.

The snapshot log shows what each commit recorded:

snap=3527536884859544448 parent=None                op=append  added-data-files=1 total-records=3000
snap=2869732689965030704 parent=3527536884859544448 op=append  added-data-files=1 total-records=6000
snap=4093333610368405196 parent=2869732689965030704 op=append  added-data-files=1 total-records=9000

added-data-files stays 1 while total-records climbs. Each commit describes only its own addition; the manifest list points at the previous snapshot’s manifests rather than copying them. Snapshots share metadata, which is what makes keeping a thousand of them affordable.

The operation field names what happened, and it is the record you consult later: append for added data, overwrite when files were replaced, delete when a delete file was written.

Optimistic concurrency, measured

Iceberg does not lock the table. Every writer plans against a snapshot, does its work, and at commit time asks the catalog to swap the pointer only if the snapshot it started from is still current. If another writer committed meanwhile, the swap fails and this writer re-plans and retries.

Two threads, each inserting 6 rows into the same table, running concurrently:

concurrent writer errors: none (all commits succeeded, retries absorbed)
final rows   = 13
snapshots    = 13
per-writer   = [('A', 6), ('B', 6), ('seed', 1)]
parent chain is linear: all snapshots after the first have a parent

Three things worth drawing out.

No rows were lost. Both writers landed all six of their inserts. This is the property that a directory-based table cannot offer, where two writers putting files into the same path simply coexist and hope.

No errors surfaced. Conflicts happened — twelve commits contending on one pointer will collide — and the retries were handled inside the client. Application code saw success.

History is linear. Thirteen snapshots, each with a parent. Concurrent writes did not create a branching history; they were serialized by the compare-and-swap into one sequence. That linearity is what makes time travel meaningful.

When retries are not enough

Retries are bounded, and a conflict that cannot be resolved surfaces as CommitFailedException. Two situations produce it in practice.

Heavy contention: many writers committing faster than retries can settle. The fix is fewer, larger commits — batching a minute of data into one commit rather than sixty.

A genuine conflict: two writers modifying the same files, where re-planning cannot simply be re-applied. Appends rarely conflict this way because they only add files. UPDATE, DELETE and MERGE against overlapping data can, because the rows one writer is rewriting are the rows another is deleting.

There is a third outcome worth knowing: CommitStateUnknownException, when the catalog call did not return a clear answer. Do not clean up after this one. The commit may have succeeded, and deleting its files would corrupt a live table.

Isolation, precisely

Readers get snapshot isolation for free. A query resolves the current metadata once and reads that snapshot throughout, so a commit landing mid-query is invisible to it. There is no read lock and no torn read.

Writers get serializable behaviour on the commit itself, since only one compare-and-swap can succeed against a given base. What that does not give you is serializability of the computation — a writer that read the table, decided something, and then wrote is not protected against the table changing in between unless the commit itself conflicts.

Row-level operations tighten this with a write.*.isolation-level property. serializable makes a MERGE fail if any concurrent commit touched the data it scanned; snapshot isolation only fails if the specific files it rewrote changed. The default is the stricter one, and relaxing it is a decision to make explicitly.

What the write path does not do

It does not reclaim space. An overwrite writes new files and leaves the old ones referenced by the previous snapshot. Storage is reclaimed by expire_snapshots and nothing else.

It does not compact. Each commit writes what it was given. A hundred small appends produce a hundred small files, and rewrite_data_files is a separate job.

It does not reorder. Data is written in the order it arrives unless a sort order is declared, and unsorted data makes column-bounds pruning useless at read time.

Common misconceptions

“Iceberg locks the table to write.” It never locks. It retries a compare-and-swap.

“Concurrent writers will lose data.” Measured: two writers, 6 rows each, all 12 landed.

“A failed write leaves the table inconsistent.” It leaves unreferenced files. The table is exactly as it was.

“Every commit rewrites the metadata.” It writes a new metadata file that reuses the previous snapshot’s manifests.

“Retry means my job is slow.” Retries are cheap when commits are infrequent and expensive when they are constant. Commit frequency is the lever.

A model worth keeping

A write is: read the current state, write data files nobody can see yet, describe them in three metadata files, then swap one pointer if nothing changed underneath.

Everything else follows. Atomicity is the swap. Isolation is resolving the pointer once. Concurrency is the retry. Metadata growth is three files per commit, which is why how often you commit matters more than how much you write.

References

Trademarks

Apache Iceberg, Apache Spark, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee