Apache Iceberg architecture: what actually happens when you commit

• Architecture: the metadata tree
• Why is the catalog part of the table?
• What does a commit actually do?
• How does a query find its files?
• Why is partitioning hidden?
• How is a row deleted without rewriting the file?
• Reading it from a running table
• When is a simpler choice better?
• Production tips
• Frequently asked questions
• Conclusion
• References
• Trademarks

TL;DR

• A table is an immutable tree: a catalog pointer to metadata.json, which names a snapshot, which points at a manifest list, which points at manifests, which list data files. Nothing already written is ever mutated.
• The catalog is part of the table, not a lookup service. Nothing on storage says which metadata.json is current, so commit atomicity is a property of the catalog.
• TableOperations.commit mandates a compare-and-swap and a distinct CommitStateUnknownException, which is the difference between a commit that failed and one whose outcome is unknown. That distinction is why orphan files exist.
• Planning is a sequence of prunes over metadata: manifest list by partition range, manifests by partition value, then per-file column bounds. No directory is ever listed.
• Hidden partitioning and partition evolution both work because partition values live in manifests rather than in directory names.

The first Iceberg surprise is usually operational rather than conceptual. The table works, queries are fast, and then two writers collide and one fails with something about stale metadata, or a VACUUM-shaped cleanup deletes files a reader still wanted. Neither makes sense if you think of Iceberg as a file layout, because neither is about files.

This is the map. It follows one commit from the statement to the catalog swap, and one query from a filter to the files it actually opens. By the end you should be able to look at a table directory and say what is current, and reason about what two concurrent writers will do to each other.

Written against the latest Iceberg release, 1.11.0 as of now. Every class, property and default below was read from the apache-iceberg-1.11.0 tag rather than recalled. Assumed knowledge: comfort with SQL and a query engine. No Iceberg internals knowledge is assumed.

Architecture: the metadata tree

Ask Iceberg “which files make up this table right now” and the answer walks a tree, every level of which is an immutable file.

flowchart LR
  CAT[("catalog<br/>one pointer")] --> MD["v2.metadata.json<br/>schemas, specs, snapshots"]
  MD --> SNAP["current snapshot<br/>7241925443479918015"]
  SNAP --> ML["manifest list<br/>snap-...avro"]
  ML --> M1["manifest m0<br/>partition ranges, counts"]
  ML --> M2["manifest m1"]
  M1 --> D1["data files"]
  M1 --> DEL["delete files"]
  M2 --> D2["data files"]

On disk:

s3a://lakehouse-prod/warehouse/trips/
├── metadata/
│   ├── v1.metadata.json
│   ├── v2.metadata.json                              <- current, per the catalog
│   ├── snap-7241925443479918015-1-a1c2....avro       <- manifest list
│   ├── a1c2f3b4-....-m0.avro                         <- manifest
│   └── a1c2f3b4-....-m1.avro
└── data/
    ├── city_id=sf/started_at_day=2026-09-15/00000-0-a1c2f3b4-....parquet
    └── city_id=nyc/started_at_day=2026-09-15/00000-1-a1c2f3b4-....parquet

Two things are worth pausing on. The data/ directory is laid out by partition for human convenience, but the engine never relies on it: partition values come from the manifest. And nothing in that listing says which metadata.json is current. Open the directory without a catalog and you cannot answer the most basic question about the table.

Why is the catalog part of the table?

Because the commit is a pointer swap, and something has to make that swap atomic.

TableOperations.commit is an interface with unusually specific documentation, and it is worth reading because it defines the guarantee:

Implementations must check that the base metadata is current to avoid overwriting updates. Once the atomic commit operation succeeds, implementations must not perform any operations that may fail because failure in this method cannot be distinguished from commit failure.

That first sentence is optimistic concurrency stated as a contract: a writer prepares a new metadata.json from the base it read, and the commit succeeds only if the base is still current. If another writer moved the pointer first, this one fails and retries from the new base.

The second half is subtler and explains a real operational problem:

Implementations must throw a CommitStateUnknownException in cases where it cannot be determined if the commit succeeded or failed. For example if a network partition causes the confirmation of the commit to be lost. This is important because downstream users of this API need to know whether they can clean up the commit or not, if the state is unknown then it is not safe to remove any files.

A failed commit and an unknown commit are different states. After a clean failure, the files the writer wrote are garbage and can be deleted. After an unknown outcome, they may be referenced by a snapshot that did commit, so deleting them would corrupt the table. Iceberg therefore leaves them, which is precisely why remove_orphan_files exists and why it takes a conservative older_than.

That last row is the caveat to carry. Two Iceberg tables can look byte-identical on storage and have different correctness guarantees under concurrent writes, because the guarantee lives in the catalog rather than in the files.

What does a commit actually do?

flowchart TB
  W["writer produces<br/>new data files"] --> M["write a new manifest<br/>listing them"]
  M --> ML["write a new manifest list,<br/>reusing unchanged manifests"]
  ML --> MJ["write a new metadata.json<br/>with the new snapshot"]
  MJ --> CAS{"catalog compare-and-swap:<br/>is the base still current?"}
  CAS -->|"yes"| OK["commit succeeds<br/>the new snapshot is live"]
  CAS -->|"no"| RETRY["CommitFailedException<br/>re-plan from the new base"]
  CAS -.->|"outcome unknown"| UNK["CommitStateUnknownException<br/>leave the files alone"]
  RETRY -.-> M

The step that makes snapshot history affordable is manifest reuse. A commit that adds one file writes a new manifest and a new manifest list, but the manifest list points at every unchanged manifest from the previous snapshot rather than copying it. Successive snapshots share most of their metadata, so keeping a thousand snapshots costs metadata proportional to what changed, not to the size of the table.

Retention is therefore not free but is cheap, and it is bounded by history.expire.max-snapshot-age-ms, which defaults to 5 days.

How does a query find its files?

Planning is a sequence of prunes, and each one reads metadata rather than data.

flowchart LR
  Q["query with a filter<br/><i>city_id = 'sf' AND fare > 50</i>"] --> A["manifest list<br/>skip whole manifests<br/>by partition range"]
  A --> B["manifests<br/>skip files by<br/>partition value"]
  B --> C["column bounds<br/>skip files whose<br/>min/max cannot match"]
  C --> D["the files actually opened"]

The manifest list carries partition_spec_id, added_files_count, existing_files_count, deleted_files_count, row counts and per-partition lower_bound and upper_bound. That is enough to discard an entire manifest, and therefore hundreds of files, without opening it.

Notice what is absent: no step lists a directory. This is the structural difference from Hive-style tables, where planning is a LIST per partition and cost grows with partition count. Iceberg’s planning cost grows with the number of manifests, which is what rewrite_manifests exists to control.

Column bounds only prune if the data is sorted enough for them to be narrow. Unsorted data gives every file a wide min and max, the bounds overlap, and nothing is skipped. That is why WRITE ORDERED BY and a sorting compaction are not cosmetic: they are what makes the third prune work at all.

Why is partitioning hidden?

Because partition values live in manifests rather than in path names, which makes two things possible that are hard to retrofit elsewhere.

Every SQL block below runs in a session with a catalog named prod. On Spark that is four settings, and prod is the name you chose, not a reserved word:

spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.prod=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.prod.type=hive \
  --conf spark.sql.catalog.prod.uri=thrift://metastore.internal:9083 \
  --conf spark.sql.catalog.prod.warehouse=s3a://lakehouse-prod/warehouse

type=hive is the line that picks the commit guarantee discussed above. Swap it for type=rest plus a uri to use a REST catalog instead.

Hidden partitioning. You declare a transform on a column, and queries filtering that column prune automatically:

CREATE TABLE prod.db.trips (
  trip_id      STRING,
  rider_id     STRING,
  fare_amount  DECIMAL(10,2),
  started_at   TIMESTAMP,
  city_id      STRING
) USING iceberg
PARTITIONED BY (city_id, days(started_at));

-- prunes to a day without naming a partition column
SELECT count(*) FROM prod.db.trips
WHERE started_at >= TIMESTAMP '2026-09-15 00:00:00'
  AND started_at <  TIMESTAMP '2026-09-16 00:00:00';

There is no dt column in the schema, so there is no way for a query to forget to filter on it, and no correlation for a user to remember.

Partition evolution. Because each manifest records the partition_spec_id it was written under, the spec can change and old files keep working:

ALTER TABLE prod.db.trips ADD PARTITION FIELD bucket(16, rider_id);
ALTER TABLE prod.db.trips DROP PARTITION FIELD days(started_at);

Files written before the change keep their original spec; files after use the new one; a scan reads both. In a directory-partitioned table that change is a full rewrite.

Schema evolution works on the same principle one level down: columns carry assigned IDs, so a rename is a metadata edit and data files are never touched.

How is a row deleted without rewriting the file?

Two strategies exist. Copy-on-write rewrites every data file that contains an affected row, so readers stay simple. Merge-on-read writes a small file recording the deletion and leaves the data file alone, so writes stay cheap and the reader does the reconciling.

write.delete.mode, write.update.mode and write.merge.mode all default to copy-on-write in 1.11.0, which surprises people: a fresh table rewrites whole data files on DELETE, UPDATE and MERGE.

ALTER TABLE prod.db.trips SET TBLPROPERTIES (
  'write.delete.mode' = 'merge-on-read',
  'write.update.mode' = 'merge-on-read',
  'write.merge.mode'  = 'merge-on-read'
);

Under merge-on-read, a delete becomes a file rather than a rewrite:

Merge-on-read buys low write latency at the cost of read-side reconciliation and a compaction job you must operate. The compaction here is rewrite_position_delete_files, and a growing count in the delete_files metadata table is the signal it is not running.

Reading it from a running table

Every claim above is queryable, because Iceberg exposes its own metadata as tables. MetadataTableType lists sixteen; these three answer most questions.

-- What is the history, and how big is the metadata?
SELECT committed_at, snapshot_id, operation FROM prod.db.trips.snapshots
ORDER BY committed_at DESC LIMIT 10;

-- Small files and skew, per partition
SELECT partition, file_count, record_count
FROM prod.db.trips.partitions ORDER BY file_count DESC LIMIT 20;

-- Is merge-on-read accumulating deletes faster than compaction clears them?
SELECT count(*) AS delete_files FROM prod.db.trips.delete_files;

When is a simpler choice better?

Append-only, single-engine, small tables that never need a row corrected. Plain partitioned Parquet with a metastore is less machinery and will not surprise you. Iceberg starts paying for itself when you need atomic commits across many files, row-level mutation, time travel, or planning that does not scale with partition count.

Workloads that are keyed upserts at high frequency. Iceberg has no record index: a MERGE plans a join whose cost scales with how much of the table it touches. A format built around record identity will do that work more cheaply.

Anywhere you cannot run a catalog with an atomic swap. Without one you have the file layout but not the guarantee, which is the part worth having.

Production tips

• Choose the catalog as deliberately as the format. It supplies the commit guarantee; a filesystem catalog on storage without atomic rename does not.
• Schedule rewrite_data_files before planning gets slow, not after. Compaction is cheaper run often.
• Declare a write order with WRITE ORDERED BY if the table is queried with predicates, otherwise column bounds prune nothing.
• Expire snapshots deliberately. Long enough for time travel and incremental reads, short enough that storage and erasure obligations stay bounded.
• Run remove_orphan_files conservatively, with a generous older_than. Files from an unknown-outcome commit may still be referenced.
• Know that the row-level defaults are copy-on-write. If your workload deletes often, set the three modes explicitly.
• Watch delete_files count on merge-on-read tables, not query duration.

Frequently asked questions

Can I point an engine at the directory and skip the catalog? Some engines allow it for read-only access using the newest metadata.json they can find, but you lose the commit guarantee and the ability to write safely. The catalog is not a convenience layer.

What is the difference between a snapshot and a metadata file? A metadata.json holds the whole table state including the list of snapshots. A snapshot is one entry inside it, naming a manifest list. One commit writes one new metadata.json containing one new snapshot.

Why does my table have files that no query reads? Either an older snapshot still references them and retention has not expired it, or they came from a commit whose outcome could not be determined. The second kind is what remove_orphan_files is for.

Is a commit conflict a bug? No. Under optimistic concurrency, a writer that loses the race does its work and then fails, which is the system behaving correctly. The pathology is the opposite: two writers both succeeding and one silently losing rows, which is what a catalog without an atomic swap allows.

How does format version affect this? 1.11.0 creates tables at version 2 and reads and writes up to version 4. Version 3 adds deletion vectors and is the minimum for row lineage. The version only moves forward, so check every reader supports it first.

Conclusion

Back to the two writers colliding, and the cleanup that deleted files someone still wanted. Neither is about files. The first is optimistic concurrency doing its job, and the second is the difference between a commit that failed and one whose outcome was never determined, a distinction the commit contract makes explicit and most descriptions of Iceberg skip.

The idea that organises the rest is that everything is immutable and addressed through one pointer. Snapshots are cheap because manifests are shared rather than copied. Planning is fast because partition values and column bounds live in metadata that can be read without touching data. Partition evolution works because a manifest records the spec it was written under. Time travel is not a feature that was added; it is what you get when nothing is ever overwritten.

The cost is the catalog. Iceberg asks you to run a service that can do an atomic compare-and-swap, and in exchange it gives you a table that many engines can read safely at once. If your workload is high-frequency keyed upserts, a format with a record index will do that specific job more cheaply. If it is large batches, many readers and layout decisions you expect to revise, this architecture is hard to beat.

References

• Iceberg table specification for snapshots, manifests, delete files and format versions
• TableOperations.java at apache-iceberg-1.11.0, the commit contract quoted above
• TableProperties.java at apache-iceberg-1.11.0 for every property and its default
• Spark procedures for the maintenance surface this post describes
• Apache Iceberg on Spark: the complete cheat sheet for the config surface this post explains

Trademarks

Apache Iceberg, Apache Spark, Apache Flink, Apache Hive, Apache Parquet, Apache Avro, Apache Hudi and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Delta Lake is a trademark of the Linux Foundation.