Apache Iceberg on Spark: the complete cheat sheet

• 1. Core architecture
• 2. Getting started
• 3. Hidden partitioning and transforms
• 4. Row-level operations
• 5. Format versions
• 6. Metadata tables
• 7. Time travel, branches and tags
• 8. Schema evolution
• 9. Procedures
• 10. Maintenance
• 11. Key table properties
• 12. Sorting, clustering and file sizing
• 13. Diagnosing a table
• 14. Concurrency
• Conclusion
• References

TL;DR

• Iceberg describes a table as an immutable tree: a catalog pointer to metadata.json, which names a snapshot, which points at a manifest list, which points at manifests, which list data files.
• The catalog is not optional. Nothing inside the table says which metadata.json is current, so commit atomicity is a property of the catalog, not of the format.
• The default that surprises everyone: write.delete.mode, write.update.mode and write.merge.mode all default to copy-on-write, so a fresh table rewrites whole files on DELETE, UPDATE and MERGE.
• Hidden partitioning means you declare days(started_at) and queries filtering on started_at prune automatically, with no separate dt column to remember.
• New tables are format version 2 by default (DEFAULT_TABLE_FORMAT_VERSION), and 1.11.0 reads and writes up to version 4.

This is the Iceberg companion to the Hudi cheat sheet, in the same shape: every section is a lookup, each row gives you the concept, a literal example you can copy, and the one or two sentences that tell you whether you want it.

Written against the latest Iceberg release, 1.11.0 as of now, on Spark. Every property, default, enum value and procedure name below was read from the apache-iceberg-1.11.0 tag rather than recalled. Diagrams captioned Source: Apache Iceberg documentation come from the project’s own site assets, © The Apache Software Foundation, used under the Apache License 2.0; the rest are my own. Where a claim is specific to a version, the version is named.

1. Core architecture

flowchart LR
  CAT["catalog<br/>one pointer"] --> MD["v2.metadata.json<br/>schema, specs, snapshots"]
  MD --> SNAP["current snapshot<br/>7241925443479918015"]
  SNAP --> ML["manifest list<br/>snap-...avro"]
  ML --> M1["manifest m0<br/>partition ranges"]
  ML --> M2["manifest m1<br/>partition ranges"]
  M1 --> D1["data files"]
  M1 --> DEL["delete files"]
  M2 --> D2["data files"]

A commit writes a new metadata.json and swaps the catalog pointer. Nothing already written is mutated, which is what makes an old snapshot readable for as long as retention keeps it.

The specification’s own view of the same structure, showing how successive snapshots share manifests rather than duplicating them:

Source: Apache Iceberg documentation.

Sharing is the part worth noticing. A commit that adds one file writes a new manifest and a new manifest list, but reuses every manifest that did not change, which is why snapshot history costs metadata rather than data.

s3a://lakehouse-prod/warehouse/trips/
├── metadata/
│   ├── v1.metadata.json
│   ├── v2.metadata.json                              <- current, per the catalog
│   ├── snap-7241925443479918015-1-a1c2....avro       <- manifest list
│   ├── a1c2f3b4-....-m0.avro                         <- manifest
│   └── a1c2f3b4-....-m1.avro
└── data/
    ├── city_id=sf/00000-0-a1c2f3b4-....parquet
    └── city_id=nyc/00000-1-a1c2f3b4-....parquet

FileContent enumerates what a manifest entry can describe: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST and DELETE_MANIFEST.

Planning is a sequence of prunes rather than a directory listing, and each step reads metadata rather than data:

flowchart LR
  Q["query with<br/>a filter"] --> A["manifest list<br/>skip whole manifests<br/>by partition range"]
  A --> B["manifests<br/>skip files by<br/>partition value"]
  B --> C["column bounds<br/>skip files by<br/>min and max"]
  C --> D["the files<br/>actually scanned"]

2. Getting started

spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.prod=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.prod.type=hive \
  --conf spark.sql.catalog.prod.uri=thrift://metastore.internal:9083

CREATE TABLE prod.db.trips (
  trip_id      STRING,
  rider_id     STRING,
  fare_amount  DECIMAL(10,2),
  started_at   TIMESTAMP,
  city_id      STRING
) USING iceberg
PARTITIONED BY (city_id, days(started_at));

INSERT INTO prod.db.trips VALUES
  ('trip-1001', 'rider-8814f2', 24.50, TIMESTAMP '2026-09-15 08:12:04', 'sf'),
  ('trip-1002', 'rider-33c9a1', 11.75, TIMESTAMP '2026-09-15 08:19:47', 'sf');

SELECT * FROM prod.db.trips;

UPDATE prod.db.trips SET fare_amount = 27.00 WHERE trip_id = 'trip-1001';

DELETE FROM prod.db.trips WHERE trip_id = 'trip-1002';

MERGE INTO prod.db.trips t
USING trip_updates s ON t.trip_id = s.trip_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Note there is no record key to declare. Iceberg has no notion of row identity, so MERGE states the matching condition itself and plans a join to find the files.

3. Hidden partitioning and transforms

-- Evolve the layout: start daily, add a bucket, drop the old field
ALTER TABLE prod.db.trips ADD PARTITION FIELD bucket(16, rider_id);
ALTER TABLE prod.db.trips DROP PARTITION FIELD days(started_at);

-- The partition spec history is queryable
SELECT * FROM prod.db.trips.partitions;

Hidden partitioning is the point. The writer declares days(started_at) and the reader filters on started_at. There is no dt column to add to the schema, no risk of a query forgetting to filter on it, and no correlation for the user to remember. This is the capability that is hardest to retrofit in another format.

4. Row-level operations

flowchart TB
  subgraph CW["copy-on-write: the default for delete, update and merge"]
    direction LR
    C1["DELETE one row"] --> C2["read the data file"]
    C2 --> C3["write a new file<br/>without that row"]
    C3 --> C4["read = plain Parquet"]
  end

  subgraph MR["merge-on-read: opt in per operation"]
    direction LR
    M1["DELETE one row"] --> M2["write a delete file<br/>or deletion vector"]
    M2 --> M3["data file untouched"]
    M3 --> M4["read = data merged<br/>with deletes"]
    M3 --> M5["rewrite_position_delete_files<br/>later"]
  end

  CW ~~~ MR

-- All three default to copy-on-write; set them together or not at all
ALTER TABLE prod.db.trips SET TBLPROPERTIES (
  'write.delete.mode' = 'merge-on-read',
  'write.update.mode' = 'merge-on-read',
  'write.merge.mode'  = 'merge-on-read'
);

Note: TableProperties at 1.11.0 sets DELETE_MODE_DEFAULT, UPDATE_MODE_DEFAULT and MERGE_MODE_DEFAULT all to RowLevelOperationMode.COPY_ON_WRITE.modeName(). This is the single most counter-intuitive default in the format: a table you have not configured rewrites files on every DELETE.

5. Format versions

ALTER TABLE prod.db.trips SET TBLPROPERTIES ('format-version' = '3');

Format version only moves forward. Check that every engine reading the table supports the version before raising it.

6. Metadata tables

MetadataTableType lists sixteen, and querying them is the fastest way to understand a table you did not create.

-- Two queries that answer most "why is this table slow" questions
SELECT partition, record_count, file_count
FROM prod.db.trips.partitions ORDER BY file_count DESC LIMIT 20;

SELECT count(*) AS delete_files FROM prod.db.trips.delete_files;

7. Time travel, branches and tags

flowchart LR
  S1["snapshot 1"] --> S2["snapshot 2"]
  S2 --> S3["snapshot 3<br/>main"]
  S2 --> B1["snapshot 4<br/>etl-wip branch"]
  B1 --> B2["snapshot 5<br/>etl-wip"]
  S2 --> T1["tag<br/>audit-2026-q3"]
  B2 -.->|"fast_forward"| S3

Branches are the audit-and-backfill mechanism: stage a risky rewrite on a branch, validate it with ordinary queries, then fast-forward main. Nothing readers see changes until the fast-forward.

8. Schema evolution

Iceberg’s schema evolution has no experimental mode and no enabling flag, because column IDs are in the format itself rather than bolted on. That is the practical difference from formats where rename and drop need a compatibility mode switched on first.

9. Procedures

SparkProcedures registers exactly twenty at 1.11.0. Call them as CALL <catalog>.system.<name>(...).

-- Compact, sorting within each file group
CALL prod.system.rewrite_data_files(
  table => 'db.trips',
  strategy => 'sort',
  sort_order => 'city_id ASC, started_at DESC'
);

-- Expire snapshots older than a week, keeping at least 10
CALL prod.system.expire_snapshots(
  table => 'db.trips',
  older_than => TIMESTAMP '2026-09-08 00:00:00',
  retain_last => 10
);

-- What changed between two snapshots
CALL prod.system.create_changelog_view(
  table => 'db.trips',
  changelog_view => 'trips_changes',
  options => map('start-snapshot-id', '7241925443479918015'),
  identifier_columns => array('trip_id')
);

SELECT trip_id, _change_type, _change_ordinal FROM trips_changes ORDER BY _change_ordinal;

10. Maintenance

Note: A deleted row stays readable through time travel until expire_snapshots removes the snapshots containing it. For a right-to-erasure obligation, expiry is part of the compliance story rather than housekeeping.

11. Key table properties

12. Sorting, clustering and file sizing

-- Declare the order once, and every writer honours it
ALTER TABLE prod.db.trips WRITE ORDERED BY city_id, started_at;

-- Compact and z-order the last week only, committing progressively
CALL prod.system.rewrite_data_files(
  table => 'db.trips',
  strategy => 'sort',
  sort_order => 'zorder(city_id, rider_id)',
  where => 'started_at >= TIMESTAMP \'2026-09-08 00:00:00\'',
  options => map('partial-progress.enabled', 'true', 'max-concurrent-file-group-rewrites', '4')
);

Sorting is what makes Iceberg’s column bounds worth having. Unsorted data gives every file a wide min and max, so bounds prune nothing and every query is a full scan regardless of how much metadata you have.

13. Diagnosing a table

-- The three-query triage
SELECT count(*) AS files, sum(file_size_in_bytes)/1024/1024 AS mb FROM prod.db.trips.files;
SELECT count(*) AS snapshots, min(committed_at) AS oldest FROM prod.db.trips.snapshots;
SELECT count(*) AS delete_files FROM prod.db.trips.delete_files;

14. Concurrency

A commit conflict is the system behaving correctly. The failure worth worrying about is the opposite: two writers both succeeding and one silently losing rows, which is what a catalog without an atomic compare-and-swap allows.

Conclusion

The three things worth carrying away are that the catalog is part of the table, that the row-level operation modes default to copy-on-write, and that hidden partitioning plus partition evolution are the capabilities hardest to get anywhere else. Everything else on this page is a property you can change later.

When you move off the release this was written against, re-check the format version range first, then the defaults in the properties table, then whether SparkProcedures has grown: procedures are the part of the surface that changes most between releases.

References

• Iceberg table specification for snapshots, manifests, delete files and format versions
• Spark procedures for the full argument list of every CALL above
• Spark configuration for catalog wiring and read and write options
• TableProperties.java at apache-iceberg-1.11.0 for every property and its default
• Apache Hudi on Spark: the complete cheat sheet for the same reference treatment of Hudi