The three tiers of an Iceberg table: catalog, metadata, data

• The three tiers
• Tier 1: the catalog
• Tier 2: metadata, in three levels
  • metadata.json
  • The manifest list
  • Manifests
• Tier 3: data
• What a table costs
• Reading each tier from SQL
• The growth rates differ, and that is the point
• A worked example of diagnosis
• Which tier is your problem in?
• Common misconceptions
• A model worth keeping
• References
• Trademarks

TL;DR

• Three tiers: the catalog holds one pointer, the metadata tier describes which files are in the table, the data tier holds rows.
• The metadata tier is itself three levels — metadata.json → manifest list → manifests — and each commit writes one of each. Measured: 2 → 5 → 8 → 11 files across three appends.
• A minimal table is 5 objects: one Parquet file, two metadata.json, one manifest list, one manifest.
• Manifests carry per-file column bounds keyed by field id, which is what turns metadata into a query accelerator rather than bookkeeping.
• Each tier has its own maintenance: data → compaction, metadata → manifest rewriting and expiry, catalog → nothing, which is why it must be reliable.

The three tiers

flowchart TB
    subgraph T1["catalog tier"]
      C["one pointer per table"]
    end
    subgraph T2["metadata tier"]
      M["metadata.json<br/>schema, specs, snapshots"] --> ML["manifest list<br/>snap-*.avro"]
      ML --> MF["manifests<br/>*.avro, per-file stats"]
    end
    subgraph T3["data tier"]
      D["*.parquet"]
      DEL["delete files"]
    end
    C --> M
    MF --> D
    MF --> DEL

Each tier answers one question. The catalog: which metadata file is current? The metadata tier: which data files are in the table, and what is in them? The data tier: the rows.

Almost every operational question resolves to “which tier is this?” — and the answer determines the fix.

Tier 1: the catalog

One pointer, swapped atomically. That is the whole tier, and its smallness is the point: the less it does, the easier it is to make correct.

It is also the only tier that cannot be rebuilt from the others. Lose the data and you have lost rows; lose the metadata and you have lost history; lose the catalog and you have lost which metadata file was current — which, with a directory full of metadata.json files, is recoverable but manual.

Its implementation decides whether concurrent commits are safe, and it is visible in the filenames the metadata tier produces. That is its own subject.

Tier 2: metadata, in three levels

metadata.json

The table’s state: current schema and every past schema, every partition spec, the snapshot list, table properties, and the pointer to each snapshot’s manifest list. One is written per commit, never edited.

format-version = 2
write.parquet.compression-codec = zstd

Schemas are kept plurally because old snapshots were written under old schemas, and time travel has to read them correctly.

The manifest list

One per snapshot, an Avro file naming the manifests that make up that snapshot — with a summary of each, including its partition range:

manifest af2b689e-...-m0.avro files=1
    partition_summaries=[lower_bound='2026-01-07', upper_bound='2026-01-07']
manifest aef10e73-...-m0.avro files=1
    partition_summaries=[lower_bound='2026-01-06', upper_bound='2026-01-06']

Those summaries let a planner discard an entire manifest — and every file in it — without opening it.

This level is also what makes snapshots cheap: a new manifest list points at unchanged manifests from the previous snapshot rather than copying them, so successive snapshots share almost all their metadata.

Manifests

The leaf level, listing data files with statistics:

file 00000-5-...parquet  rows=3000  partition=Row(event_ts_day=2026-01-06)
    lower_bounds={1: ..., 2: ..., 3: 'r0', 4: ...}
    upper_bounds={1: ..., 2: ..., 3: 'r2', 4: ...}

Record count, partition value, and per-column min/max keyed by field id. This is where metadata stops being bookkeeping and becomes a query accelerator: a filter that cannot match a file’s bounds skips the file unopened.

Each manifest also records the spec_id its files were written under, which is what lets partition layouts change without rewriting history.

Tier 3: data

Parquet (or ORC or Avro) holding rows, plus delete files on v2 and later marking rows removed without rewriting.

Nothing here is Iceberg-specific — the Parquet is ordinary Parquet. The table format is entirely the tiers above it, which is why migrating a Hive table is a metadata operation and why any engine can read the data once told which files to read.

What a table costs

A table with 5,000 rows written once, listed from object storage:

10KiB  db/trips/data/pickup_day=2026-02-10/00000-2-....parquet
 922B  db/trips/metadata/00000-fbb1be3d-....metadata.json
2.0KiB db/trips/metadata/00001-9804ab23-....metadata.json
7.3KiB db/trips/metadata/c4295385-....-m0.avro
4.3KiB db/trips/metadata/snap-7554101406213169966-1-c4295385-....avro

Five objects: one data, four metadata. Two metadata.json because there were two commits — the create and the append.

And the growth rate, across three appends to a partitioned table:

after CREATE, before any data   metadata_files=2    data_files=0
after append #1                 metadata_files=5    data_files=1
after append #2                 metadata_files=8    data_files=2
after append #3                 metadata_files=11   data_files=3

Three metadata files per commit, regardless of how many rows it carried. For a streaming table this is the number that matters: metadata grows with commit frequency, not data volume.

Reading each tier from SQL

Every tier is queryable, which makes “which tier is my problem in” a question you answer rather than guess.

-- TIER 1: what the catalog points at, and every pointer it has held
SELECT file, latest_snapshot_id, timestamp
FROM local.db.events.metadata_log_entries ORDER BY timestamp DESC LIMIT 5;

-- TIER 2a: snapshots, and what each commit did
SELECT snapshot_id, parent_id, operation,
       summary['added-data-files'] AS added,
       summary['total-records']    AS total
FROM local.db.events.snapshots ORDER BY committed_at;

-- TIER 2b: how much metadata planning must read
SELECT count(*) AS manifests, sum(added_data_files_count) AS files
FROM local.db.events.manifests;

-- TIER 3: the data itself, split by content type
SELECT content, count(*) AS files, sum(record_count) AS records,
       sum(file_size_in_bytes) AS bytes
FROM local.db.events.files GROUP BY content ORDER BY content;

That last query is the single most useful one on a production table. content is 0 for data, 1 for position deletes and 2 for equality deletes, so one row tells you whether you have a small-file problem, a delete-accumulation problem, or neither.

The growth rates differ, and that is the point

Each tier grows in response to something different, which is why one maintenance job cannot cover all three.

Read the middle column. A table that receives one large daily load has few commits and grows slowly in every metadata dimension. A table receiving a commit per minute grows metadata 1,440 times a day regardless of how many rows those commits carried — which is why a low-volume streaming table can develop a metadata problem that a high-volume batch table never does.

A worked example of diagnosis

A table where “queries got slower over the last month, and the data has not grown”. Work down the tiers.

-- 1. is it data, or metadata?
SELECT count(*) FROM local.db.events.manifests;          -- say 2,400
SELECT count(*) FROM local.db.events.files;              -- say 3,100
SELECT count(*) FROM local.db.events.snapshots;          -- say 43,000

Forty-three thousand snapshots against three thousand data files is the signature: this table’s problem is history, not data. Planning is reading 2,400 manifests, and nothing has ever been expired.

-- 2. confirm nothing is reclaimable yet
SELECT min(committed_at), max(committed_at) FROM local.db.events.snapshots;

If the oldest snapshot is months old, expiry has never run.

-- 3. fix in order: compact metadata, then reclaim
CALL local.system.rewrite_manifests(table => 'db.events');
CALL local.system.expire_snapshots(
  table => 'db.events', older_than => TIMESTAMP '2026-09-10 00:00:00', retain_last => 20);

The opposite signature — few snapshots, tens of thousands of data files — is a data problem and wants rewrite_data_files instead. The two look identical from a dashboard showing query duration, and the metadata tables are what tell them apart.

Which tier is your problem in?

The distinction between “slow before tasks start” and “slow tasks” is the diagnostic that matters most. The first is metadata — planning is reading too many manifests. The second is data — too many files or too many deletes. They look identical in a dashboard showing query duration and have different fixes.

Note that metadata.json accumulation is governed separately from snapshot expiry. Expiring snapshots does not remove old metadata files; write.metadata.delete-after-commit.enabled does.

Common misconceptions

“Metadata is small.” Four of the five objects in a fresh table are metadata, and it grows per commit rather than per row.

“One snapshot means one copy of the metadata.” Snapshots share manifests. A thousand snapshots do not cost a thousand times one.

“The catalog stores the schema.” The schema is in metadata.json. The catalog stores which one is current.

“Expiring snapshots cleans the metadata directory.” It removes snapshots and their exclusive files; metadata.json files have their own retention.

“Data files are special.” They are ordinary Parquet. All the structure is above them.

A model worth keeping

A pointer, a tree, and some Parquet.

The pointer must be atomic, the tree must stay small enough to read quickly, and the Parquet must stay large enough to read efficiently. Each has a maintenance job, and diagnosing a slow table starts with asking which of the three has grown out of shape.

References

• Iceberg table specification for metadata, manifest lists and manifests
• Apache Iceberg architecture for the same tree read from the source
• Life of a read query in Iceberg for how these tiers are traversed
• Life of a write query in Iceberg for how they are created
• Iceberg catalogs for the tier that holds the pointer

Trademarks

Apache Iceberg, Apache Spark, Apache Parquet, Apache Avro, Apache ORC, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.