All posts

Life of a read query in Iceberg: four prunes before a byte of data is read

Planning an Iceberg query is a sequence of eliminations against metadata. Each level discards work the next level would have done, and the last one only works if your data is sorted. Here is each step with the metadata it reads.

7 min read Iceberg

TL;DR

  • Planning is four steps: catalog → metadata.json → manifest list → manifests → data files, and each discards candidates before the next runs.
  • The manifest list carries a partition range per manifest. A one-day filter discarded two of three manifests without opening them.
  • Manifests carry per-file column bounds keyed by field id, not column name — the same identity that makes renames safe.
  • No step lists a directory. Planning cost scales with manifest count, which you control with rewrite_manifests, not with partition count.
  • The column-bounds prune only works if data is sorted. Unsorted files have wide, overlapping min/max, and nothing is skipped.

The four levels

A read query never asks the filesystem what exists. It asks metadata, and each level of metadata is smaller and more selective than the data below it.

flowchart TB
    Q["SELECT ... WHERE event_ts = '2026-01-06'"] --> C["1. catalog<br/>which metadata.json is current?"]
    C --> M["2. metadata.json<br/>schema, specs, snapshot list"]
    M --> ML["3. manifest list<br/>drop manifests by partition range"]
    ML --> MF["4. manifests<br/>drop files by partition value<br/>then by column bounds"]
    MF --> D["open the surviving Parquet files"]

1. The catalog resolves the pointer

One lookup: for this table name, which metadata file is current? That answer is the isolation boundary. Every subsequent step reads from that one file, so a commit landing mid-query cannot change what this query sees.

2. The metadata file selects a snapshot

The metadata file holds the schema, every partition spec the table has ever used, and the snapshot list. Normally the current snapshot is chosen; a time-travel query picks a different one by id or timestamp. Everything downstream is relative to that choice.

3. The manifest list drops whole manifests

This is where the first real elimination happens, and it is cheap because the manifest list is one small Avro file summarising each manifest, including the range of partition values inside it.

From a table with three daily partitions:

manifest af2b689e-...-m0.avro files=1
    partition_summaries=[lower_bound='2026-01-07', upper_bound='2026-01-07']
manifest aef10e73-...-m0.avro files=1
    partition_summaries=[lower_bound='2026-01-06', upper_bound='2026-01-06']
manifest 53fea6b1-...-m0.avro files=1
    partition_summaries=[lower_bound='2026-01-05', upper_bound='2026-01-05']

A filter for 2026-01-06 keeps the middle manifest and discards the other two without opening them. On a real table where one manifest covers hundreds of files, that is hundreds of files eliminated by reading a few hundred bytes.

4. Manifests drop individual files

Surviving manifests are opened, and they carry, per data file, the partition value and the min/max of each column:

file 00000-5-...parquet  rows=3000  partition=Row(event_ts_day=2026-01-06)
    lower_bounds={1: ..., 2: ..., 3: 'r0', 4: ...}
    upper_bounds={1: ..., 2: ..., 3: 'r2', 4: ...}

Two prunes happen here. The partition value eliminates files in the wrong partition. Then column bounds eliminate files whose min/max cannot contain a matching row — a filter region = 'r5' against a file whose region range is r0–r2 skips it without reading it.

The bounds keys are field ids. 3 is region because the schema says field 3 is named region today. Rename the column and the bounds keep working, because nothing recorded the name.

Hidden partitioning at read time

The query does not mention the partition column:

SELECT count(*) FROM local.db.events
WHERE event_ts >= timestamp'2026-01-06 00:00:00'
  AND event_ts <  timestamp'2026-01-07 00:00:00'

partition directories on disk: ['event_ts_day=2026-01-05',
                                'event_ts_day=2026-01-06',
                                'event_ts_day=2026-01-07']
rows matched = 3000

The table is partitioned by days(event_ts); the filter names event_ts. Iceberg applies the same transform to the predicate and matches it against partition values in the manifests. No user needs to know the physical layout, and nobody can forget to filter on it.

This also works across partition specs. A table whose layout changed from month to day keeps both, and because both specs derive from source-id: 2 — the same column — one predicate prunes files under either.

What makes the last prune fail

Column bounds are only useful when they are narrow, and that depends entirely on how the data was written.

Rows arriving in random order put a wide range of every column in every file. The min/max then overlap across all files, no file can be excluded, and the prune does nothing — while still looking healthy in the metadata.

Sorting is the fix, and it is why WRITE ORDERED BY and sorting compaction are not cosmetic:

ALTER TABLE local.db.events WRITE ORDERED BY region, event_ts;

After a sorting rewrite, each file holds a narrow slice of the sort column, the bounds stop overlapping, and filters on that column start eliminating files.

Choose the sort column to match your filters. Sorting by a column nobody filters on costs a rewrite and prunes nothing.

Why planning cost is metadata, not partitions

No step in the sequence lists a directory. That is the structural difference from Hive-style tables, where planning is a listing per partition and cost grows with partition count.

Iceberg’s planning cost grows with the number of manifests to read, which grows with the number of commits. A streaming table committing every minute accumulates manifests quickly, and the symptom is distinctive: time spent before any task starts, growing while the data stays the same size.

rewrite_manifests is the fix, and it is a different job from data compaction:

CALL local.system.rewrite_manifests(table => 'db.events');

A measured run elsewhere took a table from 9 manifests to 1.

Reading the evidence yourself

Three queries answer whether pruning can work on your table:

-- how many manifests must planning read?
SELECT count(*) FROM local.db.events.manifests;

-- do partition ranges separate cleanly, or do they overlap?
SELECT path, partition_summaries FROM local.db.events.manifests;

-- are column bounds narrow, or does every file span everything?
SELECT file_path, lower_bounds, upper_bounds FROM local.db.events.files LIMIT 10;

Overlapping bounds across most files means the third prune is doing nothing, and sorting is the lever. A large manifest count means planning is paying for history, and rewrite_manifests is the lever.

Common misconceptions

“Iceberg is fast because it is Parquet.” The data files are the same Parquet a Hive table uses. The difference is how many of them get opened.

“Partition pruning is the optimization.” It is one of four. Manifest-level pruning happens before it, and column-bounds pruning after.

“More partitions prune better.” More partitions mean more metadata and more small files. Pruning quality comes from filters matching the partition transform and from sorted data, not from partition count.

“Statistics are collected by a separate job.” Bounds are written into manifests by the writer, as part of the commit.

“Time travel is slower.” It selects a different snapshot at step 2 and then behaves identically.

A model worth keeping

Planning is a funnel: one catalog lookup, one metadata file, one manifest list, some manifests, and then only the data files that survived.

Each level exists to avoid the level below. When queries slow down, the question is which level stopped eliminating — manifests accumulating makes step 3 expensive, and unsorted data makes step 4 useless.

References

Trademarks

Apache Iceberg, Apache Spark, Apache Parquet, Apache Hive, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee