Hidden partitioning and partition evolution: changing layout without rewriting history

• The Hive problem it fixes
• How Iceberg derives partitions
  • The transforms
• Partition evolution, measured
• Choosing the transform, with arithmetic
• Evolving in practice
• Where partitioning stops and sorting starts
• What evolution does not do
• Choosing a layout
• Common misconceptions
• A model worth keeping
• References
• Trademarks

TL;DR

• Partition values are derived from a column by a transform, so a query filters on event_ts and Iceberg prunes event_ts_day for it. Measured: a plain timestamp predicate matched 3,000 rows in one day’s partition.
• Changing the layout does not rewrite data. After REPLACE PARTITION FIELD months(ts) WITH days(ts), files carried spec_id values of [1, 0, 0] — the old ones untouched.
• Both specs stay in metadata (spec-id 0 and 1) and both reference the same source-id, so one predicate prunes across both layouts.
• Directory names like ts_month=2026-01 and ts_day=2026-03-09 coexist in the same table, and all 900 rows stayed queryable.
• Hidden partitioning removes the Hive failure where forgetting the partition filter scans everything silently.

The Hive problem it fixes

On a Hive table, a partition is a physical column in the path:

/trips/dt=2026-01-06/part-0000.parquet

To get pruning, a query must filter on dt. Filtering on the actual event timestamp does not prune, because dt is a separate column the user is responsible for keeping consistent.

Two failures follow, both routine. Someone writes WHERE event_ts >= '2026-01-06' and scans the whole table, because the query is correct and simply did not mention dt. And someone writes a row whose dt disagrees with its event_ts, producing data that is invisible to correct queries.

How Iceberg derives partitions

You declare a transform on a real column:

CREATE TABLE local.db.events (
  id BIGINT, event_ts TIMESTAMP, region STRING, amount DOUBLE)
USING iceberg PARTITIONED BY (days(event_ts));

There is no dt column. Iceberg computes the day from event_ts at write time and records it in the manifests. The spec, read from the metadata file:

partition-specs = [{"spec-id": 0, "fields": [
  {"name": "event_ts_day", "transform": "day", "source-id": 2, "field-id": 1000}]}]

source-id: 2 is the field id of event_ts. The partition is defined as a function of that column, which is the indirection everything else depends on.

At read time the query names the real column:

SELECT count(*) FROM local.db.events
WHERE event_ts >= timestamp'2026-01-06 00:00:00'
  AND event_ts <  timestamp'2026-01-07 00:00:00'

partition directories on disk: ['event_ts_day=2026-01-05',
                                'event_ts_day=2026-01-06',
                                'event_ts_day=2026-01-07']
rows matched = 3000

Iceberg applies day() to the predicate and matches it against partition values. Nobody has to know the layout, and nobody can forget to filter on it.

The transforms

bucket is the one people under-use. Partitioning by identity(customer_id) on a million customers produces a million partitions and a small-file disaster; bucket(64, customer_id) gives 64 balanced partitions that still prune equality filters.

Partition evolution, measured

Monthly partitioning stops working when data grows. On Hive that means rewriting the table. On Iceberg:

ALTER TABLE local.db.pe REPLACE PARTITION FIELD months(ts) WITH days(ts);

Then one more append. The result:

dirs after months(): ['ts_month=2026-01', 'ts_month=2026-02']
dirs after days()  : ['ts_day=2026-03-09', 'ts_month=2026-01', 'ts_month=2026-02']

partition-specs: [{"spec-id": 0, ... "transform": "month", "source-id": 2, "field-id": 1000},
                  {"spec-id": 1, ... "transform": "day",   "source-id": 2, "field-id": 1001}]
default-spec-id: 1
spec_id per data file: [1, 0, 0]
rows total: 900

Two of the three data files still carry spec_id = 0. They were not rewritten, moved or touched. The new file uses spec 1. Both directory layouts sit in the same table, and all 900 rows are queryable through one name.

This works because a manifest records which spec each file was written under. A reader consults the right spec per file rather than assuming one layout for the table.

Note both specs reference source-id: 2. Because both derive from the same column, a single predicate on ts prunes files under either spec — the reader applies month() to old files and day() to new ones.

The new field also got a new field-id (1001, not 1000), which keeps old partition values unambiguous.

Choosing the transform, with arithmetic

The transform decides how many partitions exist, and partition count is the number that goes wrong.

-- how many partitions does the current layout actually produce?
SELECT count(*) AS partitions,
       avg(file_count) AS avg_files,
       avg(record_count) AS avg_rows
FROM local.db.events.partitions;

Aim for partitions holding hundreds of megabytes to a few gigabytes. Work backwards from volume: a table taking 50 GB a month partitioned by days gives ~1.6 GB per partition, which is comfortable. The same table partitioned by hours gives ~70 MB per partition and 720 partitions a month, which is the beginning of a small-file problem.

bucket is the transform for high-cardinality keys, and the bucket count is a capacity decision:

CREATE TABLE local.db.orders (order_id BIGINT, customer_id BIGINT, amount DOUBLE)
USING iceberg
PARTITIONED BY (bucket(64, customer_id));

Sixty-four balanced partitions, regardless of how many customers exist. An equality filter on customer_id prunes to one bucket; a range filter prunes nothing, because hashing destroys ordering. Use bucket for keys you look up and truncate or a time transform for keys you scan ranges of.

Multiple fields multiply:

PARTITIONED BY (days(event_ts), bucket(16, customer_id))    -- 30 x 16 = 480 per month

That is usually one field too many. Each additional field divides the data further, and the small-file cost arrives faster than the pruning benefit.

Evolving in practice

-- add a field: new data gains the finer layout, old data keeps its own
ALTER TABLE local.db.events ADD PARTITION FIELD hours(event_ts);

-- replace one: the usual month-to-day move
ALTER TABLE local.db.events REPLACE PARTITION FIELD months(event_ts) WITH days(event_ts);

-- drop one entirely
ALTER TABLE local.db.events DROP PARTITION FIELD bucket(16, customer_id);

All three are metadata-only, and all three leave existing files alone. To check what you have afterwards:

SELECT spec_id, count(*) AS files, sum(record_count) AS records
FROM local.db.events.files GROUP BY spec_id ORDER BY spec_id;

A table with files spread across three or four specs is not broken, but it is harder to reason about, and planning must consult each spec. When old data is queried often enough to matter, rewrite it into the current layout:

CALL local.system.rewrite_data_files(
  table => 'db.events',
  where => 'event_ts < date'2026-03-01'');

That is a data-moving job, unlike the ALTER, and it is the step that actually gives historical data the new pruning.

Where partitioning stops and sorting starts

Partitioning eliminates whole directories of files. Sorting narrows the column statistics inside them so individual files can be skipped. They solve adjacent problems and people reach for the wrong one regularly.

Partitioning on a low-cardinality column like region is usually a mistake: four regions means four partitions, which does nothing for parallelism and produces skew when one region dominates. A sort order handles it without the file-count cost:

ALTER TABLE local.db.events WRITE ORDERED BY region, event_ts;

What evolution does not do

It does not reorganise history. Old data keeps the old granularity. If queries need day-level pruning over historical data, that is a rewrite — rewrite_data_files with the new spec — and it is a data-movement job, distinct from the metadata-only ALTER.

It does not fix bad partitioning retroactively. Over-partitioned history stays over-partitioned until rewritten.

It does not reduce the need to choose well. Evolution is an escape hatch, not a reason to skip the decision.

Choosing a layout

Partition on what you filter on, at the granularity you filter at. Hourly partitions for queries that always span weeks produce thousands of tiny files and prune nothing useful.

Target partitions that hold real data. Hundreds of megabytes to a few gigabytes is a reasonable band. A partition per minute holding ten rows is overhead with a directory name.

Use bucket for high-cardinality keys rather than identity.

Prefer fewer partition fields. Each one multiplies the partition count, and two fields with a hundred values each is ten thousand partitions.

Remember that pruning has a second stage: column bounds inside manifests skip files within a partition, but only if data is sorted. Partitioning and sort order work together, and the read path post covers where each applies.

Common misconceptions

“Hidden partitioning means no partitions.” The partitions are real and visible on disk. What is hidden is the need to name them in queries.

“Partition evolution rewrites the table.” Measured: two of three files kept spec_id = 0 and nothing moved.

“I must query the partition column.” There is no partition column to query. Filter the real column.

“Old data becomes unqueryable after evolution.” All 900 rows stayed readable across both specs.

“More partitions prune better.” More partitions mean more metadata and smaller files. Pruning comes from matching the transform to the filter.

A model worth keeping

A partition is a function of a column, recorded per file. Queries filter the column; Iceberg applies the function. Change the function and old files keep their old one, because each file remembers which spec produced it.

That is hidden partitioning and partition evolution — one indirection, used twice.

References

• Iceberg partitioning for the transforms and hidden partitioning
• Iceberg evolution for how specs coexist
• Life of a read query in Iceberg for where partition pruning sits among the four prunes
• Schema evolution in Iceberg for the field ids partition specs point at
• Apache Iceberg architecture for manifests and spec ids

Trademarks

Apache Iceberg, Apache Spark, Apache Hive, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.