Schema evolution in Iceberg: why a rename costs nothing

• The mechanism is one integer
• What a schema change costs
• The operations, and their limits
  • Dropped ids are retired
• Nested fields get ids too
• Type promotion, precisely
• Doing a narrowing change anyway
• Checking what a schema change will affect
• What this replaces
• Doing it safely
• Common misconceptions
• A model worth keeping
• References
• Trademarks

TL;DR

• Adding a column, renaming one and changing a type left the table at 3 data files before and 3 after. Nothing was rewritten.
• Columns are identified by field id, never by name or position. amount renamed to amount_usd kept id 4; the Parquet files never learned about it.
• A new column reads as NULL on old files — verified: all 9,000 pre-existing rows returned NULL for currency.
• Dropped ids are retired, never reused, so a later column cannot inherit an old column’s data.
• Type changes are allowed only where every existing value stays representable. Widening is fine; narrowing is refused.

The mechanism is one integer

Every column in an Iceberg schema has a numeric id assigned when the column is created. Data files record values against those ids. The schema maps ids to names.

field id=1   name=id           type=long
field id=2   name=event_ts     type=timestamptz
field id=3   name=region       type=string
field id=4   name=amount_usd   type=double
field id=5   name=currency     type=string

That table was produced after renaming amount to amount_usd and adding currency. amount_usd is id 4 — the id amount had. The rename changed the schema’s label for id 4 and nothing else. Files written when the column was called amount are still correct, because they never recorded the name.

This is why the operation is instant regardless of table size, and why it is safe.

What a schema change costs

Three changes against a table of 9,000 rows in 3 data files:

ALTER TABLE local.db.events ADD COLUMN currency STRING;
ALTER TABLE local.db.events RENAME COLUMN amount TO amount_usd;
ALTER TABLE local.db.events ALTER COLUMN id TYPE BIGINT;

before: ['id', 'event_ts', 'region', 'amount']
after : ['id', 'event_ts', 'region', 'amount_usd', 'currency']
rows still readable            = 9000
old rows have NULL currency    = 9000
data files after schema change = 3   (unchanged)

Three files before, three after. All 9,000 rows readable. Every pre-existing row returns NULL for the new column, because the files contain no value for id 5 and the reader fills the gap from the schema rather than failing.

The cost of a schema change is one new metadata.json. That is the entire bill.

The operations, and their limits

Widening is permitted because every existing value remains representable. An int stored in a file is still a valid long. Narrowing is refused because the engine cannot know that every value in every file fits, and discovering otherwise mid-query would mean silent corruption or a failure deep in a scan.

Genuinely narrowing a type is a rewrite: add a new column, populate it, verify, drop the old one.

Dropped ids are retired

Drop region and its id 3 is never handed to a future column. If ids were recycled, a new column called country reusing id 3 would read old region values as its own — wrong data, no error. Retirement is why Iceberg does not have that failure and name-matched formats do.

Nested fields get ids too

The rule extends all the way down. Struct fields, map keys and list elements each carry their own id, so evolution works inside a nested column exactly as it does at the top level.

ALTER TABLE local.db.events
  ADD COLUMN payload struct<source: string, version: int>;

ALTER TABLE local.db.events
  ADD COLUMN payload.trace_id string;          -- add inside the struct

ALTER TABLE local.db.events
  RENAME COLUMN payload.source TO payload.origin;

Reading the metadata afterwards shows the nested ids alongside the top-level ones:

import json, glob
md = json.load(open(sorted(glob.glob("/work/wh/db/events/metadata/v*.metadata.json"))[-1]))
def walk(fields, depth=0):
    for f in fields:
        t = f["type"]
        print("  " * depth, "id=%-3s %s" % (f["id"], f["name"]))
        if isinstance(t, dict) and t.get("type") == "struct":
            walk(t["fields"], depth + 1)
walk(md["schemas"][-1]["fields"])

Adding a field inside a struct is as cheap as adding a top-level column, and existing files read it as NULL. This is the part that tends to surprise people coming from formats where a nested change means a rewrite.

Type promotion, precisely

“Widening is allowed” is the shape of the rule; the spec is specific about which promotions exist.

The scale restriction on decimals catches people: you can make a decimal hold bigger numbers, not more fractional digits, because rescaling would change existing values rather than merely reinterpret them.

ALTER TABLE local.db.events ALTER COLUMN amount_usd TYPE decimal(18,2);  -- from (10,2): ok
ALTER TABLE local.db.events ALTER COLUMN amount_usd TYPE decimal(18,4);  -- refused

Doing a narrowing change anyway

When you genuinely need a change the rules forbid, the pattern is add-migrate-drop, and it is worth writing down because doing it in the wrong order loses data.

-- 1. add the new column; costs one metadata write
ALTER TABLE local.db.events ADD COLUMN amount_small float;

-- 2. populate it; this is the expensive step and it rewrites data
UPDATE local.db.events SET amount_small = CAST(amount_usd AS float);

-- 3. verify before you drop anything
SELECT count(*) FROM local.db.events WHERE amount_small IS NULL AND amount_usd IS NOT NULL;

-- 4. only then retire the old column
ALTER TABLE local.db.events DROP COLUMN amount_usd;

Step 3 is not optional. A CAST that silently produced NULL for out-of-range values is exactly the case the type rules were protecting you from, and this query is how you find out before the original is gone.

The dropped column’s data remains inside the existing Parquet files until those files are rewritten, so there is a recovery window:

CALL local.system.rewrite_data_files(table => 'db.events');   -- after this, it is gone

Checking what a schema change will affect

The table tolerates the change; your consumers may not. Before renaming or dropping anything:

-- which snapshots used which schema
SELECT snapshot_id, schema_id FROM local.db.events.snapshots ORDER BY committed_at;

-- what the current schema is, field by field
DESCRIBE TABLE local.db.events;

schema_id on the snapshot is the detail that makes time travel safe: an old snapshot records which schema it was written under, so reading it does not reinterpret old files through today’s schema.

What this replaces

On a Hive table, columns are matched by name or by position, depending on the reader, and both break.

By position: inserting a column in the middle shifts everything after it. Old files then line up against the wrong schema entries, and you read a region value as a currency. No error — just wrong answers.

By name: renaming a column makes old files stop matching. The column reads NULL for all historical data, which looks like a data quality problem rather than a schema problem, and gets investigated as one.

Both are silent. Iceberg’s ids make both impossible: files record ids, the schema maps ids to names, and renaming or reordering touches only the map.

Doing it safely

The mechanism is safe; the coordination around it still is not.

Adding a column is safe to do at any time. Readers that do not know about it ignore it; readers that do get NULL for history.

Renaming is safe for the table and not for your consumers. Every query, dashboard and downstream job naming the old column breaks at once. The table is fine; the ecosystem is not. Prefer adding the new name and migrating consumers before dropping the old one, exactly as with a database column.

Dropping is the one to stage. The data stays in the files — the column is hidden, not deleted — so it is recoverable until compaction rewrites those files without it. Treat the window between ALTER and the next rewrite as your grace period.

Check what reads the table first. The schema change is a metadata edit; the blast radius is every consumer.

Common misconceptions

“Adding a column rewrites the table.” It writes one metadata file. The measured change left the data file count unchanged.

“Renaming breaks old data.” Old files are keyed by id. The measured rename kept id 4 and all 9,000 rows.

“NULLs after adding a column mean something went wrong.” They mean the files predate the column. That is the defined behaviour.

“I can change any type.” Only where every existing value stays valid. Narrowing is refused.

“Dropping a column frees space.” It hides the column. The bytes go when the files are rewritten.

A model worth keeping

The schema is a map from ids to names and types. Data files are keyed by id. Schema evolution edits the map.

Which is why renaming is free, reordering is free, adding is free, and the only expensive operation is the one that would make existing values invalid — because that is the only one that requires touching the data.

References

• Iceberg schema evolution for the permitted changes
• Iceberg table specification for field ids and type promotion rules
• Apache Iceberg architecture for where the schema lives in the metadata tree
• Hidden partitioning and partition evolution for the same trick applied to layout
• What Hive tables could not do for the failure modes this replaces

Trademarks

Apache Iceberg, Apache Spark, Apache Hive, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.