What Hive tables could not do, and why Iceberg exists

• The Hive table is a directory
• What Iceberg changes
• The consequences, measured
  • Atomic commits and history
  • Schema changes that touch no data
  • Partition layout that can change
• Seeing the difference in one session
• The failure modes it removes
• What it costs to adopt
• What you give up
• When the older design is still right
• A model worth keeping
• References
• Trademarks

TL;DR

• A Hive table is a directory, and the files in it. Planning means listing directories, so cost grows with partition count and correctness depends on the filesystem.
• An Iceberg table is a list of files recorded in metadata. Planning reads metadata, never lists a directory, and a commit is one atomic pointer swap.
• That single change is what makes atomic writes, time travel, schema evolution and partition evolution possible — they are consequences of file tracking, not separate features.
• Schema changes are metadata-only: renaming a column left the underlying data files untouched and kept its field id, 4.
• Partition layout can change without rewriting history. After switching months(ts) to days(ts), existing files kept spec_id = 0 and stayed queryable.

The Hive table is a directory

For a decade the answer to “what is a table on a data lake” was: a directory, plus a row in a metastore saying where that directory is and what columns to expect.

s3://warehouse/trips/
├── dt=2026-01-05/  part-00000.parquet  part-00001.parquet
├── dt=2026-01-06/  part-00000.parquet
└── dt=2026-01-07/  part-00000.parquet

Everything good about that design comes from its simplicity: any engine that can list a path and read Parquet can read the table. Everything bad comes from the same place — the set of files in the table is whatever happens to be in the directory right now.

Four consequences follow, and every one of them is a production problem.

Planning means listing. To find what to read, the engine lists directories. A table with 50,000 partitions costs 50,000 listings before a single row is read. On object storage, where listing is an API call with latency and rate limits, this dominates query time on exactly the tables that are big enough to matter.

A write is not atomic. Writing a partition means putting files into a directory. A reader that lists mid-write sees a partial result. There is no moment at which the table changes from the old state to the new one; there is a period during which it is neither.

There is no history. Overwriting a partition deletes the previous files. If the job that produced them was wrong, the previous data is gone. Recovery means restoring from a backup you hopefully took.

Schema is a convention. The metastore records column names and positions. The Parquet files record their own. When these disagree — because someone added a column, or reordered one — you get wrong values rather than an error, because matching is by name or position and both are fragile.

What Iceberg changes

Iceberg replaces “the files in this directory” with “the files this metadata says are in the table”. That is the whole idea, and everything else is a consequence.

flowchart TB
    C[Catalog] -->|"points at the current"| M["metadata.json<br/>schema, partition specs, snapshots"]
    M -->|"current snapshot"| ML["manifest list<br/>snap-*.avro"]
    ML -->|"manifests, with partition ranges"| MF["manifest files<br/>*.avro"]
    MF -->|"data files, with column bounds"| D["*.parquet"]

The catalog holds one pointer: which metadata file is current. A commit writes new metadata and swaps that pointer atomically. The swap is the commit — before it, readers see the old table; after it, the new one; never anything in between.

Because the metadata names every file, planning never lists a directory. It reads the manifest list, discards manifests whose partition ranges cannot match, reads the surviving manifests, and discards files whose column bounds cannot match. Cost scales with metadata size, which you control, rather than with partition count, which you do not.

The consequences, measured

These are not separate features bolted on. Each one falls out of tracking files in metadata.

Atomic commits and history

Every commit writes a new metadata file rather than editing the old one. A table created and appended to three times accumulates them predictably:

after CREATE, before any data   metadata_files=2    data_files=0
after append #1                 metadata_files=5    data_files=1
after append #2                 metadata_files=8    data_files=2
after append #3                 metadata_files=11   data_files=3

Three metadata files per commit: a manifest, a manifest list, and a new metadata.json. Old snapshots stay valid because their metadata still exists and still names its files, which is exactly what time travel reads.

Schema changes that touch no data

Adding a column, renaming one and changing a type, on a table holding 9,000 rows:

before: ['id', 'event_ts', 'region', 'amount']
after : ['id', 'event_ts', 'region', 'amount_usd', 'currency']
rows still readable                = 9000
old rows have NULL currency        = 9000
data files after schema change     = 3   (unchanged: nothing was rewritten)

Three data files before, three after. The rename was a metadata edit. It is safe because Iceberg identifies columns by field id, not by name or position:

field id=1   name=id           type=long
field id=2   name=event_ts     type=timestamptz
field id=3   name=region       type=string
field id=4   name=amount_usd   type=double
field id=5   name=currency     type=string

amount became amount_usd and kept id 4. The Parquet files still record field 4; the schema now calls it something else. This is the fix for the Hive failure where a rename silently reads the wrong column.

Partition layout that can change

A table partitioned by month, switched to day, then appended to:

partition-specs: [{"spec-id": 0, ... "transform": "month", "source-id": 2},
                  {"spec-id": 1, ... "transform": "day",   "source-id": 2}]
default-spec-id: 1
spec_id per data file: [1, 0, 0]

Two files still carry spec_id = 0. Nothing was rewritten, both layouts coexist on disk, and all rows stay queryable through one table. In a Hive table the partition layout is the directory structure, so changing it means moving every file.

Partitioning is also hidden: the query names the real column and Iceberg applies the transform.

SELECT count(*) FROM local.db.events
WHERE event_ts >= timestamp'2026-01-06 00:00:00'
  AND event_ts <  timestamp'2026-01-07 00:00:00'

That returned 3,000 rows without naming a partition column, while the files sat in event_ts_day=2026-01-06. The Hive equivalent requires the user to know the physical column and filter on it, and silently scans everything when they forget.

Seeing the difference in one session

The argument is short enough to run. Both tables hold the same rows; only the format differs.

# Hive-style: a directory of Parquet, partitioned physically
(df.write.mode("overwrite").partitionBy("region")
   .format("parquet").saveAsTable("legacy_events"))

# Iceberg: the same data, tracked in metadata
spark.sql("""
    CREATE TABLE local.db.events (id BIGINT, event_ts TIMESTAMP, region STRING, amount DOUBLE)
    USING iceberg PARTITIONED BY (days(event_ts))""")
df.writeTo("local.db.events").append()

Now ask each one a question only the Iceberg table can answer:

-- what did this table look like an hour ago?
SELECT count(*) FROM local.db.events TIMESTAMP AS OF '2026-01-06 09:00:00';

-- which files does it consist of, and how big are they?
SELECT count(*), round(avg(file_size_in_bytes)/1048576, 1) FROM local.db.events.files;

-- what has been done to it, and by which commit?
SELECT snapshot_id, operation, summary['added-records'] FROM local.db.events.snapshots;

There is no equivalent for the Hive table, and that is the whole difference stated as a capability rather than an argument: the Iceberg table knows what it is made of.

The failure modes it removes

Four concrete situations, each routine on a directory-based table.

A reader sees a half-written partition. A job overwrites dt=2026-01-06 by deleting and rewriting files; a query running at that moment reads whatever exists. On Iceberg the query resolved a snapshot before the write began and is unaffected.

Two writers both “succeed”. Two jobs write to the same partition concurrently and the result is the union, the last one, or a mixture, depending on timing. On Iceberg one commit wins the compare-and-swap and the other retries against the new state.

A rename silently empties a column. A column renamed in the metastore no longer matches old files, and historical rows read NULL. On Iceberg the files record field ids, so a rename is a label change.

A backfill has to be undone. On a directory table, the previous files are gone. On Iceberg:

CALL local.system.rollback_to_snapshot('db.events', 9044140986145286505);

Each of these is silent on the older design. None produces an error; all produce wrong answers or unrecoverable states, which is why they are expensive to find.

What it costs to adopt

Being fair about the other side of the ledger.

Three jobs you did not have before. Compaction, delete-file compaction and snapshot expiry. None is optional on a table under real write traffic, and a table that skips them degrades in ways a Hive table does not.

A catalog to operate. Something must own the pointer and swap it atomically. That is a service to deploy, secure, monitor and back up — infrastructure a directory never needed.

Reader compatibility becomes a constraint. Every engine touching the table must understand the metadata tree, and the format version becomes a contract across teams.

Metadata grows with commits, not data. A low-volume table committing every minute develops a metadata problem a high-volume daily batch table never will, which inverts the intuition most people bring.

What you give up

Iceberg is not free, and pretending otherwise sets people up for surprises.

Metadata is a thing you now operate. Every commit writes files. Streaming writes accumulate them quickly, and tables need compaction, snapshot expiry and occasionally manifest rewriting on a schedule.

Every engine needs Iceberg support. A Hive table can be read by anything that lists a directory. An Iceberg table needs a reader that understands the metadata tree. That is broadly true now — Spark, Trino, Flink, Dremio, Snowflake, BigQuery — but “broadly” is not “universally”, and a legacy tool doing directory listings will see nothing useful.

The catalog becomes infrastructure. Something has to own the pointer swap atomically. That is a service to run, secure and back up.

Small tables gain little. A table of a few files, written once daily, read by one engine, has none of the problems Iceberg solves. The metadata overhead is real and the benefit is not.

When the older design is still right

A table written once and read as a whole, small enough that listing is instant, consumed by one engine, with no update or delete requirement and no need for history — that is a directory of Parquet files, and adding a table format to it buys complexity rather than capability.

The moment any of those stops being true — concurrent writers, row-level deletes, a schema that changes, an auditor asking what the table said last Tuesday, or a partition count where listing hurts — the directory stops being enough, and the reason is always the same: nothing is tracking which files belong to the table.

A model worth keeping

A Hive table says “the data is over there”. An Iceberg table says “these exact files, with this schema, as of this snapshot”.

Every Iceberg feature people list separately — ACID, time travel, schema evolution, partition evolution, hidden partitioning — is that one sentence applied to a different problem. Which is also why they compose: they are not features, they are consequences.

References

• Iceberg table specification for the metadata tree and its guarantees
• Iceberg documentation for the engine and catalog support matrix
• Apache Iceberg architecture for the metadata tree in depth
• Open table formats in practice for how Iceberg, Hudi and Delta differ
• A local Iceberg playground for an environment to try this in

Trademarks

Apache Iceberg, Apache Hive, Apache Spark, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.