All posts

What Hive tables could not do, and why Iceberg exists

A Hive table is a directory, and every limitation follows from that. Iceberg replaces the directory with a metadata tree that names every file. Here is what that changes, from atomic commits to schema changes that do not rewrite data.

8 min read Iceberg

TL;DR

  • A Hive table is a directory, and the files in it. Planning means listing directories, so cost grows with partition count and correctness depends on the filesystem.
  • An Iceberg table is a list of files recorded in metadata. Planning reads metadata, never lists a directory, and a commit is one atomic pointer swap.
  • That single change is what makes atomic writes, time travel, schema evolution and partition evolution possible — they are consequences of file tracking, not separate features.
  • Schema changes are metadata-only: renaming a column left the underlying data files untouched and kept its field id, 4.
  • Partition layout can change without rewriting history. After switching months(ts) to days(ts), existing files kept spec_id = 0 and stayed queryable.

The Hive table is a directory

For a decade the answer to “what is a table on a data lake” was: a directory, plus a row in a metastore saying where that directory is and what columns to expect.

s3://warehouse/trips/
├── dt=2026-01-05/  part-00000.parquet  part-00001.parquet
├── dt=2026-01-06/  part-00000.parquet
└── dt=2026-01-07/  part-00000.parquet

Everything good about that design comes from its simplicity: any engine that can list a path and read Parquet can read the table. Everything bad comes from the same place — the set of files in the table is whatever happens to be in the directory right now.

Four consequences follow, and every one of them is a production problem.

Planning means listing. To find what to read, the engine lists directories. A table with 50,000 partitions costs 50,000 listings before a single row is read. On object storage, where listing is an API call with latency and rate limits, this dominates query time on exactly the tables that are big enough to matter.

A write is not atomic. Writing a partition means putting files into a directory. A reader that lists mid-write sees a partial result. There is no moment at which the table changes from the old state to the new one; there is a period during which it is neither.

There is no history. Overwriting a partition deletes the previous files. If the job that produced them was wrong, the previous data is gone. Recovery means restoring from a backup you hopefully took.

Schema is a convention. The metastore records column names and positions. The Parquet files record their own. When these disagree — because someone added a column, or reordered one — you get wrong values rather than an error, because matching is by name or position and both are fragile.

What Iceberg changes

Iceberg replaces “the files in this directory” with “the files this metadata says are in the table”. That is the whole idea, and everything else is a consequence.

flowchart TB
    C[Catalog] -->|"points at the current"| M["metadata.json<br/>schema, partition specs, snapshots"]
    M -->|"current snapshot"| ML["manifest list<br/>snap-*.avro"]
    ML -->|"manifests, with partition ranges"| MF["manifest files<br/>*.avro"]
    MF -->|"data files, with column bounds"| D["*.parquet"]

The catalog holds one pointer: which metadata file is current. A commit writes new metadata and swaps that pointer atomically. The swap is the commit — before it, readers see the old table; after it, the new one; never anything in between.

Because the metadata names every file, planning never lists a directory. It reads the manifest list, discards manifests whose partition ranges cannot match, reads the surviving manifests, and discards files whose column bounds cannot match. Cost scales with metadata size, which you control, rather than with partition count, which you do not.

The consequences, measured

These are not separate features bolted on. Each one falls out of tracking files in metadata.

Atomic commits and history

Every commit writes a new metadata file rather than editing the old one. A table created and appended to three times accumulates them predictably:

after CREATE, before any data   metadata_files=2    data_files=0
after append #1                 metadata_files=5    data_files=1
after append #2                 metadata_files=8    data_files=2
after append #3                 metadata_files=11   data_files=3

Three metadata files per commit: a manifest, a manifest list, and a new metadata.json. Old snapshots stay valid because their metadata still exists and still names its files, which is exactly what time travel reads.

Schema changes that touch no data

Adding a column, renaming one and changing a type, on a table holding 9,000 rows:

before: ['id', 'event_ts', 'region', 'amount']
after : ['id', 'event_ts', 'region', 'amount_usd', 'currency']
rows still readable                = 9000
old rows have NULL currency        = 9000
data files after schema change     = 3   (unchanged: nothing was rewritten)

Three data files before, three after. The rename was a metadata edit. It is safe because Iceberg identifies columns by field id, not by name or position:

field id=1   name=id           type=long
field id=2   name=event_ts     type=timestamptz
field id=3   name=region       type=string
field id=4   name=amount_usd   type=double
field id=5   name=currency     type=string

amount became amount_usd and kept id 4. The Parquet files still record field 4; the schema now calls it something else. This is the fix for the Hive failure where a rename silently reads the wrong column.

Partition layout that can change

A table partitioned by month, switched to day, then appended to:

partition-specs: [{"spec-id": 0, ... "transform": "month", "source-id": 2},
                  {"spec-id": 1, ... "transform": "day",   "source-id": 2}]
default-spec-id: 1
spec_id per data file: [1, 0, 0]

Two files still carry spec_id = 0. Nothing was rewritten, both layouts coexist on disk, and all rows stay queryable through one table. In a Hive table the partition layout is the directory structure, so changing it means moving every file.

Partitioning is also hidden: the query names the real column and Iceberg applies the transform.

SELECT count(*) FROM local.db.events
WHERE event_ts >= timestamp'2026-01-06 00:00:00'
  AND event_ts <  timestamp'2026-01-07 00:00:00'

That returned 3,000 rows without naming a partition column, while the files sat in event_ts_day=2026-01-06. The Hive equivalent requires the user to know the physical column and filter on it, and silently scans everything when they forget.

What you give up

Iceberg is not free, and pretending otherwise sets people up for surprises.

Metadata is a thing you now operate. Every commit writes files. Streaming writes accumulate them quickly, and tables need compaction, snapshot expiry and occasionally manifest rewriting on a schedule.

Every engine needs Iceberg support. A Hive table can be read by anything that lists a directory. An Iceberg table needs a reader that understands the metadata tree. That is broadly true now — Spark, Trino, Flink, Dremio, Snowflake, BigQuery — but “broadly” is not “universally”, and a legacy tool doing directory listings will see nothing useful.

The catalog becomes infrastructure. Something has to own the pointer swap atomically. That is a service to run, secure and back up.

Small tables gain little. A table of a few files, written once daily, read by one engine, has none of the problems Iceberg solves. The metadata overhead is real and the benefit is not.

When the older design is still right

A table written once and read as a whole, small enough that listing is instant, consumed by one engine, with no update or delete requirement and no need for history — that is a directory of Parquet files, and adding a table format to it buys complexity rather than capability.

The moment any of those stops being true — concurrent writers, row-level deletes, a schema that changes, an auditor asking what the table said last Tuesday, or a partition count where listing hurts — the directory stops being enough, and the reason is always the same: nothing is tracking which files belong to the table.

A model worth keeping

A Hive table says “the data is over there”. An Iceberg table says “these exact files, with this schema, as of this snapshot”.

Every Iceberg feature people list separately — ACID, time travel, schema evolution, partition evolution, hidden partitioning — is that one sentence applied to a different problem. Which is also why they compose: they are not features, they are consequences.

References

Trademarks

Apache Iceberg, Apache Hive, Apache Spark, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee