All posts

Migrating Hive tables to Iceberg: snapshot first, migrate second

Two procedures convert a Hive table to Iceberg without moving data. One makes a parallel copy you can test against while the original keeps serving; the other converts in place and is hard to undo. Here is what each does, and the catalog requirement that decides whether either works.

8 min read Iceberg

TL;DR

  • Neither procedure copies data. Both write Iceberg metadata that points at the Parquet files already on disk, so a terabyte table converts in about the time it takes to read its file listing.
  • system.snapshot creates a parallel Iceberg table over the same data files. The measured run produced a 6,000-row Iceberg table while the source stayed Parquet and intact — the safe way to test before cutting over.
  • system.migrate converts the table in place, replacing it. It is the cutover, not the rehearsal.
  • migrate requires a catalog whose table locations it can take over. Against a path-based session catalog it refused: Cannot set a custom location for a path-based table.
  • Writes to the original table after a snapshot do not appear in the copy. The copy is a point-in-time view of the files, not a replica.

Two procedures, two intentions

Converting a Hive table to Iceberg sounds like it should mean rewriting the data. It does not. The Parquet files a Hive table points at are already the format Iceberg uses; what is missing is the metadata tree — the manifests, the manifest list, the snapshot, the schema with field IDs.

Both procedures build that tree over the existing files. They differ in what happens to the original.

flowchart TB
    H[(Hive table<br/>parquet files)] -->|"system.snapshot"| C["new Iceberg table<br/>same files, original untouched"]
    H -->|"system.migrate"| M["same name, now Iceberg<br/>original replaced"]

system.snapshot is the rehearsal. It creates a second table, under a new name, whose metadata references the same data files. The original keeps working, keeps serving queries, and keeps accepting writes. You point a test workload at the copy and compare.

system.migrate is the cutover. The table keeps its name and becomes an Iceberg table. Existing readers now get Iceberg semantics whether or not they were ready for them.

The order in the title is the recommendation: snapshot, validate, then migrate.

Snapshot migration, measured

A partitioned Parquet table with 6,000 rows, created the ordinary way:

CALL spark_catalog.system.snapshot('legacy_a', 'ice_copy');

source is parquet      : Provider = parquet
source rows            = 6000
snapshot-table rows    = 6000
ice_copy provider      : Provider = iceberg
ice_copy snapshots     = 1
source still intact    = 6000

Both tables report 6,000 rows. One is Parquet, one is Iceberg, and there is only one copy of the data on disk. The Iceberg table has exactly one snapshot, because its history begins at conversion.

This is the state you want before a cutover. Run your real queries against ice_copy, compare results and plans against legacy_a, and find out what breaks while the original is still authoritative.

What the copy does not do is track the original. Rows written to legacy_a after the snapshot will not appear in ice_copy — the copy’s manifests list the files that existed at conversion time. For a table under active write, that makes the copy a validation target, not a mirror, and the divergence starts immediately.

That also means a snapshot copy is not a rollback plan for a migration that happened later. It is a rollback plan only for the moment it was taken.

In-place migration, and what I could not demonstrate

CALL spark_catalog.system.migrate('legacy_b');

This converts the table in place. Iceberg reads the existing partition structure and file listing, writes the metadata tree, and swaps the table definition. In most configurations it leaves the previous definition behind under a name suffixed _BACKUP_, so the original can be restored.

I was not able to run this successfully, and would rather say so than describe behaviour I did not observe. Against a path-based session catalog it failed:

IllegalArgumentException: Cannot set a custom location for a path-based table.
Expected /work/wh2/default/legacy_b but got file:/work/wh2/legacy_b

The cause is a location convention. A Hadoop, path-based catalog derives a table’s location from <warehouse>/<namespace>/<table>, while Spark’s own warehouse places default-database tables directly under the warehouse root. The two disagree about where legacy_b lives, and migrate refuses rather than guess.

Configuring the session catalog as type=hive instead — the realistic setup, since the tables being migrated are Hive tables — failed in my container for a different reason: no metastore was available to it.

So the honest summary is: snapshot verified end to end; migrate verified only to the point of its catalog requirement. What the error does establish, and it is the practically useful part, is that migrate needs a catalog that owns its tables’ locations. A path-based catalog pointed at a Spark warehouse is not that, and this is the first thing to check when the procedure refuses.

What migration does not change

Worth being explicit, because expectations here cause post-cutover surprises.

File layout stays as it was. A Hive table partitioned into region=r0/region=r1 directories keeps those directories. Iceberg records them as partition values in its metadata; it does not reorganise storage. You get Iceberg’s metadata benefits immediately and its layout benefits only after you rewrite data.

Hidden partitioning does not appear retroactively. The converted table has an explicit partition spec matching the Hive columns. Transform-based partitioning — days(ts) rather than a physical dt column — is a later change, applied to new data, with old files keeping the original spec.

File sizes stay as they were. If the Hive table had a small-file problem, the Iceberg table has the same small-file problem, now with accurate statistics that let you see it. rewrite_data_files is the fix and it is a separate, data-moving job.

Statistics come from the files. Iceberg records column bounds by reading Parquet footers during conversion. That is what makes the conversion fast, and what makes conversion of a table with millions of files not fast.

A migration sequence that works

  1. Snapshot into a copy. system.snapshot('sales', 'sales_iceberg'). The original keeps serving.
  2. Validate. Row counts, aggregate checksums per partition, and the queries that matter, run against both. Compare results, not just counts — a schema mismatch shows up in values long before it shows up in a count.
  3. Test the write path. Run your actual writers against the copy. This is where engines that cannot handle Iceberg reveal themselves, and it is much cheaper to learn now.
  4. Freeze writes to the original, briefly.
  5. Migrate in place. system.migrate('sales'), which preserves the table name so downstream references keep resolving.
  6. Verify, then keep the backup until you are confident. The _BACKUP_ table is the undo.
  7. Then improve the layout. Compaction, sort order, partition spec changes — all of it after the cutover, none of it during.

Do not combine steps 5 and 7. A cutover that also reorganises storage cannot be reasoned about when something looks wrong afterwards, and it turns a metadata operation into a data-movement operation.

Verification worth doing

Counts agreeing is necessary and not sufficient. Three checks catch most real problems:

Per-partition counts, which catch a partition that failed to map:

SELECT region, count(*) FROM sales_iceberg GROUP BY region;

Aggregate checksums on numeric columns, which catch type-mapping errors that preserve row count while changing values:

SELECT sum(amount), min(amount), max(amount) FROM sales_iceberg;

File accounting, which confirms nothing was copied:

SELECT count(*), sum(file_size_in_bytes) FROM sales_iceberg.files;

Compare that byte total against the original directory. They should match, because the files are the same files.

Common misconceptions

“Migration rewrites the data.” Neither procedure moves data. Both write metadata over the files already present.

“A snapshot copy stays in sync.” It reflects the files at conversion time. Later writes to the source do not appear in it.

“Migration fixes small files.” It inherits them, and gives you the statistics to see them clearly.

“Migration is instant regardless of table size.” It scales with the number of files, because every Parquet footer is read for statistics. Millions of files is a long job.

“I can migrate any table.” migrate requires a catalog that owns table locations. The measured failure above is the common first obstacle.

A model worth keeping

Migration is a metadata operation over data that already exists. snapshot gives you a copy to be wrong against; migrate gives you the cutover. Doing them in that order costs one extra step and converts an irreversible change into a rehearsed one.

Everything that makes Iceberg fast — compaction, sort orders, transform partitioning — comes after the cutover, not during it.

References

Trademarks

Apache Iceberg, Apache Spark, Apache Hive, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee