All posts

One table, three formats: Iceberg, Delta and Hudi interoperability

Iceberg, Delta and Hudi all describe Parquet files with metadata. That shared substrate is what lets a table be readable as more than one format without copying data — and it is also where the limits of that trick live.

9 min read Lakehouse

TL;DR

  • All three formats store ordinary Parquet and differ in the metadata describing it. Translation is therefore a metadata problem, not a data problem.
  • Apache XTable reads one format’s metadata and writes another’s over the same files, so a table becomes readable as Iceberg, Delta or Hudi with one copy of the data.
  • Delta UniForm does the same from the write side: a Delta table generates Iceberg metadata as it commits.
  • The data is shared, so storage does not double — but the metadata is generated, not live, and it is as current as the last sync.
  • One writer, many readers is the pattern that works. Writing through two formats to the same files is how you corrupt both.

Why this is possible at all

Iceberg, Delta Lake and Hudi disagree about metadata and agree about data. All three write Parquet, and all three record which Parquet files are in the table, what their schema is, and what changed in each commit.

flowchart TB
    D["the same Parquet files"] --> I["Iceberg metadata<br/>metadata.json + manifests"]
    D --> DL["Delta metadata<br/>_delta_log"]
    D --> H["Hudi metadata<br/>.hoodie timeline"]

If the data files are already acceptable to all three, then making a table readable by another engine means writing a second description of files that already exist. No bytes move.

That is the entire idea behind both tools below, and understanding it tells you where the limits are: anything expressible in one metadata model and not another is where translation gets lossy.

Apache XTable

XTable (incubating) reads the source format’s metadata and writes target metadata alongside it. The data files stay exactly where they are; the table gains a second — or third — way to be read.

The practical shape is a sync job: run it after writes, and readers of the target format see the table as of that sync. Incremental sync translates only what changed rather than re-reading the whole table, which is what makes it viable on a schedule rather than as a one-off conversion. The mechanics and the cost of getting that wrong are in Apache XTable incremental sync, and the command surface in the XTable cheat sheet.

What it is good for: an engine that only speaks one format, a migration where both old and new readers must work during the transition, a downstream consumer you do not control.

What it is not: a way to have two writers. Metadata is generated from a source of truth, and the source of truth is the format you write.

Delta UniForm

UniForm moves the same idea to write time. A Delta table configured for it generates Iceberg metadata as part of committing, so Iceberg readers see the table without a separate sync job.

The trade against XTable is directness versus generality. UniForm is built into the writer, so there is no extra job and less staleness, but it works from Delta outward. XTable is a separate process covering more source and target combinations, at the cost of running it.

Either way the shape is the same: one writer, one set of data files, several metadata descriptions.

What a translation job looks like

XTable runs as a job with a small YAML config, which makes the staleness window an explicit operational choice rather than an accident.

sourceFormat: ICEBERG
targetFormats:
  - DELTA
  - HUDI
datasets:
  - tableBasePath: s3://lake/warehouse/db/events
    tableName: events

java -jar xtable-utilities.jar --datasetConfig config.yaml

Run it after writes, on whatever cadence your consumers can tolerate. The output is additional metadata directories beside the existing data:

s3://lake/warehouse/db/events/
├── data/                  # unchanged, shared by all three
├── metadata/              # Iceberg
├── _delta_log/            # written by the sync
└── .hoodie/               # written by the sync

Nothing under data/ is touched, which is the property that makes this cheap and also the property that makes two writers catastrophic.

Verifying a translation

Row counts agreeing is necessary and not sufficient — the failure modes here preserve counts while changing values or losing pruning.

-- counts, the cheap check
SELECT count(*) FROM iceberg_cat.db.events;
SELECT count(*) FROM delta.`s3://lake/warehouse/db/events`;

-- aggregates, which catch representation differences
SELECT sum(amount), min(event_ts), max(event_ts) FROM iceberg_cat.db.events;
SELECT sum(amount), min(event_ts), max(event_ts) FROM delta.`s3://lake/warehouse/db/events`;

-- per-partition, which catches a partition that failed to map
SELECT region, count(*) FROM iceberg_cat.db.events GROUP BY region ORDER BY region;

Then check the thing most likely to be silently lost:

EXPLAIN SELECT * FROM delta.`s3://lake/warehouse/db/events`
WHERE event_ts >= date'2026-01-06';

If the translated table’s plan reads every file where the Iceberg table pruned to one partition, the rows are correct and the performance is not — which is the usual outcome when a transform-partitioned table is translated into a model that has no equivalent concept.

The staleness window, stated

A translated table is a point-in-time copy of metadata. Between syncs, readers of the target format see an older table than readers of the source.

That is fine for hourly analytics and wrong for anything that joins the two representations expecting consistency. Make it explicit:

-- how far behind is the translated view?
SELECT max(committed_at) AS source_latest FROM iceberg_cat.db.events.snapshots;
SELECT max(timestamp)    AS target_latest FROM delta.`...`.history;

Alert on the gap rather than on the sync job’s exit code — a job that succeeds while doing nothing is the failure that matters.

When not to translate

Three situations where the honest answer is to fix something else.

The engine gained support since you last checked. Iceberg support across Trino, Flink, Dremio, Snowflake and BigQuery has moved quickly, and translation layers outlive the constraint that motivated them.

The consumer reads once a day. A scheduled export to Parquet is simpler to operate and reason about than a metadata translation, and it has no ambiguity about who owns the table.

Two teams both want to write. No amount of translation makes that safe. The fix is a conversation about ownership, not a sync job.

What translation cannot carry

The formats are similar enough for this to work and different enough for it to be lossy at the edges.

Deletes are represented differently. Iceberg has position and equality delete files and, in v3, deletion vectors. Delta has deletion vectors. Hudi has log files in merge-on-read tables. A table relying on one format’s row-level delete mechanism does not always translate cleanly into another’s, and merge-on-read tables are where translation is most likely to disappoint.

Partitioning models differ. Iceberg’s hidden partitioning is a transform on a column; Hive-style partitioning in other models is a physical column. A transform-partitioned Iceberg table translated outward may lose the pruning that made it fast, even though the rows are all present.

History does not fully transfer. Time travel depends on each format’s own snapshot or version history. A translated table typically starts its history at translation, so the target format cannot travel back before that.

Table properties do not map one to one. Compaction settings, sort orders, retention policies are expressed differently and mostly do not survive.

The rule of thumb: the current state of the data translates well; the history and the optimizations translate poorly.

Choosing a strategy

Situation Approach
One engine, one team pick a format, skip all of this
Migrating, both readers live XTable during the transition
Delta writer, Iceberg readers UniForm
Many engines, indefinitely translate, and accept a staleness window
Two teams both wanting to write fix the ownership question instead

That last row is the important one. Interoperability solves reading. Two writers through two formats against the same files means two metadata layers each believing they own the table, and neither will notice the other’s commits. The result is silent corruption. One writer, many readers is the only safe shape.

Before reaching for any of this

The most common reason to translate is an engine that supports only one format. That is worth checking rather than assuming: Iceberg support has broadened considerably across Trino, Flink, Dremio, Snowflake and BigQuery, and the constraint that motivated a translation layer a year ago may no longer exist.

The differences between the formats themselves — why they made different choices rather than which is better — are in open table formats in practice.

Common misconceptions

“Interoperability means the data is copied.” The Parquet files are shared. Only metadata is written.

“A translated table is a live view.” It is as current as the last sync.

“I can write through both formats.” You can, and you will corrupt both.

“Everything translates.” Current state translates well. Deletes, history and optimizations translate poorly.

“UniForm and XTable are competitors.” They solve the same problem from different ends — one at write time from Delta, one as a job across many combinations.

A model worth keeping

The data files are the table. The metadata is a description of them, and a set of files can carry more than one description.

So interoperability is cheap for reading and impossible for writing, and the sensible arrangement is always the same: one format owns the writes, every other description is generated, and the staleness window is something you chose rather than discovered.

References

Trademarks

Apache Iceberg, Apache Hudi, Apache XTable, Apache Spark, Apache Parquet, Apache Flink, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Delta Lake is a trademark of the Linux Foundation.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee