Table maintenance for Iceberg and Hudi: compaction, expiry, and the space you did not reclaim

• Why does a table need maintenance at all?
• What the two formats keep
• Iceberg: the small-file problem, measured
  • Compaction rewrites them
  • But the space is not reclaimed
  • Expiry is what deletes
  • The other two operations
• Hudi: what a Merge-on-Read table defers
  • The read-optimized query is stale, and here is the proof
  • Compaction merges the logs in
  • Cleaning, and what I could not demonstrate
• How the two compare
• What to schedule, and how often
• Symptoms and causes
• Common misconceptions
• A model worth keeping
• References
• Trademarks

TL;DR

• Compaction does not free space. After rewriting 16 Iceberg data files into 1, the table reported 1 data file and the directory still held 17 Parquet files. Only expire_snapshots deleted the other 16.
• Every maintenance operation is itself a commit. Compacting produced a new snapshot, and rewriting manifests produced another, so a table can gain history faster than it loses it.
• In Hudi Merge-on-Read, a read-optimized query returns stale values until compaction runs. The same table summed to 20,000 read-optimized and 40,000 on a snapshot read, from identical row counts.
• Hudi compaction writes new base files and leaves the old base and log files in place: base file count went 4 → 8 while log files stayed at 16. Cleaning is the separate step that removes them.
• Maintenance that is never scheduled is the most common lakehouse failure. Both formats accumulate silently and neither runs these operations for you by default.

Why does a table need maintenance at all?

A lakehouse table is append-structured. A write does not modify existing files; it adds new ones and records, in metadata, which files now constitute the table. That is what makes snapshot isolation and time travel possible, and it has a direct consequence: the files a table no longer uses do not go anywhere.

Two costs accumulate from this. Writes that arrive frequently produce many small files, and a query that must open a thousand files to read what would fit in ten spends its time on open and close rather than on data. Separately, every version of every rewritten file stays on disk, referenced by older snapshots, long after the current version of the table stopped needing it.

Maintenance is the set of operations that repair both. The distinction that matters, and the one this post is built around, is that they are different operations and doing the first does not do the second.

Everything below was run and measured: Iceberg 1.11.0 on Spark 4.1.3, and Hudi 1.2.0 on Spark 3.5.9, each writing to a local warehouse.

What the two formats keep

The maintenance vocabulary differs because the storage models differ.

Iceberg                          Hudi
-------                          ----
Catalog                          Timeline (.hoodie/)
   |                                |
Table metadata (vN.metadata.json)   Instants: deltacommit, commit,
   |                                           compaction, clean
Snapshot                             |
   |                             File Group
Manifest list (avro)                 |
   |                             File Slice
Manifest (avro)                      |
   |                             Base file (parquet) + Log files
Data files (parquet)

In Iceberg, a snapshot points at a manifest list, which points at manifests, which list data files. Maintenance operations rewrite one or more of those levels. In Hudi, records are grouped into file groups; each slice holds a base file and, for Merge-on-Read tables, log files carrying updates that have not yet been merged in.

Iceberg: the small-file problem, measured

Eight appends to an Iceberg table, the shape produced by any frequent writer:

CREATE TABLE local.db.events (id BIGINT, ts TIMESTAMP, region STRING, amount DOUBLE)
USING iceberg

After eight appends of 2,000 rows each, the table’s own metadata tables report:

datafiles=16   snapshots=8   manifests=8       (16,000 rows)

Sixteen data files for 16,000 rows, because two tasks wrote per append. Their sizes are the problem stated numerically:

00000-0-081691b0-...  rows=1000  bytes=10437
00000-10-0c64faa9-... rows=1000  bytes=9934
00000-12-b59f7d45-... rows=1000  bytes=9927

Ten kilobyte files. On object storage each of those is a separate GET with its own latency, and the Parquet footer of each has to be read before any data is.

Compaction rewrites them

CALL local.system.rewrite_data_files(
  table => 'db.events',
  options => map('min-input-files','2'))

rewritten_data_files_count = 16
added_data_files_count     = 1
rewritten_bytes_count      = 159392

Sixteen files became one, and the table now reports datafiles=1. The query side of the problem is solved.

But the space is not reclaimed

Counting actual files in the warehouse directory immediately afterwards:

parquet files on disk = 17
metadata.json files   = 11
avro (manifests+lists)= 28

Seventeen. The sixteen originals are still there, because snapshot 8 still references them and time travel to that snapshot must still work. The table logically has one data file; the storage bill is for seventeen.

Notice too that the snapshot count went from 8 to 9. Compaction is a write, so it commits a new snapshot. Running rewrite_manifests afterwards took manifests from 9 to 1 and pushed the snapshot count to 10. Maintenance operations generate history as they remove it, which is why they cannot substitute for expiry.

Expiry is what deletes

CALL local.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '2030-01-01 00:00:00',
  retain_last => 1)

deleted_data_files_count     = 16
deleted_manifest_files_count = 17
deleted_manifest_lists_count = 9

And on disk afterwards: 1 Parquet file, with all 16,000 rows still readable. The old data files, the manifests that listed them and the manifest lists that pointed at those are gone together, because nothing references them any more.

This is the operation people forget. A table compacted nightly and never expired grows forever while reporting a healthy file count.

The other two operations

rewrite_manifests addresses metadata rather than data. Manifests accumulate one per write, and a planner that has to open hundreds of them to decide which data files to read is slow before it has read any data. The measured run took 9 manifests to 1.

remove_orphan_files deletes files in the table’s directory that no snapshot references at all — the residue of failed or killed writes. It is distinct from expiry because those files were never committed, so no snapshot ever pointed at them and expiry will never consider them. It works by listing the directory and comparing against metadata, so it should be run with a conservative age threshold: a file written by a currently running job looks exactly like an orphan.

Hudi: what a Merge-on-Read table defers

A Merge-on-Read table takes the opposite trade. Updates are written as log files next to the base file rather than rewriting it, so writes are cheap and reads must do the merging.

After an initial insert of 20,000 rows across 4 partitions, then four rounds of upserts touching 5,000 rows each:

after initial insert    base=4   log=0    instants=4
after 4 upsert rounds   base=4   log=16   instants=16

Sixteen log files: four partitions times four upsert rounds. The base files were never rewritten. The timeline records what happened:

deltacommit            5
deltacommit.inflight   5
deltacommit.requested  5

deltacommit is the Merge-on-Read write instant. Each one exists in three states, requested then inflight then completed, which is how a reader tells a finished write from one still in progress.

The read-optimized query is stale, and here is the proof

Two query types read the same table. A snapshot query merges base files with their log files; a read-optimized query reads only the base files.

The row counts are identical, because the upserts updated existing records rather than adding new ones. The values are not. Every record began with ts = 1; 5,000 of them were updated to ts = 5, giving the correct sum of 15000×1 + 5000×5 = 40000. The read-optimized query returned 20,000, which is 20000×1 — it saw none of the updates.

A row count will never reveal this. Anyone validating a read-optimized pipeline by comparing counts will find them matching and conclude the data is current.

Compaction merges the logs in

CALL run_compaction(op => 'run', table => 't3')

timestamp          operation_size  state
20260923050058927  4               COMPLETED

And the reads afterwards:

Two things to take from that table. The read-optimized query now agrees, which is what compaction is for. And the file counts moved the way Iceberg’s did: base files went from 4 to 8 and the log files did not go anywhere. Compaction wrote new base slices; the old base files and the log files that fed them remain until cleaning removes them.

Compaction also writes a commit instant rather than a deltacommit, so the timeline distinguishes the merge from the writes that made it necessary.

Cleaning, and what I could not demonstrate

Cleaning is Hudi’s equivalent of snapshot expiry: it removes file slice versions older than the retention policy.

CALL run_clean(table => 't3', retain_commits => 1)

This returned an empty result — no rows, and no files deleted — with hoodie.cleaner.commits.retained set to 1 and cleaning triggered manually. The base and log file counts were unchanged afterwards. I was not able to establish from this run whether the retention policy genuinely had nothing eligible or whether the procedure needed different arguments, and I am reporting it as unresolved rather than describing behaviour I did not observe. The Iceberg expiry above was measured end to end; this one was not.

What is certain from the file counts is the structural point, which matches Iceberg exactly: compaction alone left every pre-compaction file on disk.

How the two compare

The deep architectural differences behind these choices are covered in open table formats in practice, and the Hudi side in Hudi Merge-on-Read.

What to schedule, and how often

Compaction or rewrite: as often as writes create small files. A streaming writer committing every minute needs this hourly; a nightly batch load may need it weekly. The signal is average file size, not a calendar.

Expiry or cleaning: as often as compaction, and never less. This is the one that controls storage cost, and its retention window is a real recovery capability. Retaining seven days of snapshots means seven days in which a bad write can be undone, and the storage for every file those snapshots reference.

Manifest rewriting: when planning gets slow. Watch how long query planning takes before the first task starts.

Orphan removal: rarely, with a conservative age threshold. Days, not hours. Deleting a file a running job is about to commit is a data-loss bug you caused with a cleanup job.

Sequence them. Compact first, then expire. Expiring before compacting leaves the small files as the current state, and the compaction that follows immediately creates a fresh generation of unreferenced files.

Symptoms and causes

Storage grows while the table’s row count is flat. Expiry or cleaning is not running. Compaction alone produces exactly this.

Queries slow down over weeks with no data growth. Small files accumulating, or manifests accumulating. Check average data file size, then manifest count.

A read-optimized Hudi query returns old values. Compaction has not run for those file groups. This is by design, and the fix is scheduling, not configuration.

Query planning takes longer than query execution. Metadata, not data. Rewrite manifests.

Time travel to a recent snapshot fails. Expiry retention is shorter than you think it is. The retention window and the recovery window are the same number.

Common misconceptions

“Compaction reclaims storage.” It rewrites data into fewer files and adds a snapshot. The measured Iceberg table held 17 Parquet files immediately after compacting 16 into 1.

“The table’s file count is the number of files on disk.” It is the number referenced by the current snapshot. The two diverge the moment anything is rewritten.

“Maintenance runs automatically.” Neither format schedules these for you by default. Hudi can be configured to run compaction and cleaning inline with writes; Iceberg expects you to call the procedures.

“A read-optimized query is just a faster snapshot query.” It is a query against base files only, and on a Merge-on-Read table it is stale by exactly the updates that have not been compacted.

“Row counts prove a pipeline is correct.” The Hudi table above had identical row counts and a 2× difference in summed values.

A model worth keeping

Every lakehouse table has two independent problems: too many files, and too many versions of files. Compaction and clustering fix the first. Expiry and cleaning fix the second. Each format names them differently and neither will do either unless you schedule it.

When storage grows and nothing explains it, you are compacting without expiring. When queries slow and the data has not grown, you are writing without compacting.

References

• Iceberg maintenance procedures for rewrite_data_files, expire_snapshots, rewrite_manifests and remove_orphan_files
• Hudi compaction and cleaning for the timeline operations and their trigger strategies
• Open table formats in practice for why the two storage models differ
• Hudi Merge-on-Read for the write path that produces the log files compaction merges
• Apache Iceberg architecture for the snapshot and manifest tree these operations rewrite

Trademarks

Apache Iceberg, Apache Hudi, Apache Spark, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.