Running Iceberg in production: file sizes, object-storage paths, and the jobs you must schedule

• What actually degrades
• Object-storage paths
• Target file size
• Sizing files against your query shape
• A monitoring query set
• Where the cost actually goes
• A production checklist
• The three jobs
• Metadata retention, which is separate
• The catalog decision
• Properties worth setting at create time
• Automating it
• Common misconceptions
• A model worth keeping
• References
• Trademarks

TL;DR

• write.object-storage.enabled changes data paths to carry a hashed prefix — the measured table wrote to 0111/0110/0011/10101101/.... That distributes keys across object-store partitions instead of concentrating them under one date prefix.
• write.target-file-size-bytes is a target the writer aims at, not a guarantee. A 1 MiB target produced files of 124,966 and 496,738 bytes from one write.
• Three jobs must be scheduled from day one: compaction, delete-file compaction, and snapshot expiry. Each fixes something the others do not.
• The catalog is the production decision that is hardest to reverse. A path-based catalog cannot commit safely on plain object storage.
• Metadata files accumulate independently of snapshots, and are bounded by their own settings.

What actually degrades

An Iceberg table rarely fails. It degrades, in four ways that have different causes and different fixes, and telling them apart is most of production operation.

Notice that “the query got slow” maps to three different causes. The way to tell them apart is where the time goes: planning time before tasks are scheduled points at metadata; task time points at data files or delete files.

Object-storage paths

The default layout writes data under a directory derived from partition values:

data/pickup_day=2026-02-10/00000-2-....parquet

That is readable, and on S3 it is a problem at scale. Object stores partition their keyspace by prefix, so a table whose keys all begin data/pickup_day=2026-02-10/ concentrates a day’s requests onto one internal partition. The symptom is throttling — 503 responses under load — on a table that looks perfectly healthy.

Setting write.object-storage.enabled changes the path:

CREATE TABLE local.db.prod (id BIGINT, v STRING) USING iceberg
TBLPROPERTIES ('write.object-storage.enabled'='true')

The measured result:

0111/0110/0011/10101101/00000-66-7fc327bf-3edb-40a3-b09f-686852b...
1011/1010/0010/00000001/00000-66-7fc327bf-3edb-40a3-b09f-686852b...

Those leading binary components are a hash of the file path, inserted before the rest of the location. Two files written in the same operation landed under completely different prefixes, which is the point: requests spread across the keyspace instead of queuing behind one.

The cost is that the layout is no longer human-navigable. You cannot list a partition’s directory to see its files, because the files are not in one. The metadata knows where everything is, which is the only consumer that matters, but it does change how you debug.

Turn this on for tables on S3 or GCS that take real query concurrency. Leave it off for small tables, local filesystems, and anywhere a human browsing the layout is worth more than the throughput.

Target file size

TBLPROPERTIES ('write.target-file-size-bytes'='1048576')

Setting a 1 MiB target and writing 400,000 rows produced:

files written = 2
   124966 bytes    81000 rows
   496738 bytes   319000 rows

Neither file hit the target, and they differ from each other by a factor of four. This is worth internalising: the property is an input to the writer’s decisions, not a contract. Actual sizes depend on how the data was partitioned coming in, how well it compressed, and where the writer chose to split.

The practical reading: target file size shapes compaction more reliably than it shapes ingestion. rewrite_data_files bin-packs toward the target and gets close, because it is reading existing files and has freedom to choose boundaries. An ingesting writer is bounded by the partitions it was handed.

For production the usual target is 128 MiB to 512 MiB — large enough that per-file overhead is amortised, small enough for useful parallelism. The default is 512 MiB, and the reason to lower it is a table that is read with very selective filters, where smaller files prune better.

Sizing files against your query shape

The default target is 512 MiB, and the right value depends on how selectively your queries read.

Measure what you actually have rather than what you configured, since the property is a target and not a guarantee:

SELECT
  count(*) AS files,
  round(min(file_size_in_bytes)/1048576, 1)  AS min_mb,
  round(avg(file_size_in_bytes)/1048576, 1)  AS avg_mb,
  round(max(file_size_in_bytes)/1048576, 1)  AS max_mb,
  round(percentile_approx(file_size_in_bytes, 0.5)/1048576, 1) AS p50_mb
FROM local.db.events.files WHERE content = 0;

A median far below the target means compaction is not keeping pace with ingestion; a wide spread between min and max means partitions are receiving very different volumes and a single global target is the wrong instrument.

A monitoring query set

Five queries cover the failure modes. Run them on a schedule and alert on trends rather than absolutes.

-- 1. file health: small files and delete accumulation in one row
SELECT content,
       count(*) AS files,
       round(avg(file_size_in_bytes)/1048576, 1) AS avg_mb,
       sum(record_count) AS records
FROM local.db.events.files GROUP BY content ORDER BY content;

-- 2. metadata health: what planning must read
SELECT count(*) AS manifests, round(sum(length)/1048576, 1) AS manifest_mb
FROM local.db.events.manifests;

-- 3. history: is expiry running?
SELECT count(*) AS snapshots, min(committed_at) AS oldest, max(committed_at) AS newest
FROM local.db.events.snapshots;

-- 4. commit rate: the predictor of every metadata problem
SELECT date_trunc('day', committed_at) AS day, count(*) AS commits
FROM local.db.events.snapshots GROUP BY 1 ORDER BY 1 DESC LIMIT 14;

-- 5. metadata file accumulation, which expiry does not bound
SELECT count(*) AS metadata_files FROM local.db.events.metadata_log_entries;

Query 4 is the leading indicator. Commits per day predicts manifests, metadata files and snapshot count; watching it catches problems weeks before anything is slow.

Where the cost actually goes

Object storage bills on three axes and Iceberg touches all three differently.

Storage is dominated by unexpired snapshots, not by live data. A table compacting nightly and never expiring holds every pre-compaction generation. expire_snapshots is the only lever, and its retention is your recovery window.

Request count is dominated by file count and by listing. Iceberg never lists directories during planning, which removes the Hive-era cost entirely, but small files still mean more GETs per scan. This is what write.object-storage.enabled addresses for throttling and what compaction addresses for volume.

Egress and compute follow bytes scanned, which is what pruning controls — partition transforms matching your filters, and sort orders narrowing column bounds.

A concrete shape: a table with 10,000 small files scanned fully costs 10,000 GETs; compacted to 200 files it costs 200. Neither changes the bytes, and both change the bill and the latency.

A production checklist

Before a table carries real traffic:

• Catalog is not path-based if storage is an object store
• format-version set deliberately, and every reader supports it
• Partition transform matches the filters queries actually use
• write.target-file-size-bytes set for the query shape
• write.object-storage.enabled on, for S3 or GCS
• write.*.mode chosen per operation, not left to the default
• Sort order declared if filters go beyond the partition column
• write.metadata.delete-after-commit.enabled on, with a version cap
• Compaction scheduled, scoped with a where clause
• rewrite_position_delete_files scheduled if merge-on-read
• expire_snapshots scheduled, retention = recovery window
• Monitoring on commit rate and the delete-to-data ratio

The ordering matters: the first six are decisions that are expensive to change later, and the last six are jobs that are cheap to add at any time but costly to have omitted for months.

The three jobs

These are not optional and they are not the same job.

Compaction merges small data files:

CALL local.system.rewrite_data_files(
  table => 'db.events',
  options => map('min-input-files','5','target-file-size-bytes','536870912'));

Delete-file compaction applies and clears delete files, which data compaction does not fully do:

CALL local.system.rewrite_position_delete_files(table => 'db.events');

Expiry is the only one that reclaims storage:

CALL local.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '2026-09-17 00:00:00',
  retain_last => 10);

Compaction without expiry grows storage while improving query time, because the pre-compaction files stay referenced by older snapshots. That accounting, measured file by file, is in table maintenance for Iceberg and Hudi.

A fourth, run when planning is slow rather than on a schedule:

CALL local.system.rewrite_manifests(table => 'db.events');

Retention is a recovery decision, not a cleanup decision. older_than sets how far back you can time travel and therefore how far back you can undo a bad write. Seven days of snapshots is seven days of insurance and seven days of storage for every file those snapshots reference. Decide it as a recovery policy and let the storage follow.

Metadata retention, which is separate

Every commit writes a new metadata.json. Expiring snapshots does not remove them; they have their own settings:

ALTER TABLE local.db.events SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled'='true',
  'write.metadata.previous-versions-max'='100');

Without these, a table committing every minute accumulates metadata files indefinitely — the directory grows even though snapshots are being expired on schedule. This is the specific fix for a metadata directory with tens of thousands of JSON files, and it is easy to miss because it looks like the snapshot problem and is not.

The catalog decision

The hardest thing to change later.

A catalog’s job is to answer which metadata file is current and to make the swap atomically. That atomicity is what stops two writers from losing each other’s commits.

A path-based catalog on plain object storage is the configuration to avoid. It depends on atomic rename semantics that object stores do not offer, which turns concurrent commits into lost commits — and it fails silently, not loudly.

The REST catalog is the direction the ecosystem has settled on, because it puts the commit protocol behind an API that any engine can speak, rather than each engine reimplementing filesystem conventions. A local walkthrough is in a local Iceberg playground, and a production-shaped open-source implementation in running Iceberg on Apache Polaris.

Properties worth setting at create time

Set these when the table is created. Several can be changed later, but the ones that affect layout only apply to data written afterwards, so a table that ran for six months with the wrong settings needs a rewrite, not an ALTER.

Automating it

Maintenance that depends on someone remembering is maintenance that stops. The shape that works:

hourly   rewrite_data_files on hot tables
         rewrite_position_delete_files on CDC tables
daily    expire_snapshots  (retention = your recovery window)
weekly   rewrite_manifests where planning is slow
monthly  remove_orphan_files with a conservative age threshold

Two operational notes. remove_orphan_files deletes files no snapshot references, and a file being written right now looks exactly like an orphan — use an age threshold of days, and never run it while a long write is in flight. And maintenance jobs are writers: they commit, they can conflict with your pipelines, and they need their own compute rather than competing with the ingestion they are meant to help.

Common misconceptions

“Target file size guarantees file size.” The measured write produced 124,966 and 496,738 bytes against a 1 MiB target.

“Compaction is maintenance.” It is one of three jobs, and the only one that reclaims storage is expiry.

“Object storage mode is just a layout preference.” It is the fix for prefix throttling, and it changes the failure mode under concurrency.

“Expiring snapshots is cleanup.” It is deleting your recovery window. Choose the retention as a recovery policy.

“The catalog can be swapped later.” It can, with a migration. It is much cheaper to choose correctly at the start.

A model worth keeping

An Iceberg table has two clocks. One counts data files and delete files, and compaction resets it. The other counts snapshots, manifests and metadata files, and expiry and manifest rewriting reset it.

Production operation is keeping both wound, choosing a catalog that can commit safely, and setting the layout properties before the first write rather than after the first incident.

References

• Iceberg configuration for every write property and its default
• Iceberg Spark procedures for the maintenance calls above
• Iceberg AWS integration for object-storage paths and S3 behaviour
• Table maintenance for Iceberg and Hudi for the compaction-versus-expiry accounting
• CDC into Iceberg for the delete-file growth that drives most of this schedule

Trademarks

Apache Iceberg, Apache Spark, Apache Hive, Apache Parquet, Apache Polaris, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.