All posts

Iceberg format versions: what v1, v2, v3 and v4 actually change

The format version decides whether a table can delete a row without rewriting a file, and whether every row carries an identity. All four versions were created and written against the same engine. Here is what each adds and what upgrading costs.

7 min read Iceberg

TL;DR

  • Creating tables at format-version 1, 2, 3 and 4 all succeeded on the same engine, and all accepted writes and deletes.
  • v2 is the default for new tables and the version that introduced delete files, which is what merge-on-read needs.
  • v3 and v4 metadata carry next-row-id; v1 and v2 do not. That field is the row-lineage machinery.
  • The version only moves forward, and it is a table property, so a table upgraded past what a reader supports becomes unreadable to that reader.
  • A DELETE on a v1 table still works — by rewriting files. Version does not decide whether you can delete, only how.

What a format version is

It is the version of the on-disk specification a table is written to: which metadata fields exist, which file types are legal, what a reader must understand.

It is recorded as a table property, so it is per table rather than per cluster:

CREATE TABLE local.db.events (id BIGINT, v STRING) USING iceberg
TBLPROPERTIES ('format-version' = '2');

Every version was exercised against the same engine:

format-version=1: CREATE ok, rows=3, metadata format-version=1, extra_keys=[]
format-version=2: CREATE ok, rows=3, metadata format-version=2, extra_keys=[]
format-version=3: CREATE ok, rows=3, metadata format-version=3, extra_keys=['next-row-id']
format-version=4: CREATE ok, rows=3, metadata format-version=4, extra_keys=['next-row-id']

All four create, write and delete. The difference is what the metadata contains and what the engine is permitted to do.

v1: the table is a list of files

v1 established the idea the whole format rests on: a table is a set of data files named in metadata, committed atomically, with schema and partitioning tracked independently of directory layout.

That gives atomic commits, time travel, schema evolution by field id, hidden partitioning and partition evolution. Everything in the introduction to Iceberg is v1.

What it lacks is any way to express “this row is gone” other than rewriting the file that holds it. A DELETE against a v1 table still works:

format-version=1 after DELETE: data files=2, rows=2

It rewrote. For a table updated occasionally that is fine and is exactly what copy-on-write does on any version.

v2: delete files

v2 added delete files — separate files that mark rows as removed without touching the data file. That single addition is what makes merge-on-read possible, and therefore what makes CDC and streaming upserts practical.

Two kinds exist. Position deletes name a data file and row positions within it. Equality deletes are predicates on column values, which lets a writer delete without first finding where the row lives — the reason Flink CDC pipelines produce them.

v2 is the default for new tables, which is why a table created without specifying anything reports:

format-version = 2
write.parquet.compression-codec = zstd

The cost is that readers must apply delete files. An engine that ignores them returns deleted rows, which is worse than failing. The trade, measured on identical tables, is in copy-on-write or merge-on-read.

v3: deletion vectors and row lineage

v3 changes how deletes are represented and adds identity to rows.

Deletion vectors replace position delete files with a bitmap per data file, stored in a Puffin file. A bitmap is more compact than a list of positions and faster to apply, and crucially there is one per data file rather than one per delete operation — which directly attacks the accumulation problem where every merge adds another delete file.

Row lineage gives each row an identity that survives rewrites, so a row can be followed across compactions. The metadata evidence is visible immediately:

format-version=3: extra_keys=['next-row-id']

next-row-id is the counter that assigns those identities. It appears in v3 and v4 metadata and in neither v1 nor v2. That is the cleanest way to tell which generation a table belongs to without parsing the whole file.

v3 also brings default values for columns, so adding a column with a default does not require rewriting existing files to materialise it.

v4

v4 is the highest version the current library reads and writes. A table created at v4 behaves like v3 in the respects measured here, including carrying next-row-id.

Being able to write a version is not a reason to use it. The constraint is never your writer; it is every reader that touches the table, including engines outside your control. Treat v4 as available for experiments and v2 or v3 as the production choice, depending on whether all your readers understand deletion vectors.

Upgrading

ALTER TABLE local.db.events SET TBLPROPERTIES ('format-version' = '3');

Three properties of that statement matter.

It only goes up. There is no downgrade. A table upgraded past what a reader supports is unreadable by that reader until the reader is upgraded, and the recovery is to restore a copy, not to revert the property.

It does not rewrite data. Existing files stay as they are. The upgrade changes what future writes may do.

It is a compatibility decision, not a performance one. The question is never “is v3 better” — it is “does every engine that reads this table support v3”. Inventory your readers first, including the ones that belong to other teams.

A safe sequence: upgrade a copy, point each reader at the copy, confirm all of them work, then upgrade the original.

Which version to use

Situation Version
Append-only table, readers of unknown vintage v1 or v2
Anything with UPDATE, DELETE or MERGE v2
CDC or streaming upserts with heavy delete churn v3, if readers support deletion vectors
You need to track a row across rewrites v3
Experimenting with the newest spec v4

v2 is the right default, and it is the default for a reason: it covers row-level operations and is universally supported by engines that support Iceberg at all.

Move to v3 when delete-file accumulation is a measured problem on your CDC tables and you have confirmed every reader handles deletion vectors — not because the number is higher.

Common misconceptions

“v1 tables cannot delete rows.” They can, by rewriting files. v2 added deleting without rewriting.

“The format version controls copy-on-write versus merge-on-read.” The write.*.mode properties control that. The format version determines whether merge-on-read is available at all.

“Upgrading improves performance.” It enables representations that can. A v3 table that never uses deletion vectors performs like a v2 table.

“I can downgrade if something breaks.” You cannot. Test on a copy.

“Newer is safer.” Newer is less widely supported. The risk is a reader you forgot about.

A model worth keeping

v1 made a table a tracked list of files. v2 made a row removable without rewriting its file. v3 made that removal a bitmap and gave rows an identity. v4 is where the spec is being written next.

The version is a contract with your readers, not a tuning knob. Pick the lowest version that expresses what your workload needs, and upgrade when a measured problem — not a release note — requires it.

References

Trademarks

Apache Iceberg, Apache Spark, Apache Flink, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee