Apache Iceberg 1.12.0: what's new, breaking changes, and upgrade guide

• 1. What’s new in 1.12.0
  • 1.1 Geospatial types arrive
  • 1.2 Hilbert clustering alongside Z-order
  • 1.3 Flink: equality deletes become deletion vectors
  • 1.4 Format v4 groundwork
  • 1.5 Variant, v3 interoperability, and encryption
  • 1.6 Scan planning, metrics, and maintenance
  • 1.7 Kafka Connect stops hiding failures
  • 1.8 REST catalog: protocol and operational changes
  • 1.9 Security and supply-chain work
• 2. Apache Iceberg 1.12.0 for Java and Iceberg Rust 0.11.0
• 3. Breaking changes
  • 3.1 Spark 3.4 and Flink 2.0 support removed
  • 3.2 Position deletes with row data (PDWR) cannot be written
  • 3.3 Deprecated methods removed
  • 3.4 Old S3 signer classes removed
• 4. Migrating from 1.11 to 1.12
  • 4.1 Pre-upgrade inventory
  • 4.2 Position deletes with row data (PDWR)
  • 4.3 Spark 3.4 and Flink 2.0 migration
  • 4.4 Deprecated API removals
  • 4.5 Testing checklist
• 5. Upgrade sequence in one page
• 6. Quick reference
• Conclusion
• References
• Trademarks

TL;DR

• Apache Iceberg 1.12.0 is a substantial release: 777 commits from 142 contributors.
• Geospatial types (geometry and geography) arrive as first-class types, stored as WKB, with Spark 4.1 supporting them end to end.
• Hilbert-curve clustering joins Z-order as a rewrite_data_files sort strategy; benchmark before adopting.
• Flink gains a maintenance task that converts equality deletes to deletion vectors, cutting read cost for upsert pipelines.
• Format version 4 plumbing is everywhere in the code, but V4 is not production-ready. Do not set format-version=4 on real tables.
• Breaking: Spark 3.4 and Flink 2.0 support are removed; PDWR can no longer be written, and old S3 signer classes are gone.
• Apache Iceberg 1.12.0 for Java and Iceberg Rust 0.11.0 are separate releases. Validate Rust encryption, Variant, v3 scans, and REST API changes independently.
• Upgrade by table behavior, not by dependency file. The PDWR and Spark 3.4 changes need a plan before you bump.

1. What’s new in 1.12.0

1.1 Geospatial types arrive

Iceberg now has first-class geometry and geography types, stored as WKB (Well-Known Binary). Both Avro and Parquet support reading and writing them with self-describing logical types. Spark 4.1 supports them end to end, including row-level DELETE, UPDATE, and MERGE.

Until now, spatial data in Iceberg meant a binary or string column plus a convention your team had to remember. Native types let engines and catalogs know what the column actually holds. This is a meaningful step for teams working with location data, mapping, or any spatial analytics workload.

Behavior change to watch: GeometryType and GeographyType now include the resolved CRS (Coordinate Reference System) and edge algorithm in their toString() output. A plain geometry now prints as geometry(OGC:CRS84), and geography prints as geography(OGC:CRS84, spherical). If you have tests or tooling that compare type strings, they may need updating. The community decided to keep this change because equal types printing differently looked like a bug, and function definition IDs built from parameter type strings would otherwise treat them as different overloads.

1.2 Hilbert clustering alongside Z-order

rewrite_data_files gains a second space-filling curve. Z-order is still there, and now you can also cluster on a Hilbert curve. Hilbert curves usually keep nearby values closer together than Z-order does, which can mean tighter file-level min/max statistics when you filter on several columns at once.

-- Existing: Z-order clustering
CALL my_catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'zorder(device_id, event_ts)'
);

-- New in 1.12: Hilbert curve clustering
CALL my_catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'hilbert(device_id, event_ts)'
);

Important details:

• hilbert() has no options of its own. Each column contributes its full 8-byte primitive width to the index, so Z-order knobs like var-length-contribution and max-output-size do not apply.
• You cannot combine hilbert() with zorder() in the same sort order.
• Do not assume Hilbert always wins. Benchmark it on your real query patterns before switching.

1.3 Flink: equality deletes become deletion vectors

This may be the most useful change for streaming teams. Flink upsert pipelines typically write equality deletes, which are cheap to write but expensive to read because every reader must join them against data files. The new ConvertEqualityDeletes maintenance task turns those into row-position deletion vectors (DVs), which readers can apply cheaply.

How it works: the Flink IcebergSink writes data files and equality deletes to a staging branch. The converter then resolves the deletes and commits the data files plus DVs to the target branch (usually main).

// Flink TableMaintenance, Iceberg 1.12
.add(ConvertEqualityDeletes.builder()
    .stagingBranch("staging")
    .equalityFieldColumns(ImmutableList.of("id"))
    .scheduleOnEqDeleteFileCount(10)
    .parallelism(4))

Requirements and caveats:

• Requires table format version 3 or higher.
• Does not support row lineage yet.
• equalityFieldColumns must match the columns your writer uses.
• The task keeps a primary-key index in Flink state; RocksDB is recommended.
• Avoid running other continuous writers against the target branch, or conversion cycles can keep losing the commit race.

1.4 Format v4 groundwork

Iceberg 1.12 contains a lot of code labeled v4, but format version 4 is not ready for production tables. This is implementation groundwork that shows where the format is heading.

What is present in the code:

• Relative paths in the evolving specification, and utilities for converting absolute locations to relative ones. This can make table metadata less tied to one bucket or storage prefix.
• Parquet or Avro manifests (instead of Avro-only), which can bring richer types and reader tooling to manifest data.
• Content-statistics structures and filters, carrying structured index and average-size information alongside tracked files.
• Tracked-file builders and adapters, plus a v4 manifest reader.
• A draft Mumbling Bitmap specification and a read-only implementation.
• EntryStatus.MODIFIED and TrackingBuilder status derivation, expressing changes that do not fit the older added/existing/deleted model cleanly.

What you should not do:

• Do not set format-version=4 on production tables.
• Do not build application code that parses V4 structures.
• Do not build a roadmap around draft field names.

The useful action today is to keep table movement and metadata assumptions out of application code, and follow the specification through later releases.

1.5 Variant, v3 interoperability, and encryption

Iceberg 1.12 fills in several engine paths for v3 features. Spark adds vectorized Parquet reads for Variant; Flink, Kafka Connect, and generic record writers gain Variant shredding paths. Java also strengthens v3 support for geometry and geography, row lineage, deletion vectors, and encrypted metadata. These features are useful only when every client that touches a table can interpret them. A table opening successfully is not proof that each engine supports every feature in it.

Before enabling a v3 feature, test the actual writer-reader combinations in your stack. Include schema evolution, nulls, decimal edge cases, and failure recovery. Keep the table format upgrade separate from the library upgrade so a compatibility failure has a clear cause.

1.6 Scan planning, metrics, and maintenance

The less visible changes in 1.12 may save more time than the headline features.

• Partition-statistics scans can now project columns and apply filters, reducing the amount of metadata an optimizer must read.
• Snapshot expiration correctly reads delete manifests in cases that previously went wrong.
• A scan-based action removes dangling delete files.
• RESTMetricsReporter.report() no longer blocks the calling thread, so emitting metrics cannot stall the work they describe. Still monitor queue size, delivery failures, and dropped reports.
• Adaptive split-sizing options are documented, making it easier to balance planning overhead against parallelism.

Reader and file-format fixes cover Arrow integer-to-long promotion, dictionary-encoded INT96 timestamps with offsets, large decimals, decimal default values, end-of-file handling in object-store streams, Parquet row-group size tracking, and faster decimal handling that avoids unnecessary BigInteger conversions.

1.7 Kafka Connect stops hiding failures

Kafka Connect’s most important change is simple: commit failures are surfaced instead of silently swallowed.

• Adds a metric for partial commit failures.
• Bounded retries for transient commit exceptions.
• Fixes rebalance scenarios that could commit files from a prior attempt.
• Tracks control-topic offsets as a high-water mark.

Schema and conversion fixes cover null-valued records after a schema change, UUID conversion from Avro, decimal inference, MongoDB timestamp and date arrays, and Variant shredding. Removed deprecated members in TableReference and IcebergWriterResult may affect custom transforms or extensions at compile time.

1.8 REST catalog: protocol and operational changes

Iceberg 1.12 expands the REST catalog contract. The OpenAPI model adds table-unregister and function list/load endpoints, Variant types, catalog object identifiers, labels, and remote-signing configuration. Path segments now use RFC 3986 percent-encoding, fixing interoperability for namespaces and object names with reserved characters. Spark can delegate DROP TABLE PURGE to the REST catalog so the server can enforce storage-deletion policy.

The changes affect both compatibility and operations:

• Check capability negotiation. Confirm that the deployed catalog server and any proxy implement the endpoints and request/response shapes your Java or Rust client needs. Do not infer support from a product name or version string alone.
• Test names end to end. Exercise reserved characters through client, proxy, and server. Inconsistent decoding can address the wrong object or make it unreachable.
• Review authorization. Test permissions for unregister, function metadata operations, remote signing, and purge delegation.
• Define deletion ownership. Unregister removes the catalog registration but leaves files behind; purge requests file deletion. Record the table metadata location and recovery procedure before either operation.
• Drill purge failures. Verify behavior when storage deletion is denied, partially succeeds, or times out. Establish whether the table remains registered, whether cleanup can be retried, and where the action is audited.
• Check authentication refresh. Exercise OAuth token refresh, SigV4/assumed-role signing, and retries against the actual catalog implementation.

Test Java 1.12.0 and Rust 0.11.0 clients against the same catalog where both are deployed. Their independent release versions do not guarantee identical REST feature support.

1.9 Security and supply-chain work

• The project published an Iceberg security model describing trust boundaries across catalogs, clients, storage, credentials, and metadata.
• CI now makes vulnerability scanning blocking on relevant pull requests.
• Jackson versions were aligned and updated to address reported vulnerabilities.
• OAuth token refresh keeps optional parameters during non-exchange refreshes.
• AWS REST signing can use assumed-role credentials.
• RemoteSigningConfig implements the formalized remote signing spec, and old S3 signer classes are removed.

A library upgrade does not secure an Iceberg deployment by itself. Use the security model to review who can alter catalog metadata, who can request temporary storage credentials, who can delete files, where tokens are cached, and how audit events connect catalog actions to object-store actions.

2. Apache Iceberg 1.12.0 for Java and Iceberg Rust 0.11.0

These are companion releases in the Iceberg ecosystem, not matching version numbers for the same library. Java Iceberg 1.12.0 and Iceberg Rust 0.11.0 have independent release artifacts, APIs, and feature coverage. Upgrade the implementations separately, then validate their interaction wherever they share a catalog or table. The linked release coverage highlights end-to-end table encryption, stronger v3 reads, Variant support, row-lineage metadata, and REST catalog changes. For teams with Rust readers or writers, these can close important cross-engine gaps, but validate the final release artifacts and changelog before adopting them.

The Rust changes to review include:

• Encryption: Rust reads and writes encrypted data, manifests, manifest lists, and Puffin files, with AES-256 support and a KMS factory integration. Test encrypted files written by Rust with Java readers, and vice versa, including key rotation and missing-key failures.
• V3 scans: Rust adds deletion-vector decoding/application, Variant support, row-lineage metadata columns, and additional metadata columns. Test tables containing each feature with your actual catalog, storage, and other engines.
• REST authentication: the AuthManager / AuthSession API changes authentication setup. Review OAuth2 and SigV4 configuration and confirm the client respects endpoints advertised by GET /v1/config.
• API and behavior changes: review lazy table-property parsing, validated FileScanTask construction, write.metadata.path, and the GCS host property change. Rebuild downstream Rust applications and integration crates.
• DataFusion: check the DataFusion version line required by the selected Rust release and track the integration’s separate release lifecycle.

The key upgrade rule is cross-engine testing. A file accepted by the Rust writer must also be readable by every Java, Spark, Flink, and query-engine client that shares the table. Test both directions before enabling new table features in production.

3. Breaking changes

This is the section to share with whoever owns your Spark and Flink jobs. Several removals land in 1.12.0, and they break code, not just deprecate it.

3.1 Spark 3.4 and Flink 2.0 support removed

PR #14122 removes the Spark 3.4 runtime entirely. The 1.12.0 Java artifacts support Spark 3.5, 4.0, and 4.1, with version bumps to Spark 3.5.9, 4.0.4, and 4.1.3. Spark 4.2 is excluded from the source and binary artifacts, even though its module is present on the project main branch.

Flink 2.0 support is also removed. Iceberg 1.12 adds Flink 2.2 and 2.3 while retaining Flink 1.20 and 2.1. Inventory both batch and streaming jobs, then test the exact Flink/Iceberg pair. For stateful jobs, include savepoint restore, checkpoint recovery, serializers, and commit behavior—not only compilation.

3.2 Position deletes with row data (PDWR) cannot be written

PR #17706 removes the feature the community deprecated in 1.11. The rewrite_position_delete_files and rewrite_table_path actions now fail with an explicit exception on files that still carry row data.

This was a deliberate community decision. At the August 26 sync, the group agreed that table owners should make explicit decisions about existing PDWR rather than having maintenance actions silently drop the row column.

3.3 Deprecated methods removed

• Deprecated SparkTableUtil methods in Spark 4.0 and 4.1 are removed.
• SparkFilters is deprecated in favor of SparkV2Filters.

3.4 Old S3 signer classes removed

PR #17627 drops deprecated AWS signer classes and properties. The new RemoteSigningConfig implementation follows the formalized remote signing spec.

4. Migrating from 1.11 to 1.12

Upgrade by table behavior, not by dependency file. The following sequence catches the failures most likely to escape a basic smoke test.

4.1 Pre-upgrade inventory

Build an engine and feature inventory before touching anything:

This inventory tells you which breaking changes affect you and which do not.

4.2 Position deletes with row data (PDWR)

If you still run V2 tables with position deletes that include row data, your maintenance jobs will start failing after the upgrade. You have two ways out.

Option A: upgrade to V3 and rewrite as deletion vectors.

-- Upgrade table format version
ALTER TABLE prod.db.trips SET TBLPROPERTIES ('format-version' = '3');

-- Then rewrite position delete files (now allowed on V3)
CALL prod.system.rewrite_position_delete_files(table => 'db.trips');

Option B: stay on V2 and let data compaction fold those deletes into data files.

-- Run data compaction to rewrite position deletes into data files
CALL prod.system.rewrite_data_files(
  table => 'db.trips',
  strategy => 'sort'
);

Pick one before you upgrade, not after the first failed job.

How to find affected tables:

-- Query position deletes across tables to find PDWR
SELECT * FROM prod.db.trips.position_deletes LIMIT 10;
-- If the row column is populated, this table has PDWR

4.3 Spark 3.4 and Flink 2.0 migration

If any job runs Spark 3.4 or Flink 2.0, move it to a supported runtime before upgrading that integration to Iceberg 1.12.0. For Flink, the supported lines are 1.20, 2.1, 2.2, and 2.3.

Recommended path:

1. Upgrade Spark 3.4 jobs to Spark 3.5.x first (3.5.9 is the 1.12.0 baseline). Upgrade Flink 2.0 jobs to a supported Flink line.
2. Run engine and state-recovery tests on the new runtime.
3. Then upgrade the Iceberg integration to 1.12.0.

Spark 4.2 users should stay on Iceberg 1.11.x until a later Iceberg release adds official 4.2 artifacts. Do not treat Flink 2.0 as supported by substituting a neighboring runtime artifact.

4.4 Deprecated API removals

Search your codebase for:

• SparkTableUtil method calls (in Spark 4.0/4.1 projects)
• SparkFilters usage (replace with SparkV2Filters)
• Any code referencing removed S3 signer classes or properties

These fail at compile time, not runtime, so a full build catches them.

4.5 Testing checklist

After upgrading in staging, run these targeted tests:

For Rust clients, compile against the target release and test REST authentication, metadata paths, GCS configuration, encrypted file interoperability, and v3 scans. For every shared table, verify that each active client can read all features written by the others.

For Kafka Connect specifically, the release notes recommend three forced-failure drills: make a catalog commit fail, trigger a task rebalance during a pending commit, and restart from a known control-topic offset. Confirm the connector reports the failure, avoids duplicate table data, and resumes from the expected point.

5. Upgrade sequence in one page

The order that catches the failures most likely to escape a smoke test:

1. Build an engine and feature inventory. List every Spark, Flink, Kafka Connect, Python, Rust, query-engine, and maintenance client that touches each catalog. Record its runtime version, Iceberg library version, catalog client, authentication method, and file I/O implementation.
2. Inventory table format versions and delete-file strategies. Any V2 table writing PDWR needs a plan before upgrade.
3. Test the specific changes that matter to your tables. Arrow integer-to-long promotion, dictionary-encoded INT96 timestamps with offsets, large decimals, decimal default values, end-of-file handling in object-store streams, Parquet row-group size tracking.
4. Force failure drills on Kafka Connect. Make a catalog commit fail, trigger a task rebalance during a pending commit, restart from a known control-topic offset. Confirm the connector reports the failure, avoids duplicate table data, and resumes from the expected point.

6. Quick reference

Conclusion

Apache Iceberg 1.12.0 for Java and Iceberg Rust 0.11.0 are separate releases that reward coordinated planning. Upgrade each client deliberately, and validate shared tables and REST catalog behavior across both implementations. Geospatial types and Hilbert clustering are worth testing; Flink equality-delete conversion is worth evaluating for streaming CDC pipelines. Plan for the Spark 3.4 and Flink 2.0 removals, along with the PDWR maintenance boundary, before upgrading.

The V4 code is worth watching, not using. The specification work on relative paths, Parquet manifests, and content statistics shows where the format is heading, but setting format-version=4 on production tables today is not a supported path.

When you move off this release, re-check the format version range first, then the defaults in the properties table, then whether SparkProcedures has grown: procedures are the part of the surface that changes most between releases.

References

• Apache Iceberg 1.12.0 release notes
• Apache Iceberg 1.12.0 GitHub release
• Dremio: Apache Iceberg 1.12.0 what’s new, breaking changes, and upgrade guide
• Dremio: preview of upcoming Apache Iceberg 1.12 and Iceberg Rust 0.11
• Apache Mail Archives: PDWR handling discussion
• Apache Iceberg on LinkedIn: 1.12.0 announcement
• Apache Hudi on Spark: the complete cheat sheet
• Apache Iceberg on Spark: the complete cheat sheet

Trademarks

Apache Iceberg, Apache Spark, Apache Flink, Apache Hive, Apache Parquet, Apache Avro, Apache Hudi, Apache Polaris (incubating) and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Delta Lake is a trademark of the Linux Foundation.