Iceberg catalogs: the one pointer that makes a table a table

• The job
• The implementations
• Configuring each one
• Where the pointer actually lives
• Registering a table in a different catalog
• The choice is visible on disk
• Choosing one
• What changing later costs
• Trying one locally
• Common misconceptions
• A model worth keeping
• References
• Trademarks

TL;DR

• A catalog’s whole job is to hold one pointer per table and swap it atomically. Everything else it does is convenience.
• The choice is visible on disk: a Hadoop catalog writes v1.metadata.json plus version-hint.text; a REST catalog writes 00000-<uuid>.metadata.json and no hint file. Both verified in one session.
• A path-based catalog needs atomic rename, which object stores do not provide — the reason it is unsafe on S3 and fine on a local filesystem.
• REST decouples engines from the metadata backend: the engine speaks HTTP, the service owns the commit.
• Catalog choice is the hardest thing to change later, because table identity lives in it.

The job

An Iceberg table is a tree of metadata files pointing at data files. The tree is immutable — every commit writes new files. So something must record which root is current, and must change that record atomically.

flowchart LR
    E[engine] -->|"which metadata for db.trips?"| C[catalog]
    C -->|"s3://.../00001-....metadata.json"| E
    E -->|"commit: swap if base unchanged"| C

That is the contract. The catalog is not a query engine, does not store data, and in most implementations does not even read the metadata files it points at. It holds a pointer and swaps it safely.

Everything about concurrency follows: a commit succeeds only if the pointer still holds the value the writer started from. Two writers cannot both win, so no write is lost. The measured behaviour — two concurrent writers, 13 snapshots, a linear parent chain, no lost rows — is in life of a write query.

The implementations

Hadoop (path-based) keeps the pointer in the filesystem, as version-hint.text naming the current version number. No service to run, which is why every tutorial starts here. It depends on atomic rename and on listing being consistent — true on HDFS and a local disk, not true on S3, where two concurrent commits can both believe they won.

Hive Metastore stores the pointer in the metastore database, making the commit a database transaction. The right choice when you already run a metastore and want Hive-era tools to see the tables. It inherits the metastore’s operational profile, including being a single regional service.

JDBC keeps the pointer in any relational database. Atomicity comes from the database. Light to run if you already have Postgres, and the schema is simple.

REST defines the catalog as an HTTP API rather than a storage layout. The engine calls loadTable and updateTable; the service behind it may use a database, a metastore, or a vendor system. This is where the ecosystem has settled, because it stops every engine from reimplementing commit protocols and lets the backend change without touching the engines.

Vendor catalogs — Polaris, Glue, Nessie, Unity — are backends behind one of those interfaces, usually REST, adding governance, access control and multi-engine credential vending.

Configuring each one

The Spark side of every catalog is the same four or five properties, which makes switching mostly a matter of knowing the type name.

# REST - the production default
--conf spark.sql.catalog.prod=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.prod.type=rest
--conf spark.sql.catalog.prod.uri=https://catalog.internal:8181
--conf spark.sql.catalog.prod.warehouse=s3://lake/warehouse

# Hive Metastore
--conf spark.sql.catalog.hive_cat=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.hive_cat.type=hive
--conf spark.sql.catalog.hive_cat.uri=thrift://metastore:9083

# JDBC
--conf spark.sql.catalog.jdbc_cat=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.jdbc_cat.type=jdbc
--conf spark.sql.catalog.jdbc_cat.uri=jdbc:postgresql://db:5432/iceberg
--conf spark.sql.catalog.jdbc_cat.jdbc.user=iceberg

# Hadoop - development only
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.local.type=hadoop
--conf spark.sql.catalog.local.warehouse=/tmp/warehouse

There is a fifth arrangement worth knowing: SparkSessionCatalog replaces Spark’s built-in spark_catalog, so Iceberg tables and ordinary Hive tables coexist under unqualified names. That is what the migration procedures need:

--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog
--conf spark.sql.catalog.spark_catalog.type=hive

Several catalogs can be configured at once, which is how a migration or a cross-catalog read works:

SELECT a.id, b.segment
FROM prod.db.orders a JOIN hive_cat.db.customers b ON a.customer_id = b.customer_id;

Where the pointer actually lives

Each implementation stores the same thing — a path to the current metadata file — somewhere different, and the difference is the whole safety argument.

Hive-backed catalogs make this visible: BaseMetastoreTableOperations stores metadata_location and previous_metadata_location as table properties, and the commit is a conditional update — set metadata_location to the new value only if it currently equals the old one. That comparison is the compare-and-swap, and a database is a reasonable place to perform it.

A path-based catalog has no such primitive on object storage, which is the entire reason it is unsafe there.

Registering a table in a different catalog

Migrating catalogs does not move data — it re-points a new catalog at metadata that already exists:

CALL prod.system.register_table(
  table         => 'db.events',
  metadata_file => 's3://lake/warehouse/db/events/metadata/00042-....metadata.json');

Find the current metadata file from the old catalog first:

SELECT file FROM old_cat.db.events.metadata_log_entries
ORDER BY timestamp DESC LIMIT 1;

Two cautions. Only one catalog should own a table at a time — two catalogs pointing at the same table will both accept commits and neither will see the other’s, which corrupts the table exactly the way concurrent writers are supposed to be prevented from doing. And drop the old registration without purging:

DROP TABLE old_cat.db.events;          -- metadata entry only
-- NOT: DROP TABLE old_cat.db.events PURGE;   -- this deletes the data files

That PURGE distinction has cost people their tables.

The choice is visible on disk

Same Iceberg version, same engine, two catalogs. A Hadoop catalog produces:

metadata/v1.metadata.json
metadata/v2.metadata.json
metadata/version-hint.text
metadata/snap-3527536884859544448-1-53fea6b1-....avro
metadata/53fea6b1-....-m0.avro

A REST catalog produces:

metadata/00000-fbb1be3d-4ab4-46bc-acb5-4df4ba54eef6.metadata.json
metadata/00001-9804ab23-868a-4cbe-ba66-b851437255be.metadata.json
metadata/snap-7554101406213169966-1-c4295385-....avro
metadata/c4295385-....-m0.avro

Two differences. The Hadoop catalog numbers metadata files sequentially and writes version-hint.text, because it finds the current metadata by reading that file. The REST catalog names them with a UUID and writes no hint, because the pointer lives in the service.

That sequential numbering is exactly what makes the Hadoop catalog unsafe on object storage: deciding the next version number requires knowing the current one, and two writers can reach the same answer.

If you have read two Iceberg walkthroughs that describe the metadata directory differently, neither was wrong — they used different catalogs.

Choosing one

The rule that matters: do not run a path-based catalog on object storage. It appears to work, and the failure — a lost commit under concurrency — is silent and arrives at the worst time.

Beyond that, the questions are about who else touches the tables. One engine and one team can live with a metastore or JDBC catalog for a long time. Several engines, several teams, and access control that needs to be consistent between them is what REST and the vendor catalogs exist for.

What changing later costs

Table identity lives in the catalog: prod.db.trips means whatever the catalog says it means. Migrating catalogs means re-registering every table, and every consumer’s fully qualified names change.

The data and metadata files do not move — register_table points a new catalog at an existing metadata file — so it is not a data migration. It is a coordination problem across every job, dashboard and notebook that names a table.

Which is why this is the decision to make deliberately at the start, and the reason to prefer REST even when something simpler would do today: the interface outlives the backend.

Trying one locally

A REST catalog is one container. The configuration on the Spark side is four properties:

--conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.demo.type=rest
--conf spark.sql.catalog.demo.uri=http://rest:8181
--conf spark.sql.catalog.demo.warehouse=s3://warehouse/

Check the catalog before debugging anything else:

curl -s http://localhost:8181/v1/config

If that returns nothing, every Spark failure is a catalog failure and no amount of storage configuration will help. The full three-service setup is in a local Iceberg playground, and a production-shaped open-source catalog in running Iceberg on Apache Polaris.

Common misconceptions

“The catalog stores the table.” It stores a pointer. The table is metadata and data files in storage.

“Any catalog works anywhere.” A path-based catalog on S3 can lose commits.

“REST is a specific product.” It is an API specification with many implementations.

“Switching catalogs means migrating data.” It means re-registering tables and updating every name consumers use — no data moves.

“The catalog is a performance component.” It is one lookup per query. It is a correctness component.

A model worth keeping

One pointer per table, swapped atomically. That is the catalog.

Judge an implementation by whether its swap is genuinely atomic on your storage, and by who needs to agree on table identity. The first rules out path-based catalogs on object stores; the second is what pushes teams to REST.

References

• Iceberg catalogs for the catalog contract
• Iceberg REST catalog spec for the API itself
• Iceberg Spark configuration for catalog properties
• A local Iceberg playground for a REST catalog you can run
• Running Iceberg on Apache Polaris for a governed catalog

Trademarks

Apache Iceberg, Apache Spark, Apache Hive, Apache Polaris, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.