Running Iceberg on Apache Polaris: one compose file, two storage backends, and the things that block you

• Architecture: who holds what
• How do you start the stack?
  • Why are there flags just to use a local directory?
• How do you create a catalog?
  • If that command printed nothing at all
• How do you point a catalog at MinIO instead?
• What is the grant chain, and why does Spark fail without it?
  • Do it once, in a script
• Where are you running Spark?
• How do you point Spark at it?
  • If you launch pyspark or spark-shell first
• Can I put the settings in environment variables?
  • Launching the shell with them
  • Forget one catalog’s settings and the error blames something else
• Does the full table lifecycle work?
• Does the same code work on both backends?
• What broke, and what the error actually meant
• Common misconceptions
• Production notes
• Frequently asked questions
• Where this leaves you
• References
• Trademarks

TL;DR

• Polaris is an Iceberg REST catalog plus an authorization model. Spark talks to it with type=rest and never learns where the data lives until the catalog tells it.
• Access needs a four-link chain: principal, principal role, catalog role, privilege. Miss any link and Spark fails with a permission error that names none of them.
• Local FILE storage is refused in production mode. Polaris aborts startup on a severe readiness check, and you must explicitly accept the risk to run it for a test. The MinIO path avoids that entirely and is the one worth learning.
• With an S3-compatible server, endpoint and endpointInternal are different values: the server and the client reach the same bucket by different names.
• The default metastore is in-memory. Restarting the container silently loses every catalog and role, and the next Spark query fails with Unable to find warehouse.
• spark.jars.packages set inside SparkSession.builder does nothing. The JVM is already running by then; the Iceberg runtime has to arrive on the spark-submit line.

A catalog is the part of a lakehouse people postpone. Iceberg works fine against a filesystem path, right up to the day two engines need a consistent view of the same table and something has to arbitrate. That something is a catalog, and Apache Polaris is the one that speaks Iceberg’s REST protocol natively while adding the access control that a shared catalog actually needs.

This post builds the whole thing from nothing: a Polaris server, a catalog, the grants, and a Spark session that creates a partitioned table, merges into it, and travels back to an earlier snapshot. Every command here was run against Polaris 1.7.0 and Spark 4.1.3, and the four places it broke are documented where they happened rather than tidied away.

Architecture: who holds what

The division of labour is the thing to get straight first, because it explains every failure later.

flowchart LR
  S["<b>Spark</b><br/>SparkCatalog, type=rest"]
  P["<b>Polaris</b><br/>Iceberg REST API<br/>+ authorization"]
  M["<b>metastore</b><br/>catalogs, roles, grants,<br/>table pointers"]
  W["<b>warehouse</b><br/>Parquet data<br/>Iceberg metadata"]
  S -->|"1  OAuth2 token"| P
  S -->|"2  load table"| P
  P --> M
  P -->|"3  metadata location"| S
  S -->|"4  read and write files"| W
  P -->|"commits metadata"| W

Three facts follow from that picture.

Polaris holds pointers, not data. It records which metadata file is current for a table. The Parquet lives in the warehouse and Spark reads and writes it directly.

Both processes touch the warehouse. Polaris writes the initial table metadata; Spark writes the data files and later metadata. If only one of them can write to that location, table creation fails in a way that looks like a Spark problem and is not.

Authorization happens at the catalog. Spark presents a token, and Polaris decides what that principal may do. This is the part with four moving pieces.

How do you start the stack?

Three services: MinIO for S3 storage, a one-shot job that creates the bucket, and Polaris. Save this as docker-compose.yml:

# Apache Polaris with two storage backends: a local directory and MinIO.
# Bring it up with:  docker compose up -d
services:
  minio:
    image: minio/minio:latest
    container_name: minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    ports:
      - "9000:9000"   # S3 API
      - "9001:9001"   # web console
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      timeout: 3s
      retries: 20

  # Creates the bucket, then exits. `depends_on` waits for the healthcheck.
  minio-init:
    image: minio/mc:latest
    container_name: minio-init
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      mc alias set local http://minio:9000 minioadmin minioadmin &&
      mc mb --ignore-existing local/warehouse &&
      mc ls local &&
      echo 'bucket ready'"

  polaris:
    image: apache/polaris:latest
    container_name: polaris
    depends_on:
      minio-init:
        condition: service_completed_successfully
    ports:
      - "8181:8181"
      - "8182:8182"
    volumes:
      # The host path and the container path must be IDENTICAL. Polaris writes
      # the first metadata file and Spark writes everything after it, and each
      # resolves file:///tmp/polaris-warehouse in its own filesystem. A relative
      # mount like ./warehouse silently splits one table across two directories.
      - /tmp/polaris-warehouse:/tmp/polaris-warehouse
    environment:
      POLARIS_BOOTSTRAP_CREDENTIALS: "POLARIS,root,s3cr3t"
      quarkus.otel.sdk.disabled: "true"
      # Accepts the risk of FILE storage. Test only, see the post.
      polaris.readiness.ignore-severe-issues: "true"
      # Credentials Polaris uses to reach MinIO
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: minioadmin
      AWS_REGION: us-east-1
      JAVA_OPTS_APPEND: >-
        -Dpolaris.features."ALLOW_INSECURE_STORAGE_TYPES"=true
        -Dpolaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"=["FILE","S3"]
    healthcheck:
      test: ["CMD-SHELL", "exec 3<>/dev/tcp/localhost/8181"]
      interval: 5s
      timeout: 3s
      retries: 30

Create the warehouse directory first, then bring it up:

mkdir -p /tmp/polaris-warehouse && chmod 777 /tmp/polaris-warehouse
docker compose up -d

Verify before going further. The bucket job should have run and exited, and both long-running services should be healthy:

docker compose ps --format '{{.Name}}\t{{.Status}}'

minio     Up 39 seconds (healthy)
polaris   Up 32 seconds (healthy)

docker compose logs minio-init | tail -3

minio-init  | Bucket created successfully `local/warehouse`.
minio-init  | [2026-09-21 06:18:03 UTC]     0B warehouse/
minio-init  | bucket ready

MinIO’s console is on http://localhost:9001 with minioadmin / minioadmin if you would rather watch the objects appear in a browser.

Two details in that file matter more than they look.

The warehouse mount uses the same absolute path on both sides. This is not tidiness. Polaris writes the first metadata file and Spark writes everything after it, and each resolves file:///tmp/polaris-warehouse in its own filesystem. A relative mount like ./warehouse:/tmp/polaris-warehouse gives the container one directory and a host Spark a different one, and the table silently splits in half: Polaris’s metadata in one place, the Parquet and manifests in another. It appears to work, because Spark never re-reads Polaris’s copy inside a single session, and it is broken for anyone who opens the table later.

chmod 777 is a laptop shortcut. The Polaris process runs as uid 10000, which will not match your user, and this is the crude way to let both write. It is one of several reasons the local-file backend is for tests only.

Why are there flags just to use a local directory?

Because Polaris does not want you to. Without them, a catalog backed by file:// is rejected outright:

{"error":{"message":"Unsupported storage type: FILE","type":"IllegalArgumentException","code":400}}

Enabling the feature is not enough either. Polaris then refuses to start:

🛑 Must not enable a configuration that exposes known and severe security risks:
   Allow usage of FileIO implementations that are considered insecure.
   Offending configuration option: 'polaris.features."ALLOW_INSECURE_STORAGE_TYPES"'.
🛑 The storage type 'FILE' is considered insecure and exposes the service to
   severe security risks!
Caused by: java.lang.IllegalStateException: Severe production readiness issues
   detected, startup aborted!

polaris.readiness.ignore-severe-issues=true is the documented escape hatch, and its own source comment is candid about what you are agreeing to:

Setting this to true means that Polaris will start up even if severe security risks have been detected, accepting the risk of denial-of-service, data-loss, corruption and other risks.

Local files are a laptop convenience. Any shared deployment should use S3, GCS or Azure storage configuration, which is also where Polaris earns its keep by vending scoped, short-lived storage credentials to each engine instead of handing out long-lived keys.

How do you create a catalog?

Everything on the management API needs a bearer token. Polaris issues one through the OAuth2 client-credentials flow:

TOKEN=$(curl -s -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \
  -d 'grant_type=client_credentials' \
  -d 'client_id=root' -d 'client_secret=s3cr3t' \
  -d 'scope=PRINCIPAL_ROLE:ALL' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')

# always check it, because every later call fails silently without it
echo "token length: ${#TOKEN}"     # a real token is several hundred characters

The token is valid for one hour: the response carries "expires_in": 3600. Re-run this when you come back to a shell you left open, and remember it lives in one shell only, so a second terminal has no $TOKEN at all.

Then the catalog itself. default-base-location is where tables land, and allowedLocations is the boundary Polaris enforces on them:

curl -s -X POST http://localhost:8181/api/management/v1/catalogs \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -w '\nHTTP %{http_code}\n' \
  -d '{"catalog":{
        "name":"local_catalog","type":"INTERNAL","readOnly":false,
        "properties":{"default-base-location":"file:///tmp/polaris-warehouse"},
        "storageConfigInfo":{"storageType":"FILE",
                             "allowedLocations":["file:///tmp/polaris-warehouse"]}}}'

That -w '\nHTTP %{http_code}\n' is not decoration. Keep it on every call in this section, for the reason in the next paragraph. A successful create answers HTTP 201 with the catalog as its body:

{"type":"INTERNAL","name":"local_catalog","properties":{"default-base-location":
"file:///tmp/polaris-warehouse"},"createTimestamp":1789971062932,"lastUpdateTimestamp":0,
"entityVersion":1,"storageConfigInfo":{"storageType":"FILE","allowedLocations":
["file:///tmp/polaris-warehouse"]}}
HTTP 201

If that command printed nothing at all

Not an empty result, literally no output. There are three things this call can answer, and only one of them is silent:

The last one catches people making a second catalog: Polaris will not let two catalogs claim overlapping storage, so give each its own directory.

The 401 returns an empty body, so curl -s prints nothing whatsoever and looks like it did nothing. $TOKEN is empty more often than you would think: the variable lives in one shell, it expires after an hour, and a failed token fetch leaves it set to the empty string rather than failing loudly, because .get("access_token", "") on an error response returns nothing:

echo "token length: ${#TOKEN}"
# 0  -> refetch the token; anything else -> the token is not your problem

A wrong client secret is the usual cause, and it does report itself if you look at the token call rather than the catalog call:

{"error":"unauthorized_client","error_description":"The client is not authorized"}

How do you point a catalog at MinIO instead?

The local backend is the shortest path to a working table. Object storage is what you would actually deploy, and MinIO gives you the same S3 API on your laptop. The catalog is the only thing that changes:

curl -s -X POST http://localhost:8181/api/management/v1/catalogs \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -w '\nHTTP %{http_code}\n' \
  -d '{"catalog":{
        "name":"s3_catalog","type":"INTERNAL","readOnly":false,
        "properties":{"default-base-location":"s3://warehouse/polaris"},
        "storageConfigInfo":{
          "storageType":"S3",
          "allowedLocations":["s3://warehouse/polaris"],
          "roleArn":"arn:aws:iam::000000000000:role/minio-unused",
          "region":"us-east-1",
          "endpoint":"http://localhost:9000",
          "endpointInternal":"http://minio:9000",
          "pathStyleAccess":true,
          "stsUnavailable":true}}}'

Four of those fields exist for exactly this situation.

endpoint and endpointInternal are the same split as before. Polaris reaches MinIO at http://minio:9000, its service name on the compose network. Spark, on your host, reaches the identical bucket at http://localhost:9000. endpointInternal is what the server uses; endpoint is what Polaris hands to clients. The spec is explicit that “Iceberg REST API clients never see this value” for endpointInternal. Set only one and whichever side is not on that network fails to connect.

pathStyleAccess must be true. Real S3 addresses buckets as warehouse.s3.amazonaws.com; MinIO expects localhost:9000/warehouse.

stsUnavailable must be true. Polaris normally vends short-lived scoped credentials by calling STS, which is the main security reason to run it. MinIO has no STS endpoint here, so this tells Polaris not to try. The consequence is real and worth stating: with STS unavailable, Polaris is not vending credentials, so the engine needs its own. That is fine on a laptop and is not what you want in production.

roleArn is required by the schema and unused here. With stsUnavailable set, nothing assumes that role, so any syntactically valid ARN is accepted. It is a wart of reusing the AWS storage type for an S3-compatible server.

A successful create echoes the config back with HTTP 201, including the two endpoints:

{"type":"INTERNAL","name":"s3_catalog","properties":{"default-base-location":"s3://warehouse/polaris"},
...,"entityVersion":1,"storageConfigInfo":{"roleArn":"arn:aws:iam::000000000000:role/minio-unused",
"region":"us-east-1","endpoint":"http://localhost:9000","stsUnavailable":true,
"endpointInternal":"http://minio:9000","pathStyleAccess":true,"storageType":"S3",
"allowedLocations":["s3://warehouse/polaris"]}}
HTTP 201

What is the grant chain, and why does Spark fail without it?

This is the step that costs people an afternoon. Creating a catalog grants nobody access to it, including the principal that created it. Access is a chain of four links, and Spark’s error message names none of them.

flowchart LR
  A["<b>principal</b><br/>root"] -->|"is assigned"| B["<b>principal role</b><br/>engineer"]
  B -->|"is granted"| C["<b>catalog role</b><br/>admin_role"]
  C -->|"holds privilege"| D["<b>CATALOG_MANAGE_CONTENT</b><br/>on local_catalog"]

Read it right to left: a privilege lives on a catalog role, a catalog role is granted to a principal role, and a principal role is assigned to a principal. Four calls, each returning 201:

API=http://localhost:8181/api/management/v1
auth=(-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json')

# 1. a catalog role, scoped to this catalog
curl -s -X POST "$API/catalogs/local_catalog/catalog-roles" "${auth[@]}" \
  -d '{"catalogRole":{"name":"admin_role"}}'

# 2. give it a privilege on the catalog
curl -s -X PUT "$API/catalogs/local_catalog/catalog-roles/admin_role/grants" "${auth[@]}" \
  -d '{"grant":{"type":"catalog","privilege":"CATALOG_MANAGE_CONTENT"}}'

# 3. a principal role, which is catalog-independent
curl -s -X POST "$API/principal-roles" "${auth[@]}" \
  -d '{"principalRole":{"name":"engineer"}}'

# 4. connect the two roles, then the role to the principal
curl -s -X PUT "$API/principal-roles/engineer/catalog-roles/local_catalog" "${auth[@]}" \
  -d '{"catalogRole":{"name":"admin_role"}}'
curl -s -X PUT "$API/principals/root/principal-roles" "${auth[@]}" \
  -d '{"principalRole":{"name":"engineer"}}'

The separation looks like ceremony at this scale and is the point at a larger one. Principal roles describe people and jobs; catalog roles describe what may be done to a specific catalog. One engineer principal role can be granted different catalog roles in different catalogs, so a team can be an admin of its own data and a reader of somebody else’s without a new identity.

Verify the chain landed:

curl -s "${auth[@]}" "$API/principals/root/principal-roles"
# {"roles":[{"name":"service_admin",...},{"name":"engineer",...}]}

Do it once, in a script

You will rebuild this more often than you expect, because the default metastore is in-memory. Keep the whole provisioning step re-runnable:

#!/usr/bin/env bash
set -euo pipefail
BASE=${BASE:-http://localhost:8181}

TOKEN=$(curl -s -X POST "$BASE/api/catalog/v1/oauth/tokens" \
  -d 'grant_type=client_credentials' -d 'client_id=root' -d 'client_secret=s3cr3t' \
  -d 'scope=PRINCIPAL_ROLE:ALL' | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')
[ ${#TOKEN} -gt 100 ] || { echo "token fetch failed"; exit 1; }

API="$BASE/api/management/v1"
post() { curl -s -o /dev/null -w "%{http_code}" -X "$1" "$API/$2" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d "$3"; }

# the principal role first, so the later assignments have something to bind to
echo "principal role    : $(post POST principal-roles '{"principalRole":{"name":"engineer"}}')"
echo "local_catalog     : $(post POST catalogs '{"catalog":{"name":"local_catalog","type":"INTERNAL","readOnly":false,"properties":{"default-base-location":"file:///tmp/polaris-warehouse"},"storageConfigInfo":{"storageType":"FILE","allowedLocations":["file:///tmp/polaris-warehouse"]}}}')"
echo "s3_catalog        : $(post POST catalogs '{"catalog":{"name":"s3_catalog","type":"INTERNAL","readOnly":false,"properties":{"default-base-location":"s3://warehouse/polaris"},"storageConfigInfo":{"storageType":"S3","allowedLocations":["s3://warehouse/polaris"],"roleArn":"arn:aws:iam::000000000000:role/minio-unused","region":"us-east-1","endpoint":"http://localhost:9000","endpointInternal":"http://minio:9000","pathStyleAccess":true,"stsUnavailable":true}}}')"

for CAT in local_catalog s3_catalog; do
  echo "$CAT role   : $(post POST "catalogs/$CAT/catalog-roles" '{"catalogRole":{"name":"admin_role"}}')"
  echo "$CAT grant  : $(post PUT "catalogs/$CAT/catalog-roles/admin_role/grants" '{"grant":{"type":"catalog","privilege":"CATALOG_MANAGE_CONTENT"}}')"
  echo "$CAT assign : $(post PUT "principal-roles/engineer/catalog-roles/$CAT" '{"catalogRole":{"name":"admin_role"}}')"
done
echo "root -> engineer  : $(post PUT principals/root/principal-roles '{"principalRole":{"name":"engineer"}}')"

Every line should print 201:

principal role    : 201
local_catalog     : 201
s3_catalog        : 201
local_catalog role   : 201
local_catalog grant  : 201
local_catalog assign : 201
s3_catalog role   : 201
s3_catalog grant  : 201
s3_catalog assign : 201
root -> engineer  : 201

Note the ordering: the principal role is created first. Assigning a catalog role to a principal role that does not exist yet returns 404, and because the call targets the catalog, the 404 reads as though the catalog is missing.

Where are you running Spark?

This decides two settings, and getting either wrong produces an error that blames something else. There are two sensible setups.

The catalog URI is resolved by Spark, not by Polaris. A container name like polaris only resolves inside a Docker network. Run pyspark on your laptop against that hostname and the OAuth call fails before anything else happens:

org.apache.iceberg.exceptions.RESTException: Error occurred while processing POST request
Caused by: java.net.UnknownHostException: polaris: nodename nor servname provided, or not known

From the host, use localhost. The stack trace points at OAuth2Util.fetchToken because the token exchange is the first call the REST client makes, which makes this look like an authentication problem. It is DNS.

With FILE storage, both processes must see the same directory. Polaris writes the first metadata file and Spark writes everything after it, and each resolves file:///tmp/polaris-warehouse in its own filesystem. If Polaris only has that path inside its container and Spark is on the host, they are two different directories and the commit fails:

org.apache.iceberg.exceptions.CommitStateUnknownException: Service failed: 503:
Failed to create file: file:/tmp/polaris-warehouse/sales/orders/metadata/00000-....metadata.json
Cannot determine whether the commit was successful or not

Bind-mounting the host directory at the same path inside the container makes one directory serve both, which is what the docker run above does. The Polaris process runs as uid 10000, so the directory has to be writable by it:

mkdir -p /tmp/polaris-warehouse && chmod 777 /tmp/polaris-warehouse
docker exec polaris sh -c 'touch /tmp/polaris-warehouse/.probe && echo OK'
# OK

That chmod 777 is a laptop shortcut, and another reason FILE storage is for tests only. With S3 there is no shared-filesystem problem at all, because both sides address the same bucket.

How do you point Spark at it?

Spark needs the Iceberg runtime, and this is the fourth trap. Setting spark.jars.packages inside SparkSession.builder does nothing useful, because by the time that code runs the JVM has already started and its classpath is fixed. The symptom is a class that plainly exists being missing:

org.apache.spark.SparkException: Cannot find catalog plugin class for catalog
'polaris': org.apache.iceberg.spark.SparkCatalog
Caused by: java.lang.ClassNotFoundException: org.apache.iceberg.spark.SparkCatalog

The runtime has to arrive on the submit line instead:

spark-submit \
  --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0 \
  your_job.py

The artifact name encodes the Spark major line and the Scala binary version, and it must match your Spark exactly:

There is no iceberg-spark-runtime-4.1_2.13: the 4.0 build serves the whole 4.x line. Spark 4.x is Scala 2.13 only, while 3.5 is usually 2.12, and picking the wrong Scala suffix surfaces as a missing class rather than a version error. Everything in this post was run on both combinations.

With the jar on the classpath, the catalog is seven configuration lines:

Both catalogs share the same eight REST settings and differ only in the warehouse name and, for S3, the file IO. A small helper keeps that honest:

from pyspark.sql import SparkSession

def cat(b, name, warehouse, extra=None):
    """Attach one Polaris catalog to the builder."""
    p = f"spark.sql.catalog.{name}"
    b = (b.config(p, "org.apache.iceberg.spark.SparkCatalog")
          .config(f"{p}.type", "rest")
          .config(f"{p}.uri", "http://localhost:8181/api/catalog")
          .config(f"{p}.warehouse", warehouse)
          .config(f"{p}.credential", "root:s3cr3t")
          .config(f"{p}.scope", "PRINCIPAL_ROLE:ALL")
          .config(f"{p}.rest.auth.type", "oauth2")
          .config(f"{p}.oauth2-server-uri",
                  "http://localhost:8181/api/catalog/v1/oauth/tokens"))
    for k, v in (extra or {}).items():
        b = b.config(f"{p}.{k}", v)
    return b

b = (SparkSession.builder.appName("polaris-demo").master("local[2]")
     .config("spark.sql.extensions",
             "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"))

b = cat(b, "local_cat", "local_catalog")
b = cat(b, "s3_cat", "s3_catalog", {
    "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
    "s3.endpoint": "http://localhost:9000",
    "s3.path-style-access": "true",
    "client.region": "us-east-1",
})
spark = b.getOrCreate()

spark.sql("CREATE NAMESPACE IF NOT EXISTS local_cat.sales")
print(spark.sql("SHOW NAMESPACES IN local_cat").collect())
# [Row(namespace='sales')]

The S3 catalog needs three things the local one does not: S3FileIO as the IO implementation, the endpoint and path-style flag pointing at MinIO, and AWS credentials in the environment, because stsUnavailable means Polaris is not vending any:

export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_REGION=us-east-1

spark-submit \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0,org.apache.iceberg:iceberg-aws-bundle:1.11.0 \
  your_job.py

iceberg-aws-bundle is the second jar, and leaving it out is the usual reason S3FileIO cannot be found.

Two of those deserve a note. warehouse is not a path here, it is the catalog name: the REST client sends it to /v1/config and Polaris answers with the real location and a prefix to use for every later call.

curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8181/api/catalog/v1/config?warehouse=local_catalog"

{"defaults":{"default-base-location":"file:///tmp/polaris-warehouse"},
 "overrides":{"namespace-separator":"%1F","prefix":"local_catalog"},
 "endpoints":["GET /v1/{prefix}/namespaces", ...]}

And credential is client-id:client-secret, which the Iceberg REST client exchanges for a token itself. You do not pass the token you fetched earlier.

The last two settings suppress warnings that are really deprecation notices:

WARN AuthManagers: Inferring rest.auth.type=oauth2 since property credential was
provided. Please explicitly set rest.auth.type to avoid this warning.
WARN OAuth2Manager: Iceberg REST client is missing the OAuth2 server URI
configuration and defaults to .../v1/oauth/tokens. This automatic fallback will
be removed in a future Iceberg release.

Both describe behaviour the Iceberg client infers today and intends to stop inferring. Setting rest.auth.type and oauth2-server-uri explicitly costs two lines now and avoids a breakage on a future upgrade.

If you launch pyspark or spark-shell first

A session already exists by the time your code runs, so getOrCreate() returns that one and tells you what it ignored:

WARN SparkSession: Using an existing Spark session; only runtime SQL
configurations will take effect.

Catalog properties are runtime SQL configs and do apply. spark.sql.extensions is not: extensions are installed when the session is built, so setting it afterwards has no effect even though reading the property back shows your value. Pass the configuration on the launch line instead:

pyspark \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.local_cat=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.local_cat.type=rest \
  --conf spark.sql.catalog.local_cat.uri=http://localhost:8181/api/catalog \
  --conf spark.sql.catalog.local_cat.warehouse=local_catalog \
  --conf spark.sql.catalog.local_cat.credential=root:s3cr3t \
  --conf spark.sql.catalog.local_cat.scope=PRINCIPAL_ROLE:ALL \
  --conf spark.sql.catalog.local_cat.rest.auth.type=oauth2 \
  --conf spark.sql.catalog.local_cat.oauth2-server-uri=http://localhost:8181/api/catalog/v1/oauth/tokens

Writing a script and running it with spark-submit avoids the problem entirely, because the session is built once with your configuration.

Can I put the settings in environment variables?

The launch line is long and most of it is the same for every catalog, so it is worth keeping in one sourceable file. Save this as polaris-env.sh:

# Source this, do not execute it:  source polaris-env.sh

export POLARIS_URI='http://localhost:8181/api/catalog'
export POLARIS_SCOPE='PRINCIPAL_ROLE:ALL'
export CLIENT_ID='root'
export CLIENT_SECRET='s3cr3t'
export POLARIS_CREDENTIAL="${CLIENT_ID}:${CLIENT_SECRET}"

# Versions in one place, so the two jars can never drift apart
export SPARK_VERSION='3.5'
export SCALA_VERSION='2.12'
export ICEBERG_VERSION='1.11.0'
export ICEBERG_PACKAGES="org.apache.iceberg:iceberg-spark-runtime-${SPARK_VERSION}_${SCALA_VERSION}:${ICEBERG_VERSION},org.apache.iceberg:iceberg-aws-bundle:${ICEBERG_VERSION}"

# MinIO. Polaris has stsUnavailable set, so it vends nothing and the engine
# supplies its own credentials.
export AWS_ACCESS_KEY_ID='minioadmin'
export AWS_SECRET_ACCESS_KEY='minioadmin'
export AWS_REGION='us-east-1'
export MINIO_ENDPOINT='http://localhost:9000'

# A management-API token, valid for one hour. Spark does not use this; it has
# POLARIS_CREDENTIAL and does its own OAuth2 exchange.
export POLARIS_TOKEN=$(curl -s -X POST "${POLARIS_URI}/v1/oauth/tokens" \
  -d 'grant_type=client_credentials' \
  -d "client_id=${CLIENT_ID}" -d "client_secret=${CLIENT_SECRET}" \
  -d "scope=${POLARIS_SCOPE}" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')

if [ ${#POLARIS_TOKEN} -lt 100 ]; then
  echo "POLARIS_TOKEN looks wrong (length ${#POLARIS_TOKEN}). Is the stack up?" >&2
else
  echo "POLARIS_TOKEN length ${#POLARIS_TOKEN}"
fi

No spaces around the =. export POLARIS_SCOPE = 'PRINCIPAL_ROLE:ALL' is not an assignment. Bash reports export: '=': not a valid identifier and carries on with the variable unset, and zsh fails harder with zsh: bad assignment. Either way $POLARIS_SCOPE ends up empty, and you are back to the silent 401 above.

source polaris-env.sh
# POLARIS_TOKEN length 630

Launching the shell with them

Both catalogs, Spark 3.5 and the matching Iceberg build, with nothing hardcoded:

pyspark \
  --packages "${ICEBERG_PACKAGES}" \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  \
  --conf spark.sql.catalog.local_cat=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.local_cat.type=rest \
  --conf spark.sql.catalog.local_cat.uri="${POLARIS_URI}" \
  --conf spark.sql.catalog.local_cat.warehouse=local_catalog \
  --conf spark.sql.catalog.local_cat.credential="${POLARIS_CREDENTIAL}" \
  --conf spark.sql.catalog.local_cat.scope="${POLARIS_SCOPE}" \
  --conf spark.sql.catalog.local_cat.rest.auth.type=oauth2 \
  --conf spark.sql.catalog.local_cat.oauth2-server-uri="${POLARIS_URI}/v1/oauth/tokens" \
  \
  --conf spark.sql.catalog.s3_cat=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.s3_cat.type=rest \
  --conf spark.sql.catalog.s3_cat.uri="${POLARIS_URI}" \
  --conf spark.sql.catalog.s3_cat.warehouse=s3_catalog \
  --conf spark.sql.catalog.s3_cat.credential="${POLARIS_CREDENTIAL}" \
  --conf spark.sql.catalog.s3_cat.scope="${POLARIS_SCOPE}" \
  --conf spark.sql.catalog.s3_cat.rest.auth.type=oauth2 \
  --conf spark.sql.catalog.s3_cat.oauth2-server-uri="${POLARIS_URI}/v1/oauth/tokens" \
  --conf spark.sql.catalog.s3_cat.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
  --conf spark.sql.catalog.s3_cat.s3.endpoint="${MINIO_ENDPOINT}" \
  --conf spark.sql.catalog.s3_cat.s3.path-style-access=true \
  --conf spark.sql.catalog.s3_cat.client.region="${AWS_REGION}"

The same arguments work unchanged with spark-submit yourjob.py, because both commands hand them to the same launcher. Inside the shell:

spark.sql("CREATE NAMESPACE IF NOT EXISTS local_cat.demo")
spark.sql("CREATE TABLE local_cat.demo.t (id BIGINT, city STRING) USING iceberg")
spark.sql("INSERT INTO local_cat.demo.t VALUES (1,'pune'),(2,'hyderabad')")
spark.sql("SELECT file_path FROM local_cat.demo.t.files LIMIT 1").show(truncate=False)

Run against both, the only difference is where the bytes land:

>>> [SHELL] local rows: 2
>>> [SHELL] local loc : file:/tmp/polaris-warehouse/demo/t/data/00000-0-674d0d02-....parquet
>>> [SHELL] s3 rows   : 2
>>> [SHELL] s3 loc    : s3://warehouse/polaris/demo/t/data/00000-5-d9845c91-....parquet

Forget one catalog’s settings and the error blames something else

If a catalog name has no spark.sql.catalog.<name> configuration, Spark does not say so. It falls back to the built-in session catalog and complains about the shape of the name:

AnalysisException: [REQUIRES_SINGLE_PART_NAMESPACE] spark_catalog requires a
single-part namespace, but got `s3_cat`.`demo`.`t`.

On Spark 3.5 the same mistake wears a second, much worse face. A CREATE NAMESPACE against an unconfigured catalog hits a missing error-message template and reports an internal error instead of the real problem:

SparkException: [INTERNAL_ERROR] Undefined error message parameter for error
class: '_LEGACY_ERROR_TEMP_1055'. Parameters: Map(database -> local_cat.demo)

There is nothing internally wrong with Spark. The analyzer decided local_cat was not a catalog, tried to report “invalid database name local_cat.demo”, and found no text for that error class. Both messages mean the same thing: that catalog was never configured.

The usual cause is a name mismatch rather than a missing line. Configure spark.sql.catalog.polaris and then query local_cat.demo and you get exactly this, because the alias in your --conf flags and the alias in your SQL have to be the same word. In this post that word is local_cat, and the Polaris-side catalog it points at is local_catalog. They are deliberately different so it is obvious which is which:

spark_catalog in the first message is the tell. It means the first part of your identifier was not recognised as a catalog, so it was read as a database name instead. Check that every --conf for that catalog made it onto the line, which is exactly why they belong in a file rather than in your shell history.

Does the full table lifecycle work?

This is the part worth running before trusting any of it. A partitioned table, an insert, metadata queries, schema evolution, a MERGE, and time travel:

spark.sql("""
  CREATE OR REPLACE TABLE local_cat.sales.orders (
    order_id BIGINT, city STRING, amount DECIMAL(10,2), ordered_at TIMESTAMP
  ) USING iceberg PARTITIONED BY (days(ordered_at))""")

spark.sql("""
  INSERT INTO local_cat.sales.orders VALUES
    (1, 'pune',      1200.50, TIMESTAMP '2026-09-01 10:15:00'),
    (2, 'hyderabad',  845.00, TIMESTAMP '2026-09-01 11:00:00'),
    (3, 'pune',       310.25, TIMESTAMP '2026-09-02 09:30:00')""")

spark.sql("SELECT city, sum(amount) AS total FROM local_cat.sales.orders GROUP BY city ORDER BY city").show()

+---------+-------+
|     city|  total|
+---------+-------+
|hyderabad| 845.00|
|     pune|1510.75|
+---------+-------+

The Iceberg metadata tables work through Polaris exactly as they do against any other catalog, which is the clearest evidence the integration is real:

spark.sql("SELECT partition, record_count FROM local_cat.sales.orders.partitions ORDER BY partition").show()

+-----------------+------------+
|        partition|record_count|
+-----------------+------------+
|{2026-09-01}     |           2|
|{2026-09-02}     |           1|
+-----------------+------------+

Three rows landed in two day-partitions, and .files reports two data files, one per partition. Schema evolution and a row-level merge:

spark.sql("ALTER TABLE local_cat.sales.orders ADD COLUMN channel STRING")
spark.sql("""
  MERGE INTO local_cat.sales.orders t
  USING (SELECT 2 AS order_id, 'web' AS channel) s
  ON t.order_id = s.order_id
  WHEN MATCHED THEN UPDATE SET t.channel = s.channel""")
spark.sql("SELECT order_id, city, channel FROM local_cat.sales.orders ORDER BY order_id").show()

+--------+---------+-------+
|order_id|     city|channel|
+--------+---------+-------+
|       1|     pune|   NULL|
|       2|hyderabad|    web|
|       3|     pune|   NULL|
+--------+---------+-------+

And time travel, reading the table as it was at its first snapshot:

first = spark.sql("SELECT snapshot_id FROM local_cat.sales.orders.snapshots ORDER BY committed_at").collect()[0].snapshot_id
spark.sql(f"SELECT count(*) AS rows_at_first_snapshot FROM local_cat.sales.orders VERSION AS OF {first}").show()
# rows_at_first_snapshot = 3

On disk, the layout is ordinary Iceberg. Polaris changed who decides which metadata file is current, and nothing about the format:

WAREHOUSE/sales/orders/data/ordered_at_day=2026-09-01/00000-21-....parquet
WAREHOUSE/sales/orders/data/ordered_at_day=2026-09-02/00000-3-....parquet
WAREHOUSE/sales/orders/metadata/00000-b7a47371-....metadata.json
WAREHOUSE/sales/orders/metadata/00003-66d1fd6b-....metadata.json
WAREHOUSE/sales/orders/metadata/644aa163-...-m0.avro
WAREHOUSE/sales/orders/metadata/snap-6200640515757339866-1-....avro

Does the same code work on both backends?

That is the payoff of configuring two catalogs in one session: the only thing that changes between them is the catalog name in the SQL. Running the same create, insert, schema evolution, merge and time travel against each:

[local_cat] rows=[(1, 'pune', None), (2, 'hyderabad', 'web'), (3, 'pune', None)]
[local_cat] snapshots=2 time_travel_rows=3
[local_cat] location=file:/tmp/polaris-warehouse/sales/orders/data/ordered_at_day=2026-09-01/00000-8-....parquet

[s3_cat]    rows=[(1, 'pune', None), (2, 'hyderabad', 'web'), (3, 'pune', None)]
[s3_cat]    snapshots=2 time_travel_rows=3
[s3_cat]    location=s3://warehouse/polaris/sales/orders/data/ordered_at_day=2026-09-01/00000-26-....parquet

Identical results, different location. Confirm it on disk and in the bucket:

find /tmp/polaris-warehouse -type f | grep -v crc | sort

WAREHOUSE/sales/orders/data/ordered_at_day=2026-09-01/00000-8-....parquet
WAREHOUSE/sales/orders/data/ordered_at_day=2026-09-02/00000-2-....parquet
WAREHOUSE/sales/orders/metadata/00000-d52326b4-....metadata.json
WAREHOUSE/sales/orders/metadata/00001-ef55a0a4-....metadata.json
WAREHOUSE/sales/orders/metadata/00002-e1162218-....metadata.json

That 00000 file is the one Polaris wrote and the rest are Spark’s, all in one directory, which is the check that the mount is right. For MinIO, either browse http://localhost:9001 or list it:

docker run --rm --network compose_default --entrypoint /bin/sh minio/mc:latest -c \
  "mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \
   mc ls -r local/warehouse/"

Seeing Polaris’s 00000 metadata object next to Spark’s Parquet in the same bucket is what proves endpointInternal and endpoint both resolved to it, from two processes that address it by different hostnames.

What broke, and what the error actually meant

Four failures, in the order they happened. None of their messages named the real cause.

The third is worth expanding because the message points at Spark and the fault is elsewhere. Polaris writes the first metadata file itself, server-side. A Docker named volume is created owned by root, and the Polaris process is uid 10000, so it cannot write there:

docker exec polaris id
# uid=10000(polaris) gid=10001(polaris)
docker exec polaris ls -ld /tmp/polaris-warehouse
# drwxr-xr-x 2 root root

Both processes need write access to the same location:

docker run --rm -v polariswh:/wh alpine:latest \
  sh -c 'chown -R 10000:10001 /wh && chmod -R 775 /wh'

The fourth is the one that will catch you twice. The startup log says it plainly and it is easy to read past:

⚠️ The current metastore is intended for tests only.
   Offending configuration option: 'polaris.persistence.type'.

Catalogs, roles and grants live in memory. docker restart polaris keeps the realm bootstrap, because that is re-run at startup, and loses everything you created through the management API. The next Spark query fails with Unable to find warehouse, which sounds like a client misconfiguration and is not. Keep the provisioning calls in a script you can re-run, and configure PostgreSQL persistence for anything you intend to keep.

Common misconceptions

“The warehouse config is a path.” It is the catalog name. The path comes back from Polaris in the /v1/config response, which is what lets the catalog relocate storage without a client change.

“Creating a catalog gives its creator access.” It does not. The principal that created local_catalog could not read it until the four-link grant chain was built, and the failure arrives as a permission error that mentions no role.

“Polaris stores my data.” It stores pointers and permissions. Spark reads and writes Parquet directly, which is why the warehouse must be reachable and writable from the engine, not only from Polaris.

”--packages and spark.jars.packages are interchangeable.” Only before the JVM starts. In SparkSession.builder, the second one is ignored and you get a ClassNotFoundException for a class that is genuinely not on the classpath.

“A REST catalog means I can skip the Iceberg runtime jar.” The REST protocol is spoken by the Iceberg client library. Without iceberg-spark-runtime there is no SparkCatalog to configure.

Production notes

• Do not use FILE storage. It exists for tests, Polaris tries to stop you, and the flag you need is called ignore-severe-issues for a reason.
• Configure a real metastore. The default is in-memory and loses every catalog and grant on restart.
• Keep provisioning in a re-runnable script. You will rebuild the catalog more often than you expect, and the grant chain is four calls that are easy to half-apply.
• Give the storage location to both parties. Polaris commits the first metadata file; the engine writes everything else, so with FILE storage the path must resolve to one directory for both, and with object storage it must be reachable by both.
• Prefer credential vending over shared keys. With S3 or GCS, Polaris hands each engine scoped, short-lived credentials, which is the main reason to run it rather than a filesystem catalog.
• Pin the Iceberg runtime version. The artifact encodes the Spark line and Scala version, and a mismatch surfaces as a missing class rather than a version error.

Frequently asked questions

Which Iceberg runtime works with my Spark? iceberg-spark-runtime-4.0_2.13:1.11.0 for Spark 4.x, and iceberg-spark-runtime-3.5_2.12:1.11.0 for Spark 3.5.x. The number in the artifact is the Spark major line, not the Iceberg version, and there is no 4.1 build because 4.0 serves the whole 4.x line.

A curl against the management API printed nothing. Is the server down? Almost certainly not. A 401 from that API has an empty body, so curl -s shows nothing at all. Check echo ${#TOKEN} first: tokens expire after an hour, live in a single shell, and a failed fetch silently leaves the variable empty. Add -w '\nHTTP %{http_code}\n' to every call so a status is always visible.

I am running pyspark on my laptop and Polaris in Docker. What changes? Two things. The catalog uri must be http://localhost:8181/api/catalog, because a container name does not resolve on the host. And with FILE storage the warehouse directory has to be bind-mounted into the Polaris container at the same path, so both processes write to one directory.

Do I pass the bearer token I fetched with curl to Spark? No. Give Spark credential as client-id:client-secret and the Iceberg REST client performs its own OAuth2 exchange. The curl token is for the management API, which Spark never calls.

Why does my first table creation fail with a 503? Most often because Polaris cannot write to the warehouse location. It creates the initial metadata file server-side, so the location must be writable by the Polaris process, not only by Spark.

What does CATALOG_MANAGE_CONTENT actually allow? Content management across the catalog: namespaces and tables, and reads and writes of them. It is the broad privilege that makes a quickstart work. Narrower privileges exist and are what you would use for a reader role.

Can two engines share one Polaris catalog? That is the reason it exists. Any Iceberg REST client can attach, and the catalog arbitrates commits, so Spark and another engine see the same table state rather than two filesystem views that drift.

Where this leaves you

A catalog is worth adding at the point where a second reader appears, and the work splits cleanly in two. The Iceberg half is what you already know: the same table format, the same metadata files, the same snapshots and time travel, all untouched. The Polaris half is the part that is new, and it is almost entirely about identity: who is asking, which role they hold, and what that role may do to this catalog.

If you take one habit from this post, make it the grant chain. Every access failure in Polaris is one of those four links missing, and checking them in order is faster than reading the error, which will not tell you which one.

References

• Apache Polaris documentation, for the management API and the production configuration guide referenced by the readiness checks
• Iceberg Spark configuration for the catalog properties used above
• Iceberg REST catalog spec, the protocol Polaris implements
• Apache Iceberg architecture for what a commit actually swaps, which is what the catalog arbitrates
• Apache Iceberg on Spark: the complete cheat sheet for the table surface once the catalog is wired up

Trademarks

Apache Polaris, Apache Iceberg, Apache Spark, Apache Parquet, Apache Avro and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.