A local Iceberg playground: Spark, MinIO and a REST catalog in one compose file

• Why three services?
• The compose file
• Pointing Spark at it
• Creating the first table
• What landed in object storage
  • The naming difference that makes tutorials disagree
• Checking it from the other side
• Adding a notebook
• When this setup is not enough
• Cleaning up
• References
• Trademarks

TL;DR

• Three services, not one: MinIO for S3-compatible storage, an Iceberg REST catalog for the metadata pointer, and Spark as the engine. Each needs the other two configured correctly or nothing works.
• The MinIO images are on quay.io, not Docker Hub. minio/minio:latest does not resolve; quay.io/minio/minio:latest does.
• Spark needs two Iceberg jars: iceberg-spark-runtime for the engine integration and iceberg-aws-bundle for S3FileIO. One without the other fails at the first write.
• A 5,000-row table produced exactly 5 objects: one Parquet data file, two metadata.json, one manifest and one manifest list.
• Under a REST catalog the metadata files are named 00000-<uuid>.metadata.json. The v1.metadata.json plus version-hint.text pattern belongs to the Hadoop catalog, and confusing the two makes tutorials contradict each other.

Why three services?

A single-node Spark shell with a local warehouse directory is enough to try Iceberg’s SQL surface, and it hides the thing that matters in production: an Iceberg table is a contract between storage, a catalog and an engine, and most real problems live where those three meet.

Storage holds the data files and the metadata files. In production that is S3, GCS or ADLS; MinIO speaks the S3 API, so the configuration you write locally is the configuration you write in the cloud.

The catalog answers one question: for this table name, which metadata file is current? That is the whole job, and it is the reason two writers cannot silently overwrite each other. A local directory cannot do this safely, which is why the Hadoop catalog is fine for experiments and wrong for production.

The engine reads the catalog, reads the metadata, then reads the data.

flowchart LR
    S[Spark] -->|"1. which metadata file is current?"| R[REST catalog]
    R -->|"2. s3://warehouse/db/trips/metadata/00001-....json"| S
    S -->|"3. read metadata, manifests, data"| M[(MinIO<br/>S3 API)]
    R -.->|"commits update the pointer"| R

Running all three locally means the mistakes you make are the same mistakes you would make against real object storage, which is the point of a playground.

The compose file

services:
  minio:
    image: quay.io/minio/minio:latest
    container_name: minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password
    ports: ["9000:9000", "9001:9001"]
    networks: [lake]

  mc:
    image: quay.io/minio/mc:latest
    container_name: mc
    depends_on: [minio]
    networks: [lake]
    entrypoint: >
      /bin/sh -c "
      until mc alias set m http://minio:9000 admin password 2>/dev/null; do sleep 1; done;
      mc mb --ignore-existing m/warehouse;
      echo BUCKET_READY;
      tail -f /dev/null"

  rest:
    image: apache/iceberg-rest-fixture:latest
    container_name: rest
    depends_on: [minio]
    ports: ["8181:8181"]
    networks: [lake]
    environment:
      CATALOG_WAREHOUSE: s3://warehouse/
      CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
      CATALOG_S3_ENDPOINT: http://minio:9000
      CATALOG_S3_PATH__STYLE__ACCESS: "true"
      AWS_ACCESS_KEY_ID: admin
      AWS_SECRET_ACCESS_KEY: password
      AWS_REGION: us-east-1

networks:
  lake:
    name: lake

docker compose up -d

Three details in there are load-bearing.

The images are on quay.io. minio/minio:latest on Docker Hub does not resolve; MinIO publishes to quay.io, and tutorials that predate the change fail at docker compose up with a pull error.

The REST catalog’s environment variables use a naming convention. Everything after CATALOG_ becomes an Iceberg catalog property, with __ standing in for a dot and _ for a dash. So CATALOG_S3_PATH__STYLE__ACCESS sets s3.path-style-access. Getting the underscores wrong produces a catalog that starts cleanly and fails on first use.

Path-style access is required. The default S3 addressing puts the bucket in the hostname (warehouse.minio:9000), which does not resolve on a compose network. Path-style keeps it in the path (minio:9000/warehouse).

Wait for the bucket, which the mc container creates once MinIO is accepting connections:

docker compose logs mc | grep BUCKET_READY

Pointing Spark at it

docker run --rm --network lake -v "$PWD":/work -w /work \
  -e AWS_ACCESS_KEY_ID=admin -e AWS_SECRET_ACCESS_KEY=password -e AWS_REGION=us-east-1 \
  apache/spark:4.1.3-python3 /opt/spark/bin/spark-submit \
  --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0,org.apache.iceberg:iceberg-aws-bundle:1.11.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.demo.type=rest \
  --conf spark.sql.catalog.demo.uri=http://rest:8181 \
  --conf spark.sql.catalog.demo.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
  --conf spark.sql.catalog.demo.s3.endpoint=http://minio:9000 \
  --conf spark.sql.catalog.demo.s3.path-style-access=true \
  --conf spark.sql.catalog.demo.warehouse=s3://warehouse/ \
  --master 'local[2]' /work/test.py

Two jars, not one. iceberg-spark-runtime is the engine integration: the catalog implementation, the SQL extensions, the readers and writers. iceberg-aws-bundle supplies S3FileIO and the AWS SDK underneath it. With only the first, the session starts, CREATE TABLE may even succeed, and the first write fails on a missing S3 class. Match both to the same Iceberg version.

The Scala suffix must match the engine. iceberg-spark-runtime-4.1_2.13 is Spark 4.1 on Scala 2.13. A mismatch here produces NoSuchMethodError at runtime rather than a clean failure at startup.

--network lake matters. The Spark container has to resolve minio and rest by name, which only works on the compose network.

Creating the first table

spark.sql("CREATE NAMESPACE IF NOT EXISTS demo.db")
spark.sql("""CREATE TABLE demo.db.trips (
               trip_id BIGINT, pickup TIMESTAMP, fare DOUBLE, city STRING)
             USING iceberg PARTITIONED BY (days(pickup))""")

(spark.range(0, 5000)
   .withColumn("pickup", F.to_timestamp(F.lit("2026-02-10 08:00:00")))
   .withColumn("fare", (F.col("id") % 120).cast("double"))
   .withColumn("city", F.concat(F.lit("city-"), (F.col("id") % 4).cast("string")))
   .selectExpr("id AS trip_id", "pickup", "fare", "city")
   .writeTo("demo.db.trips").append())

rows       = 5000
namespaces = ['db']
snapshots  = 1
sample     = [city-0: 1250, city-1: 1250, city-2: 1250, city-3: 1250]

What landed in object storage

This is the part worth looking at, because it is the table.

docker exec mc mc ls -r m/warehouse

10KiB  db/trips/data/pickup_day=2026-02-10/00000-2-08479275-...-00001.parquet
 922B  db/trips/metadata/00000-fbb1be3d-4ab4-46bc-acb5-4df4ba54eef6.metadata.json
2.0KiB db/trips/metadata/00001-9804ab23-868a-4cbe-ba66-b851437255be.metadata.json
7.3KiB db/trips/metadata/c4295385-545a-4b9f-b472-4a7947ad2b53-m0.avro
4.3KiB db/trips/metadata/snap-7554101406213169966-1-c4295385-....avro

Five objects for one table. One holds data; the other four are the metadata tree:

Two metadata files exist because there were two commits: one creating the table, one appending. Every commit writes a new metadata file rather than editing the old one, which is what makes time travel possible and what makes metadata accumulate.

The data path is data/pickup_day=2026-02-10/, a directory nobody asked for. The table was declared PARTITIONED BY (days(pickup)) and Iceberg derived the partition value from the timestamp — hidden partitioning, visible on disk.

The naming difference that makes tutorials disagree

Under this REST catalog the metadata files are 00000-<uuid>.metadata.json. Run the same exercise against a Hadoop catalog and you get v1.metadata.json, v2.metadata.json and a version-hint.text alongside them.

Both are correct; they are different catalog implementations. The Hadoop catalog finds the current metadata by reading version-hint.text, which is why it needs a filesystem with atomic renames and why it is not safe on plain S3. A REST catalog keeps the pointer in a service, so the filenames need no ordering convention at all.

If you have read two Iceberg tutorials that describe the metadata directory differently, this is why.

Checking it from the other side

The MinIO console is on http://localhost:9001, logging in with admin / password. Browsing warehouse/db/trips shows the same five objects, which is a useful sanity check when a write appears to succeed but writes nowhere you expected.

The catalog API answers directly:

curl -s http://localhost:8181/v1/config

{"defaults":{},"overrides":{"namespace-separator":"%2E"},"endpoints":["POST v1/oauth/tokens", ...

If this returns nothing, Spark’s failures are catalog failures and no amount of S3 configuration will fix them. Check this before anything else.

Adding a notebook

For interactive work, swap the spark-submit container for a Jupyter service on the same network, with the same packages and catalog configuration passed through PYSPARK_SUBMIT_ARGS or set on the SparkSession builder. The configuration does not change — only where you type it.

The one thing worth keeping out of the notebook is the credentials. They are environment variables above for a reason: a notebook gets committed, and admin/password in a repo is a habit that survives into environments where the values are real.

When this setup is not enough

This playground is deliberately a single node with one engine. It will not show you what matters about concurrent writers on the same table, because there is one writer. It will not show you S3 throttling, cross-region latency or IAM, because MinIO has none of them. And the iceberg-rest-fixture image is what its name says — a fixture for testing, not a catalog to run a business on. For production the catalog question deserves its own decision, and running Iceberg on Apache Polaris covers one open-source answer to it.

What it does give you is a correct mental model of the three-way contract, and a place to break things cheaply.

Cleaning up

docker compose down -v

The -v removes the volume, and with it the warehouse bucket. Without it, the next up starts with the tables you left behind, which is occasionally what you want and usually a surprise.

References

• Iceberg REST catalog spec for what a catalog is required to do
• Iceberg AWS integration for S3FileIO and its properties
• MinIO documentation for the container setup and console
• Apache Iceberg architecture for the metadata tree these five objects form
• Running Iceberg on Apache Polaris for a production-shaped catalog

Trademarks

Apache Iceberg, Apache Spark, Apache Parquet, Apache Polaris, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. MinIO is a trademark of MinIO, Inc.