Run every demo on this blog locally: one compose stack for Hudi, Iceberg and Delta

• Why a local stack
• What is in it
• Getting it running
• Choosing a Spark line
• Running the demos
• The demos map to the posts
  • The numbers come out the same
• Things that cost me time
• Resetting and cleaning up
• What this does not cover
• References
• Trademarks

TL;DR

• datalake-docker brings up MinIO, a Hive Metastore, Spark, Kafka and Trino as one Compose project, so the demos need no cloud account.
• It now builds on two Spark lines: SPARK_VERSION=3.5.5 (Scala 2.12) and SPARK_VERSION=4.0.2 (Scala 2.13). Everything else follows from that choice.
• demos/ holds scripts that reproduce the posts’ measurements. The copy-on-write versus merge-on-read figures came out identical here to the numbers in that post.
• Spark 4.1 is deliberately not offered. Delta publishes no Scala 2.13 build past 4.0.0, so a 4.1 image would come without Delta, and the point is all three formats side by side.
• Hudi’s first write to a new table bootstraps its metadata table and takes around twenty seconds. A smoke test run immediately after start can read that as a failure.

Why a local stack

Every measured claim in these posts — the file counts in copy-on-write versus merge-on-read, the snapshot arithmetic, the spill numbers — came from running something, not from recalling it. That is only sustainable if running something is cheap.

A cloud account is not cheap in the way that matters here: it is slow to reset, it costs money to leave running, and a demo that depends on your IAM setup is not a demo anyone else can reproduce.

The stack below runs on a laptop, resets with one command, and gives the three table formats a shared catalog and a shared object store, which is exactly the environment where the interesting differences between them show up.

What is in it

flowchart TB
    subgraph ingest["ingestion"]
      PG[(Postgres)] --> DBZ[Debezium / Kafka Connect]
      DBZ --> K[Kafka]
    end
    subgraph compute["compute"]
      SM[Spark master] --- SW[Spark worker]
      TR[Trino]
      FL[Flink]
    end
    subgraph meta["catalog"]
      HMS[Hive Metastore]
    end
    subgraph store["storage"]
      MO[(MinIO, S3 API)]
    end
    K --> SM
    SM --> HMS
    TR --> HMS
    FL --> HMS
    SM --> MO
    TR --> MO
    HMS --> MO

MinIO is the S3 endpoint, so s3a://warehouse/... in a demo is the same path shape you would write against real object storage. One Hive Metastore is shared by Spark, Trino and Flink, which is what makes a table written by one engine readable by another without extra registration.

Getting it running

git clone https://github.com/rangareddy/datalake-docker.git
cd datalake-docker

# build the images you need - this is the slow part, once
IMAGES=spark SPARK_VERSION=4.0.2 ./docker_build/build_docker_images.sh

# start the stack on the same pin
SPARK_VERSION=4.0.2 sh docker_run/run_datalake.sh start

IMAGES limits the build to a subset. Building all eight images takes a long time because the XTable image compiles from source, and most demos need only Spark.

Check what came up:

sh docker_run/run_datalake.sh status

minio           Up (healthy)
hive-metastore  Up (healthy)
spark-master    Up (healthy)
spark-worker    Up (healthy)
postgres        Up (healthy)
kafka           Up (healthy)

Choosing a Spark line

SPARK_VERSION selects a profile, and the rest of the matrix is derived from it, because the Spark line dictates the Scala binary, the bundled Hadoop, and which builds of the three connectors exist at all:

Spark 4.1 is not a profile, and the reason is Delta. Iceberg publishes iceberg-spark-runtime-4.1_2.13 and Hudi publishes hudi-spark4.1-bundle_2.13, but delta-spark_2.13 stops at 4.0.0. A 4.1 image would therefore run two of the three formats, which defeats the purpose of a stack that exists to compare them. The moment Delta ships a 4.1 build, the profile is a three-line change.

One consequence of the Spark 4 line is worth knowing if you build images yourself: Hadoop 3.4 switched S3A from AWS SDK v1 to v2. Hadoop 3.3.x wants com.amazonaws:aws-java-sdk-bundle; 3.4.x wants software.amazon.awssdk:bundle. Shipping the wrong one gives a ClassNotFoundException on the first s3a:// call. The build script now caches the jars per Hadoop version and stages a clean directory for the image, so switching profiles does not leave the previous SDK behind on the classpath.

Running the demos

docker cp demos spark-master:/opt/demos
docker exec -it spark-master bash /opt/demos/smoke_test_formats.sh

== environment ==
  spark        version 4.0.2
  scala        2.13
  hudi         1.2.0   hudi-spark4.0-bundle_2.13-1.2.0.jar
  iceberg      1.11.0
  delta        4.0.0
  java         "17.0.15"
== iceberg ==
  PASS  iceberg write/update/delete: ROWS=1 DEPT=Analytics
  PASS  iceberg snapshot history: SNAPS=3
== hudi ==
  PASS  hudi write/update: ROWS=2 DEPT=Software
== delta ==
  PASS  delta write/update/delete: ROWS=1 DEPT=Analytics

== summary ==
  passed 4, failed 0

That is the check to run after building an image: all three formats writing, updating and reading against MinIO.

Individual demos go through a wrapper that supplies the jars and catalog configuration each format needs:

docker exec -it spark-master bash /opt/demos/run_demo.sh iceberg/cow_vs_mor.py

The wrapper infers the format from the first path component, so a demo script contains only its own logic, and jar names are resolved with globs rather than pinned versions — which is what lets the same demo run on both Spark profiles unchanged.

The demos map to the posts

The numbers come out the same

This is the part worth checking rather than trusting. The copy-on-write versus merge-on-read post measured its file accounting on a standalone Spark 4.1.3 with Iceberg 1.11.0. Running the same demo here, on Spark 4.0.2 inside the stack:

  copy-on-write before        data=2f/4000r                            query returns 4000
  copy-on-write after DELETE  data=2f/3900r                            query returns 3900
  snapshot ops                ['append', 'overwrite']
  merge-on-read before        data=2f/4000r                            query returns 4000
  merge-on-read after DELETE  data=2f/4000r  position-delete=1f/100r   query returns 3900
  snapshot ops                ['append', 'delete']

Identical to the post: copy-on-write rewrote to 3,900 records under an overwrite snapshot, merge-on-read kept 4,000 records and added a 100-record delete file under a delete snapshot, and both queries return 3,900 rows.

The evolution demo reproduces its post the same way:

== schema evolution ==
  rows still readable   9000
  NULL currency rows    9000
  data files 3 -> 3  (unchanged = no rewrite)
== partition evolution ==
  spec_id per data file [1, 0, 0]

Three data files before the schema change and three after, and two of the three files still carrying the old partition spec.

Things that cost me time

Four, each found by running it rather than reading about it.

Hudi’s first write is slow, and it looks like a hang. Creating a table and inserting into it for the first time bootstraps Hudi’s metadata table, which took about twenty seconds on a cold stack against MinIO. A smoke test run immediately after start read that as a failure; the same test passed on a second run with no change. If you are timing anything Hudi-related, warm the table first.

minio/minio no longer resolves on Docker Hub. The stack could not start at all until the compose files were pointed at quay.io/minio/minio. If you have an older checkout and start fails with pull access denied for minio/minio, that is why.

A failed apt can still produce an image. The Spark image’s package layer was failing against a Debian mirror whose index advertised package versions its pool no longer had. The build carried on, because wget already exists in the base image and every later layer kept working — and the breakage only surfaced at runtime as java: command not found. The layer now verifies java -version before it finishes, so the failure happens where it is diagnosable.

Version-agnostic jar globs matter more than they look. The README said ls $HUDI_HOME/hudi-spark3.5-bundle_*.jar, which resolves to nothing on a Spark 4 image where the bundle is hudi-spark4.0-bundle_2.13. Every demo now globs hudi-spark*-bundle_*.jar.

Resetting and cleaning up

sh docker_run/run_datalake.sh stop

stop does a down without -v, so the Postgres data directory and the MinIO object store under docker_run/data/ survive. That is usually what you want between runs and occasionally a surprise — for a genuinely clean slate, delete that directory.

Individual demos are safe to re-run: each drops and recreates its own tables, so a second run is not polluted by the first.

What this does not cover

The stack is a single node with one writer. It will not show you concurrent-writer behaviour at scale, S3 throttling, cross-region latency, or anything about IAM, because MinIO has none of them. The Spark-internals measurements in these posts — shuffle internals and performance tuning — were taken against a plain Spark container rather than this stack, because they are about the engine rather than the lakehouse around it.

What it does give you is a correct, reproducible environment for the table-format behaviour, which is where most of the surprises live.

References

• datalake-docker — the stack, the demos and the build script
• Apache Iceberg documentation for the Spark procedures the demos call
• Apache Hudi Spark guide for the table properties Hudi requires
• Delta Lake documentation for the Scala 2.13 build matrix
• A local Iceberg playground for a smaller three-service alternative when you only need Iceberg

Trademarks

Apache Iceberg, Apache Hudi, Apache Spark, Apache Hive, Apache Kafka, Apache Flink, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Delta Lake is a trademark of the Linux Foundation. MinIO is a trademark of MinIO, Inc. Trino is a trademark of the Trino Software Foundation.