Run every demo on this blog locally: one compose stack for Hudi, Iceberg and Delta

• Overview
• The core problem and architecture
  • What a lakehouse demo actually needs
  • The services and how they connect
  • Why the stack ships its own images
• Under the hood: images, connectors and file layout
  • What is in the repo
  • One tag for the whole stack
  • Getting all three formats onto Spark 4.1
  • Three formats in one Spark session
• Hands-on: start and verify
  • Prerequisites
  • Step 1: get the compose files
  • Step 2: start the stack
  • Step 3: check the Spark worker is registered
  • Step 4: check each table format
  • Step 5 (all profile): send records through Kafka
  • Step 6: run a post’s script
  • The numbers come out the same
• Real-world scenarios and failures
• Optimisation, tuning and day-to-day operation
  • Day-to-day commands
  • Configuration lives in two places
  • Checklist
• Summary and key takeaways
• References
• Trademarks

TL;DR

• datalake-playground brings up MinIO, a Hive Metastore, Spark and Postgres, plus Kafka, Trino and Jupyter in its all profile, as one Compose project, so the demos need no cloud account.
• The public repo is run-only. It holds the compose files and a start script; the images are pulled from Docker Hub on the first start, and every setting, including the MinIO credentials, is inline in the compose file.
• It runs on two Spark lines, and one variable, SPARK_VERSION, picks the Scala binary, the Hadoop version and all three table-format connectors.
• Both lines carry Hudi, Iceberg and Delta in one Spark session. That needed one catalog arrangement, one extension order and one jar nobody ships, all below.
• Every image is tagged with one stack version rather than the version of the component inside it, so the compose files hold one pin.
• A container that says Up (healthy) is not a working service. The stack’s own Spark worker was once healthy and unregistered, and every cluster job waited forever.

Overview

Every measured claim in these posts, the file counts in copy-on-write versus merge-on-read, the snapshot arithmetic in time travel and rollback, the spill numbers, came from running something, not from recalling it. That is only sustainable if running something is cheap, and only useful if you can run the same thing.

A cloud account is not cheap in the way that matters here. It is slow to reset, it costs money to leave running, and a demo that depends on your IAM setup is not a demo anyone else can reproduce.

This post covers the stack those posts run on:

• What is in it, and why the stack ships its own pinned images.
• What is inside the images, including the Spark 4.1 connector problems they had to solve.
• How to start and verify it from the public repo.
• The failures it has produced, and what each one looked like.

The core problem and architecture

What a lakehouse demo actually needs

A table-format demo needs four things that agree with each other. There is an S3 endpoint, a catalog both engines share, a compute engine with the right connector jars, and a database behind the catalog. Getting any pair wrong produces errors that look like format bugs: a Hudi write that fails on a missing metastore host, or an Iceberg read that fails on a Java version. The stack’s job is to make those four agree once, so each post can be about the format.

The services and how they connect

 ingestion (all profile)
 ┌──────────┐   ┌───────────────┐   ┌───────┐
 │ Postgres │──>│ Kafka Connect │──>│ Kafka │
 └──────────┘   │ (Debezium)    │   └───┬───┘
                └───────────────┘       │ kafka:29092
    ┌───────────────────────────────────┘
    v
 compute                                          query (all profile)
 ┌──────────────────┐     ┌──────────────┐        ┌───────┐
 │ spark-master     │─────│ spark-worker │        │ Trino │
 │ + Jupyter (all)  │     └──────────────┘        └─┬───┬─┘
 └──┬────────────┬──┘                               │   │
    │ thrift://hive-metastore:9083                  │   │
    v            │                                  │   │
 catalog         │                                  │   │
 ┌───────────────┴───────────────┐<─────────────────┘   │
 │ Hive Metastore + HiveServer2  │                      │
 │ backed by Postgres            │                      │
 └──────────────┬────────────────┘                      │
                v            s3a:// from Spark          v
 storage ┌──────────────────────────────────────────────────┐
         │ MinIO   http://minio:9000   s3a://warehouse/     │
         └──────────────────────────────────────────────────┘

Walk it from the bottom.

1. MinIO is the S3 endpoint. s3a://warehouse/... in a demo is the same path shape you would write against real object storage.
2. One Hive Metastore is shared by Spark and Trino. A table written by one engine is readable by the other without extra registration.
3. Postgres backs the metastore. It also holds a sample employees table loaded from db_scripts/, as does MySQL in the all profile.
4. Spark carries all three connectors. The Hudi, Iceberg and Delta jars are already on disk under $HUDI_HOME, $ICEBERG_HOME and $DELTA_HOME.
5. In the all profile, changes arrive through Kafka. Kafka Connect runs Debezium’s Postgres connector, so a row changed in Postgres becomes a Kafka record. Spark reads it with the Kafka source already in its image (spark-sql-kafka-0-10 and its three dependencies), with no extra --jars.

Everything sits on one Docker network, datalake, so services reach each other by name.

Why the stack ships its own images

Every image the stack starts is its own rangareddy1988/ranga-* image with its upstream base pinned. Ten of them are thin wrappers whose only job is to pin a base under a name the stack controls.

That trade-off is not tidiness. docker.io/minio/minio stopped resolving, and two other upstream images the stack once used were discontinued or lost their pinned build. Each of those broke a stack that pulled upstream images at start time. Owning the image means an upstream change no longer reaches you until the stack chooses to take it.

The public repo holds what you need to run the stack: the compose files, the start script, and the SQL files they mount. The images are published as rangareddy1988/ranga-*:1.0.0, built for linux/amd64, and Compose pulls them the first time you start. On Apple Silicon they run under emulation, which works and is slower.

Under the hood: images, connectors and file layout

What is in the repo

datalake-playground/
├── README.md
├── run_datalake.sh          start | stop | restart | status | logs | validate
├── docker-compose.yml       core profile
├── docker-compose_all.yml   adds MySQL, Trino and Jupyter
├── db_scripts/              Postgres and MySQL init SQL
└── lakehouse_check.py       one check function per table format

Everything sits at the root, and data/ and logs/ appear next to the compose files on the first start.

There are no .env files. The MinIO endpoint and credentials are set inline in the environment: block of every service that reads or writes s3a://:

    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - AWS_DEFAULT_REGION=us-east-1
      - S3_PATH_STYLE_ACCESS=true
      - AWS_ENDPOINT_URL=http://minio:9000

Those are local demo values. The same credentials are baked into the images’ core-site.xml and Trino catalogs, so they change only as a set.

One tag for the whole stack

All images carry one tag, for example ranga-spark:1.0.0 and ranga-hive:1.0.0. It is the version of the stack, not of the component inside it. So the compose files hold one pin, ${IMAGE_VERSION:-1.0.0}, rather than twelve, and one component moving cannot leave the compose files asking for a tag that does not exist.

The Spark 3.5 and 4.1 lines are separate images, ranga-spark and ranga-spark4. run_datalake.sh picks one from SPARK_VERSION:

Only these two lines exist. The Spark line dictates the Scala binary, the bundled Hadoop, and which releases of the three connectors exist at all.

Hadoop 3.4 switched S3A from AWS SDK v1 to v2. Hadoop 3.3.x wants com.amazonaws:aws-java-sdk-bundle; 3.4.x wants software.amazon.awssdk:bundle. Shipping the wrong one gives a ClassNotFoundException on the first s3a:// call. Each Spark image carries the SDK that matches its Hadoop, and only that one, so switching lines does not leave the previous SDK on the classpath.

Getting all three formats onto Spark 4.1

The 4.1 image carried Iceberg alone for a while. Both exclusions were real, and they were not the same kind of problem.

Delta was a version mistake on my part. Delta 4.0.0 fails on Spark 4.1 with NoSuchMethodError: org.apache.spark.internal.LogKey.$init$, an internal trait Spark 4.1 changed. I read that as “Delta has no Spark 4.1 build”. It does: 4.1.0. A version bump was the entire fix, and I would have found it sooner by checking Maven Central rather than trusting my own earlier note.

Hudi was more interesting, because the symptom pointed away from the cause. hudi-spark4.1-bundle_2.13:1.2.0 installs cleanly and writes fine. Every UPDATE died:

NoClassDefFoundError: org/apache/parquet/variant/VariantConverters$ParentConverter
  at org.apache.parquet.avro.AvroReadSupport.getDataModel
  at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge

Inserts working and merges failing reads like a data problem: a bad record, a schema mismatch, a lock. It is neither. Hudi’s bundle brings its own parquet-avro, which calls VariantConverters. Parquet 1.16.0, the version Spark 4.1 ships, moved those classes out of parquet-column into a separate parquet-variant module, and the Spark distribution does not include it. Counting the classes makes it obvious:

parquet-column-1.16.0.jar    0 VariantConverters classes
parquet-variant-1.16.0.jar  32 VariantConverters classes

Adding that one jar fixes it. The 4.1 image now ships parquet-variant at exactly the version of the parquet-column Spark ships, because a mismatched one would reintroduce the problem somewhere harder to see.

Three formats in one Spark session

lakehouse_check.py writes and updates a table in Iceberg, Hudi and Delta from one Spark session. Three things had to be true, and none of them is in the quick-start docs.

Delta and Hudi both want to be spark_catalog. Delta refuses a DataFrame write without DeltaCatalog installed there; the session extension alone is not enough. So Delta takes that slot, Hudi writes by path where no catalog is involved, and Iceberg stays on its own named catalog:

 spark_catalog  ->  DeltaCatalog               Delta tables
 ice            ->  Iceberg SparkCatalog       Iceberg tables, type=hive
 (by path)      ->  hudi data source           Hudi tables at s3a://...

That is the only arrangement where all three coexist.

Delta’s extension has to come before Iceberg’s. With IcebergSparkSessionExtensions listed first in spark.sql.extensions, Delta’s SQL UPDATE fails at analysis:

scala.MatchError: DeltaTableV2(org.apache.spark.sql.SparkSession@3e056063,
  s3a://warehouse/playground_check/delta_orders,None,None,None,Map())
  (of class org.apache.spark.sql.delta.catalog.DeltaTableV2)

Iceberg’s UPDATE rewrite rule runs first and does not recognise a Delta table. The first guess, Hudi’s extension, was wrong: removing it changed nothing. Only the order did. With the same three jars and Delta’s extension listed first, the UPDATE succeeded on Delta, and Iceberg’s UPDATE still succeeded too.

Do not put every Hudi jar on the classpath. hudi-utilities-bundle registers a hudi data source as well as hudi-spark<N>-bundle, and both together fail the write with Multiple sources found for hudi.

Hands-on: start and verify

Prerequisites

Step 1: get the compose files

git clone https://github.com/rangareddy/datalake-playground.git
cd datalake-playground

Step 2: start the stack

SPARK_VERSION=3.5.9 sh run_datalake.sh start              # core profile
PROFILE=all SPARK_VERSION=3.5.9 sh run_datalake.sh start  # adds Kafka, MySQL, Trino, Jupyter

The first start pulls the images. Startup is ordered by health checks; expect a couple of minutes while Hive initialises its metastore schema. The start script defaults PLATFORM to linux/amd64 because that is what the images are built for. Setting it to linux/arm64 fails the pull with no matching manifest.

sh run_datalake.sh status

Every service should read Up (healthy), except the mc sidecar, which has no health check. The mc sidecar runs from ranga-minio and the notebook from ranga-spark, so those add no images of their own. The core profile is seven containers and was healthy 20 to 40 seconds after start. all is sixteen, with Kafka, and took 92 to 102 seconds; Kafka Connect is the slowest, because it scans every plugin jar before it opens its port.

Step 3: check the Spark worker is registered

Up (healthy) means a port answers. It does not mean the worker joined the master, and a job submitted to spark://spark-master:7077 without a worker waits forever:

curl -s http://localhost:8080/json/ | python3 -c \
  "import sys, json; print('alive workers:', json.load(sys.stdin)['aliveworkers'])"

It should print alive workers: 1. If it prints 0, run docker restart spark-worker and check again.

Step 4: check each table format

lakehouse_check.py, in the root of the repo, has one function per format: check_iceberg, check_hudi and check_delta. Each writes three orders, updates one, and reads them back; cleanup then deletes everything the checks created. create_spark_session holds the catalog arrangement and extension order described above, so one session serves all three.

docker cp lakehouse_check.py spark-master:/tmp/
docker exec spark-master bash -c '$SPARK_HOME/bin/spark-submit --master "local[2]" \
  --driver-memory 1g \
  --jars "$(ls $ICEBERG_HOME/*.jar $DELTA_HOME/*.jar $HUDI_HOME/hudi-spark*-bundle_*.jar \
            | paste -sd, -)" \
  /tmp/lakehouse_check.py'

On the 3.5.9 line it printed:

spark 3.5.9
  PASS  iceberg  rows=3 shipped=1
  PASS  hudi     rows=3 shipped=1
  PASS  delta    rows=3 shipped=1
passed 3 of 3

On the 4.1.3 line the same three checks pass, with spark 4.1.3 on the first line. A passing run still prints a handful of warnings, such as NativeCodeLoader: Unable to load native-hadoop library and, from Hudi, Unable to attach Serviceability Agent. The repo’s README lists each one and why it is harmless.

--driver-memory 1g is there on purpose; the failures section explains why.

Step 5 (all profile): send records through Kafka

The broker is kafka:29092 inside the network. A topic, three records and a read back:

docker exec kafka kafka-topics --bootstrap-server kafka:29092 \
  --create --topic orders --partitions 1 --replication-factor 1
printf '1001,PLACED\n1002,PLACED\n1003,SHIPPED\n' | \
  docker exec -i kafka kafka-console-producer --bootstrap-server kafka:29092 --topic orders
docker exec kafka kafka-console-consumer --bootstrap-server kafka:29092 \
  --topic orders --from-beginning --max-messages 3

Created topic orders.
1001,PLACED
1002,PLACED
1003,SHIPPED

Spark reads the same topic with spark.read.format("kafka") and kafka.bootstrap.servers=kafka:29092; the repo’s README has the full script. Kafka Connect’s REST API is on localhost:8083 and Kafka UI on localhost:9082.

Step 6: run a post’s script

Each post that uses the stack publishes its scripts in full. They all run the same way: copy the script into spark-master and submit it with the jars it needs, which are already in the image under $ICEBERG_HOME, $HUDI_HOME and $DELTA_HOME.

docker cp time_travel_rollback.py spark-master:/tmp/
docker exec spark-master bash -c \
  '$SPARK_HOME/bin/spark-submit --master "local[2]" \
     --jars $(ls $ICEBERG_HOME/iceberg-spark-runtime*.jar) /tmp/time_travel_rollback.py'

That one is from time travel and rollback. Jupyter Lab (all profile) runs from the same Spark image at http://localhost:8888, with the same jars and environment.

The numbers come out the same

This is the part worth checking rather than trusting. The copy-on-write versus merge-on-read post measured its file accounting on Spark 4.1.3. The same measurement on the 3.5.9 line:

== copy-on-write vs merge-on-read ==
  copy-on-write before           data=2f/4000r                                        query returns 4000
  copy-on-write after DELETE     data=2f/3900r                                        query returns 3900
  snapshot ops                   ['append', 'overwrite']
  merge-on-read before           data=2f/4000r                                        query returns 4000
  merge-on-read after DELETE     data=2f/4000r  position-delete=1f/100r               query returns 3900
  snapshot ops                   ['append', 'delete']

Identical to the post. Copy-on-write rewrote to 3,900 records under an overwrite snapshot. Merge-on-read kept 4,000 records and added a 100-record delete file under a delete snapshot. Both queries return 3,900 rows. The claims are about Iceberg, not about one Spark build.

Real-world scenarios and failures

Each of these was found by running the stack, not by reading about it.

The Spark worker was healthy and doing nothing. While re-verifying this post, the stack’s end-to-end test failed spark:worker-registered with expected ["aliveworkers":1]. The worker container read Up (healthy), and its JVM was running, but the master’s /json/ endpoint listed no workers. The custom images end with a keepalive loop, so container state says nothing about the service inside. The next check then submitted SparkPi to the cluster, and with no worker to run it, the job waited for resources indefinitely. Local-mode scripts (--master "local[2]") were unaffected, which is why everything else in this post still ran. docker restart spark-worker re-registered it in 15 seconds. Check the master before blaming a job:

curl -s http://localhost:8080/json/ | python3 -c \
  "import sys, json; print(json.load(sys.stdin)['aliveworkers'])"

Hudi’s first write is slow, and it looks like a hang. Creating a table and inserting into it for the first time bootstraps Hudi’s metadata table, which took about twenty seconds on a cold stack against MinIO. A smoke test run immediately after start read that as a failure; the same test passed on a second run with no change. Warm the table before timing anything.

“It has no build that runs” is a conclusion worth re-testing. I wrote that about Hudi on Spark 4.1 and it was half right at best: it ran, it just could not merge, and the fix was a jar rather than a wait. The same note said Delta had no Scala 2.13 build past 4.0.0; 4.1.0 was on Maven Central. Version claims about someone else’s project go stale quietly.

A full Docker disk does not announce itself. Seven unrelated checks failed at once: Hive could not create a temp directory and Spark’s S3A writes threw. The cause was the VM’s filesystem at 100%. Freeing 24 GB made every check pass on the next run. docker system df shows where the space went.

A Spark script run on the host uses the host’s Spark. A format check run with spark-sql from a laptop’s PATH reported spark version 2.4.4, failed every check, and left a metastore_db/ and derby.log in the current directory. Spark scripts belong inside spark-master, through docker exec.

Exit code 137 in the middle of a job is memory, not the job. The format check passed Iceberg and Hudi and then died with exit code 137, a SIGKILL, in two of five runs. The Docker VM had 7.8 GiB while running the full 16-container all profile, which wants 10 GB, and about 1.2 GB was available. The driver starts with the image’s spark.driver.memory 2g, and by the third connector the JVM outgrew what was left, so the kernel killed it. --driver-memory 1g fits what the check needs. Short of memory, a job can also just crawl: in one all-profile run the check took 17 minutes, where the next run took under 2. The real fix is giving Docker the memory the profile asks for.

A restart can quietly drop half the stack. SPARK_VERSION=4.1.3 sh run_datalake.sh restart on a running all stack exited 0 and left seven containers: Kafka, MySQL, Trino and Jupyter were gone. restart works on the profile it is given, core by default, and docker compose down --remove-orphans removes every container that profile does not define. Pass the profile you started with: PROFILE=all SPARK_VERSION=4.1.3 sh run_datalake.sh restart.

The first start logs a metastore connection failure, then recovers. On an empty data/, hive-metastore logged Connection to postgres:5432 refused and HiveMetaException: Failed to get schema version, then initialised its schema and came up healthy. Postgres runs its init SQL on a temporary server that listens only on a Unix socket, shuts it down, and opens TCP three seconds later. Its health check connects through the socket, so it passes during that window, and the metastore starts early. It happened in one of four first starts, and the metastore’s retry absorbed it every time.

Optimisation, tuning and day-to-day operation

Day-to-day commands

stop does a down without -v, so the Postgres data directory and the MinIO object store under data/ survive. For a clean slate:

sh run_datalake.sh stop
rm -rf data logs
sh run_datalake.sh start

Configuration lives in two places

Service wiring, ports and the MinIO credentials are in the compose files, and the start script reads three optional variables: PROFILE (core or all), SPARK_VERSION and PLATFORM. Engine configuration, spark-defaults.conf, core-site.xml, hive-site.xml and the Trino catalogs, is baked into the images. Editing a compose value takes effect on restart; changing engine configuration means passing --conf to your job instead.

Checklist

• Give Docker 4 GB of memory and 10 GB of disk for core, 10 GB and 20 GB for all.
• Leave PLATFORM at linux/amd64; the images are built for it.
• Check aliveworkers on the master’s /json/ endpoint before submitting to the cluster.
• Run lakehouse_check.py after the first start.
• Run Spark scripts inside spark-master, not on the host.
• List Delta’s session extension before Iceberg’s when a session uses both.
• Warm a new Hudi table before timing it.
• Create your own tables under a namespace no post uses, and drop them with PURGE afterwards.

Summary and key takeaways

The stack is a single node with one writer. It will not show you concurrent writers at scale, S3 throttling, cross-region latency, or anything about IAM, because MinIO has none of them. The Spark-internals measurements in shuffle internals and performance tuning were taken against a plain Spark container, because they are about the engine rather than the lakehouse around it.

What it does give you is a correct, reproducible environment for table-format behaviour, which is where most of the surprises live.

• Clone and start. The public repo is the compose files and a start script, and the pinned images are pulled once.
• One variable picks the Spark line, and everything version-coupled follows from it.
• All three formats share one session only with Delta on spark_catalog, Iceberg on a named catalog, Hudi by path, and Delta’s extension listed first.
• Up (healthy) is not a test. Check the worker count and run lakehouse_check.py before trusting the stack.

References

• datalake-playground, the compose files, start script and lakehouse_check.py
• Apache Iceberg Spark procedures for the calls the demos make
• Apache Hudi Spark guide for the table properties Hudi requires
• Delta Lake quick start for the DeltaCatalog session configuration
• Docker Compose environment for inline service variables

Trademarks

Apache Spark, Apache Iceberg, Apache Hudi, Apache Kafka, Apache Hive, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Delta Lake is a trademark of the Linux Foundation. Trino is a trademark of the Trino Software Foundation. MinIO is a trademark of MinIO, Inc. Docker is a trademark of Docker, Inc.