Apache Hudi, Iceberg and Spark internals
Hands-on notes, production troubleshooting and interactive tools for Apache Hudi, Iceberg, Spark, Kafka and the lakehouse stack underneath them.
Newest first. 8 posts in total.
All three formats give you ACID commits, time travel and schema evolution, so a feature checklist will not tell you which to pick. What separates them is how each one tracks table state, how each one applies a row-level update, and what each one asks you to operate. This guide takes all three to source level against Iceberg 1.11.0, Hudi 1.2.0 and Delta Lake 4.4.0.
Read the postXTable exposes one Hudi, Iceberg or Delta table as the other two by writing metadata rather than copying data. It picks FULL or INCREMENTAL per target format on every run, and one setting on your side keeps it on the fast path. Here is how the decision works, from source, and how to confirm it in the log.
Read the postFour techniques that pay for themselves on nearly every Spark investigation: swap in a custom Log4j 2 config, trace class loading, raise the JVM stack size, and route traffic through an HTTP proxy. All four go through the same two configs, so learning one teaches you the rest.
Read the postPick a catalog type (Hive, Hadoop, REST or JDBC) plus Spark, Iceberg and Scala versions, and get the spark-shell command with the right iceberg-spark-runtime coordinates and catalog configs. Version pairs come from each Iceberg release's own build, up to Iceberg 1.11.0 on Spark 4.1.
Read the postparquet-cli is the maintained command-line tool for Parquet, and it does more than parquet-tools ever did. Here is the command-for-command migration, every transcript captured from a real run, plus what to know about rewrite.
Read the postPaste a spark-submit command and get it back formatted across lines or minified onto one line, with every Spark option resolved to the configuration property it sets and your application's own arguments kept separate.
Read the post