Apache Hudi, Iceberg and Spark internals
Hands-on notes, production troubleshooting and interactive tools for Apache Hudi, Iceberg, Spark, Kafka and the lakehouse stack underneath them.
Newest first. 7 posts in total.
XTable exposes one Hudi, Iceberg or Delta table as the other two by writing metadata rather than copying data. It picks FULL or INCREMENTAL per target format on every run, and one setting on your side keeps it on the fast path. Here is how the decision works, from source, and how to confirm it in the log.
Read the postFour techniques that pay for themselves on nearly every Spark investigation: swap in a custom Log4j 2 config, trace class loading, raise the JVM stack size, and route traffic through an HTTP proxy. All four go through the same two configs, so learning one teaches you the rest.
Read the postPick a catalog type (Hive, Hadoop, REST or JDBC) plus Spark, Iceberg and Scala versions, and get the spark-shell command with the right iceberg-spark-runtime coordinates and catalog configs. Version pairs come from each Iceberg release's own build, up to Iceberg 1.11.0 on Spark 4.1.
Read the postparquet-cli is the maintained command-line tool for Parquet, and it does more than parquet-tools ever did. Here is the command-for-command migration, every transcript captured from a real run, plus what to know about rewrite.
Read the postPaste a spark-submit command and get it back formatted across lines or minified onto one line, with every Spark option resolved to the configuration property it sets and your application's own arguments kept separate.
Read the postEnter the node count, cores per node and memory per node for your cluster and get back executor count, executor cores, executor memory and memory overhead, ready to paste into spark-shell.
Read the post