TL;DR

• XTable (incubating) converts table metadata, not data. One set of Parquet files gets a second and third set of metadata so Iceberg and Delta readers can see a Hudi table without a copy.
• SyncMode has two values, FULL and INCREMENTAL. INCREMENTAL is a request that XTable validates against the source before honouring it, which is what makes the result trustworthy.
• The decision is made per target format. In one run Iceberg can sync incrementally while Delta rebuilds a full snapshot.
• Incremental needs the source to still hold history back to the last sync. isIncrementalSyncSafeFrom checks that per format: a live commit on the Hudi timeline, an unbroken Iceberg snapshot chain, an active Delta commit at or before the instant.
• Keeping source retention comfortably longer than your sync interval is the one setting that keeps you on the incremental path, and a single log string confirms it.

What XTable gives you

A lakehouse table is Parquet files plus metadata that says which files are live, what the schema is, and how the table has changed over time. The files are ordinary Parquet. The metadata is what makes it a Hudi, Iceberg or Delta table.

Apache XTable (incubating) works from that observation. Rather than copying data between formats, it reads the source table’s metadata into a format-agnostic internal model and writes out metadata in the target formats alongside it. The Parquet files are never rewritten and never duplicated. Point Trino at the Iceberg metadata and Databricks at the Delta metadata and both read the same bytes.

That design means correctness is not the thing you tune. The lever worth understanding is cost: whether XTable appends the last few commits to the target metadata or rebuilds it from scratch, which is the difference between a sync that finishes in seconds and one that reads the whole table.

This post is written against XTable 0.4.0-incubating. Class and method references are to the 0.4.0-incubating tag.

At that release, TableFormat declares HUDI, ICEBERG, DELTA, PAIMON and PARQUET, with values() returning the first four. Prerequisites: the XTable docs and a working knowledge of at least one of the three main formats.

Architecture: the internal model and the two sync paths

XTable is deliberately not an N-by-N set of converters. Every source is read into one internal model, and every target is written from that model, so adding a format is two adapters rather than six.

flowchart LR
  H[Hudi timeline] --> S[ConversionSource]
  I[Iceberg snapshots] --> S
  D[Delta log] --> S
  S --> M[Internal model:<br/>schema, partitioning,<br/>files, commits]
  M --> C{ConversionController}
  C -->|INCREMENTAL<br/>per target| INC[syncIncrementalChanges]
  C -->|FULL or<br/>declined| FULL[syncSnapshot]
  INC --> T[ConversionTarget<br/>per format]
  FULL --> T
  T --> OUT[Iceberg metadata/<br/>Delta _delta_log/<br/>Hudi .hoodie]

ConversionController is where the mode is chosen, and the two paths it can take are syncIncrementalChanges and syncSnapshot:

• syncSnapshot extracts an InternalSnapshot: every file relevant to the table at a point in time. The comment on SyncMode.FULL puts it plainly, it “will create a checkpoint of ALL the files relevant at a certain point in time”. Cost scales with the table.
• syncIncrementalChanges extracts the differential structures needed to move the target from its last synced instant to the current one. Cost scales with what changed.

On a table with a million files, that difference is the difference between a sync that finishes in a scheduled window and one that does not.

The decision, from source

Two methods decide this. The first, getFormatsToSyncIncrementally, filters the target formats down to those that can go incremental:

if (syncMode == SyncMode.FULL) {
  // Full sync requested by config, hence no incremental sync.
  return Collections.emptyMap();
}
return conversionTargetByFormat.entrySet().stream()
    .filter(entry -> {
      Optional<Instant> lastSyncInstant = ...getLastInstantSynced();
      List<Instant> pendingInstants = ...getInstantsToConsiderForNextSync();
      return isIncrementalSyncSufficient(conversionSource, lastSyncInstant, pendingInstants);
    })
    .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));

The .filter is worth reading closely, because it is a nice piece of design: the predicate is evaluated per target format. Each target carries its own TableSyncMetadata with its own last-synced instant, so one target catching up never forces the others to redo work. Add Delta as a new target to a job that has been syncing Iceberg for six months and that run will bootstrap Delta from a full snapshot while Iceberg continues incrementally, in the same job.

The second method, isIncrementalSyncSufficient, is the actual test:

Optional<Instant> earliestInstant =
    lastSyncInstant
        .map(instant -> Stream.concat(Stream.of(instant), pendingInstantsStream)
            .min(Instant::compareTo))
        .orElseGet(() -> pendingInstantsStream.min(Instant::compareTo));

if (!earliestInstant.isPresent()) {
  log.info("No previous InternalTable sync for target. Falling back to snapshot sync.");
  return false;
}

boolean isIncrementalSafeFromInstant =
    conversionSource.isIncrementalSyncSafeFrom(earliestInstant.get());
if (!isIncrementalSafeFromInstant) {
  log.info("Incremental sync is not safe from instant {}. Falling back to snapshot sync.",
      earliestInstant);
  return false;
}
return true;

Note that the instant it validates is the earliest of the last synced instant and any instants left pending from a previous run. That is the conservative and correct choice, and it is worth knowing about: pending instants widen the history window the source needs to retain, so clearing them keeps the window tight.

There are exactly two conditions under which XTable chooses a snapshot instead, and both announce themselves with a log line ending in “Falling back to snapshot sync”.

What “safe from this instant” means per format

isIncrementalSyncSafeFrom is implemented by each source, and each implementation maps onto a retention setting you already manage, which makes the requirement concrete.

Hudi requires the commit to still be on the timeline and to be untouched by cleaning:

public boolean isIncrementalSyncSafeFrom(Instant instant) {
  return doesCommitExistsAsOfInstant(instant) && !isAffectedByCleanupProcess(instant);
}

Both halves matter. Archival moves old instants off the active timeline, and cleaning removes old file slices, so a replayable commit needs both to have left it alone. Keeping hoodie.cleaner.commits.retained generous relative to your sync interval is all it takes to stay on the incremental path.

Iceberg walks the snapshot parent chain backwards from the current snapshot looking for one at or before the instant, and gives up if the chain runs out or is broken:

Snapshot parentSnapshot = iceTable.snapshot(parentSnapshotId);
if (parentSnapshot == null) {
  // chain is broken due to expired snapshot

expire_snapshots governs this. Whatever history.expire.max-snapshot-age-ms you have chosen, XTable needs the chain intact back to its last sync, so those two numbers are worth setting together.

Delta asks the history manager for the active commit at that time and checks it really is at or before the instant, because the API returns the earliest commit when you ask for something older than the table:

DeltaHistoryManager.Commit deltaCommitAtOrBeforeInstant =
    deltaLog.history().getActiveCommitAtTime(Timestamp.from(instant), true, false, true);
// There is a chance earliest commit of the table is returned if the instant is before the
// earliest commit of the table, hence the additional check.
Instant deltaCommitInstant = Instant.ofEpochMilli(deltaCommitAtOrBeforeInstant.getTimestamp());
return deltaCommitInstant.equals(instant) || deltaCommitInstant.isBefore(instant);

Log retention and VACUUM govern this one.

One rule covers all three: keep the source’s history retention comfortably longer than your sync interval. It is a single, checkable relationship between two numbers, and getting it right is what keeps every run incremental.

Running it

The RunSync utility takes a YAML dataset config and a small set of options. The config shape comes from RunSync.DatasetConfig:

# trips-sync.yaml
sourceFormat: HUDI
targetFormats:
  - ICEBERG
  - DELTA
datasets:
  - tableBasePath: s3://lakehouse-prod/warehouse/trips
    tableName: trips
    partitionSpec: city_id:VALUE
    namespace: analytics

partitionSpec is a comma-separated list of field:TRANSFORM entries, with an optional third :format part. The transform comes from PartitionTransformType, which at this release is YEAR, MONTH, DAY, HOUR, VALUE or BUCKET, so city_id:VALUE means “partitioned by the literal value of city_id” and event_date:DAY:yyyy-MM-dd would describe a day-partitioned table with an explicit date format.

sourceFormat can be auto-detected, but the field’s own documentation recommends setting it explicitly “for cases where the directory contains metadata of multiple formats”, which is precisely the situation XTable creates. Set it.

The utilities bundle is not published to Maven Central, so build it from the release tag. Only xtable-api, xtable-core_2.12, xtable-spark-runtime_2.12 and a few others are published; xtable-utilities is not among them.

git clone https://github.com/apache/incubator-xtable.git
cd incubator-xtable
git checkout 0.4.0-incubating
mvn clean package -DskipTests

java -jar xtable-utilities/target/xtable-utilities_2.12-0.4.0-incubating-bundled.jar \
  --datasetConfig trips-sync.yaml \
  --hadoopConfig /etc/hadoop/conf/core-site.xml \
  --icebergCatalogConfig iceberg-catalog.yaml

The full option set at 0.4.0-incubating:

--continuousMode is the option to reach for. It reloads the config file on every iteration, so tables can be added to or removed from a running job without a restart, and it keeps the sync interval short, which is the most effective way to stay comfortably inside your source’s retention window.

Confirming you are on the incremental path

XTable tells you which path it took, which makes this easy to monitor. A snapshot sync is announced with an INFO line, and there are only two of them to know:

One command confirms a healthy job:

grep -c "Falling back to snapshot sync" xtable-sync.log

Zero on a steady-state job means every target is on the incremental path, which is exactly what you are aiming for. A count matching your number of target formats on the first run and zero afterwards is the normal, healthy pattern.

Because the decision is per target, the message also names the format, so you can tell a target that is still bootstrapping from a retention window worth widening.

Where XTable fits best

XTable is at its best when several readers need different metadata over the same files, and it solves that cleanly. A Hudi ingest pipeline serving an Iceberg based query engine, or a Databricks team reading Delta over data another team writes as Hudi, are exactly the shapes it was built for, and it handles them without a second copy of the data.

Two adjacent problems have better tools, and knowing the boundary makes XTable more useful rather than less.

If you have decided to move formats permanently, a one-off migration is simpler than a sync you run forever, since it leaves you with one metadata tree to maintain instead of two or three.

If writers on both sides need to write, keep the source format authoritative. XTable’s targets are derived metadata, rebuilt from the source, so the clean pattern is one writer on the source and readers everywhere else.

And if a single engine needs a single format it does not read natively, check for a connector first. When one exists it is less machinery than a sync job with its own schedule.

Production tips

• Set sourceFormat explicitly. After the first sync the directory holds metadata for several formats, so naming the source removes any ambiguity.
• Make source retention exceed the sync interval with margin. For Hudi that is the cleaner and archival configs, for Iceberg expire_snapshots, for Delta log retention and VACUUM.
• Alert on “Falling back to snapshot sync” rather than on job duration. It is the leading indicator, and it is a single string to match.
• Prefer --continuousMode with a short interval over an external scheduler with a long one, both for the shorter history window and the config reload.
• Add all target formats at once where you can, so the one-off bootstrap snapshot happens once for every target rather than on separate days.
• Keep pending instants clear. The safety check uses the earliest of the last synced and pending instants, so an empty pending list keeps the required history window as tight as possible.

Conclusion

The useful mental model for XTable is that INCREMENTAL is a request rather than a switch. You ask for it in config, ConversionController checks per target format whether the source can still support it, and when it cannot it produces a correct result the slower way. That is the right trade to make by default: you never get a subtly wrong target table, and the conservative path is always available as a fallback.

What makes this worth understanding rather than just monitoring is that the one thing standing between you and permanently cheap syncs lives outside XTable. Incremental sync works while the source still holds history back to your last sync, which puts the Hudi cleaner config, the Iceberg snapshot expiry and the Delta vacuum schedule squarely in your control. They are ordinary settings you already own, and aligning them with your sync interval is a one-time conversation with whoever runs the source pipeline.

So two numbers are worth writing down for any XTable deployment: the source’s effective history retention, and the sync interval. Keep the first comfortably larger than the second and every run takes the incremental path. Add one alert on “Falling back to snapshot sync” and you will know immediately if that ever changes.

References

• Apache XTable documentation for setup and the dataset config
• ConversionController.java at 0.4.0-incubating, where the per-target FULL versus INCREMENTAL decision is made
• SyncMode.java at 0.4.0-incubating, the two modes and their definitions
• RunSync.java at 0.4.0-incubating for the CLI options and the dataset config schema
• Hudi indexing documentation, if the source side of your sync is a Hudi table you are still tuning

TL;DR

• Nearly every JVM-level Spark change lands in one of two configs: spark.driver.extraJavaOptions or spark.executor.extraJavaOptions. Learn where each applies and the rest is detail.
• Spark 3.3.0 moved to Log4j 2, so the file is log4j2.properties and the flag is -Dlog4j.configurationFile=. Start from the template Spark ships and your config is picked up first time.
• -verbose:class turns a NoClassDefFoundError into a direct answer, because it prints which JAR each class actually came from.
• A java.lang.StackOverflowError in Spark usually points at a deep query plan rather than a bug in your loop. Raising -Xss on the side that threw resolves most of them.
• In client mode the driver JVM is already running by the time your SparkConf executes, so pass driver JVM options with --driver-java-options or the properties file.

Why these four belong together

I spent four and a half years as Cloudera’s Spark backline engineer and now do similar work on Apache Hudi. The same four techniques come up again and again, and each one is a small investment that pays off repeatedly: getting exactly the logs you want, finding out which JAR a class came from, giving a deep query plan the stack it needs, and getting the JVM through a corporate proxy.

Those four tasks look unrelated, and they are really one task. All four are JVM flags delivered through the same pair of Spark configs, so the thing worth learning is where a flag goes and when. Learn that once and every future JVM setting is a lookup in the JVM documentation.

This post is written against Spark 4.2.0 (Java 17, Scala 2.13.18, Hadoop 3.5.0 per its pom.xml). Everything except the logging section applies unchanged back to Spark 2.x. The logging section has a hard version boundary at 3.3.0, called out below.

Prerequisites: the Spark configuration reference and, if you are on YARN, Running Spark on YARN.

Architecture: how a JVM flag reaches the driver and the executors

Spark runs your code in two kinds of JVM. The driver builds the plan and schedules work. The executors run the tasks. A JVM flag affects exactly one of them, so the first question on any of these problems is which JVM is failing.

Two rules worth committing to memory:

Deploy mode changes where the driver lives. In client mode the driver runs in the spark-submit process on the machine you typed the command on. In cluster mode it runs inside the cluster, as the YARN ApplicationMaster or a Kubernetes driver pod. Any file the driver needs must be shipped there.

In client mode the driver JVM has already started. Spark’s own documentation for spark.driver.extraJavaOptions is explicit: in client mode the config “must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point,” and you should use --driver-java-options or the properties file instead. Pass it on the command line and it applies cleanly.

One more: it is illegal to set the maximum heap size (-Xmx) through extraJavaOptions. Use spark.driver.memory and spark.executor.memory, or --driver-memory and --executor-memory. -Xss, the flag in the stack-size section below, is a different flag and is perfectly legal here.

Custom logging, and the Log4j 2 boundary

By default Spark reads its logging config from $SPARK_HOME/conf, which is set at the cluster level. What you usually want while investigating is DEBUG on two specific packages for one application, leaving the cluster default and everyone else’s jobs untouched. Spark supports exactly that.

The Log4j 2 boundary at Spark 3.3.0

Spark used Log4j 1.x up to and including 3.2.x, then switched to Log4j 2. You can see the switch in the shipped templates:

The practical consequence is worth knowing. Log4j 2 ignores the older -Dlog4j.configuration property rather than raising an error, and falls back to its normal configuration lookup, so an application keeps running and logs at the cluster default. If a custom config ever seems to have no effect, checking the file name and the flag against the table above is the quickest thing to try, and a snippet saved before 2022 is worth refreshing against it.

Spark 4.2.0 also ships log4j2-json-layout.properties.template and a spark.log.structuredLogging.enabled config (default false) if you want JSON logs rather than the pattern layout.

Write the Log4j 2 config

Start from the shipped template rather than from memory, because the property names are not guessable:

cp /opt/spark/conf/log4j2.properties.template /tmp/log4j2-debug.properties

Then set the levels you actually want. This raises the root logger to DEBUG, turns Spark’s SQL execution and Hudi client packages up, and keeps the noisy third-party loggers down so the output stays readable:

rootLogger.level = debug
rootLogger.appenderRef.stdout.ref = console

appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
# %ex rather than the implicit %xEx: the extended form resolves the JAR each
# stack frame came from, which is measurable overhead on a hot error path.
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex

# The two packages under investigation.
logger.sqlexec.name = org.apache.spark.sql.execution
logger.sqlexec.level = debug

logger.hudiclient.name = org.apache.hudi.client
logger.hudiclient.level = debug

# Third-party loggers that make DEBUG unusable if left at the root level.
logger.jetty.name = org.sparkproject.jetty
logger.jetty.level = warn

logger.parquet.name = org.apache.parquet
logger.parquet.level = error

logger.hmshandler.name = org.apache.hadoop.hive.metastore.RetryingHMSHandler
logger.hmshandler.level = fatal

Ship it to the driver and the executors

The file has to exist on every JVM that reads it. --files uploads it and places it in each container’s working directory, which is why the flag value on the executor side is a bare filename with no path.

Cluster mode, where the driver is also a container:

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --files /tmp/log4j2-debug.properties \
  --conf spark.driver.extraJavaOptions="-Dlog4j.configurationFile=log4j2-debug.properties" \
  --conf spark.executor.extraJavaOptions="-Dlog4j.configurationFile=log4j2-debug.properties" \
  --class com.rangareddy.pipeline.TripsIngest \
  s3a://lakehouse-prod/artifacts/trips-ingest-2.4.1.jar \
  --input s3a://lakehouse-prod/raw/trips/ \
  --table s3a://lakehouse-prod/warehouse/trips/

Client mode, where the driver reads the file straight off local disk and only the executors need the upload:

spark-submit \
  --master yarn \
  --deploy-mode client \
  --files /tmp/log4j2-debug.properties \
  --driver-java-options "-Dlog4j.configurationFile=/tmp/log4j2-debug.properties" \
  --conf spark.executor.extraJavaOptions="-Dlog4j.configurationFile=log4j2-debug.properties" \
  --class com.rangareddy.pipeline.TripsIngest \
  s3a://lakehouse-prod/artifacts/trips-ingest-2.4.1.jar \
  --input s3a://lakehouse-prod/raw/trips/ \
  --table s3a://lakehouse-prod/warehouse/trips/

Note the asymmetry, which is the one detail to get right: in client mode the driver flag carries an absolute local path, while the executor flag carries a bare filename resolved inside the container. Keep those two straight and the config lands on both sides first time.

Different levels on driver and executors

Sometimes you want DEBUG on the driver, where planning happens, and WARN on two thousand executors, because DEBUG across all of them will fill the disk. Ship two files and point each side at its own:

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --files /tmp/log4j2-driver.properties,/tmp/log4j2-executor.properties \
  --conf spark.driver.extraJavaOptions="-Dlog4j.configurationFile=log4j2-driver.properties" \
  --conf spark.executor.extraJavaOptions="-Dlog4j.configurationFile=log4j2-executor.properties" \
  --class com.rangareddy.pipeline.TripsIngest \
  s3a://lakehouse-prod/artifacts/trips-ingest-2.4.1.jar

On YARN, if you switch to a file appender, write it under YARN’s own log directory so log aggregation still collects it and a long-running streaming job does not fill the local disk:

appender.file_appender.fileName = ${sys:spark.yarn.app.container.log.dir}/spark.log

Tracing class loading with -verbose:class

ClassNotFoundException tells you a class was absent. NoClassDefFoundError tells you something more specific: the class was present at compile time, and at runtime either it is missing or a different version of it loaded first. On a cluster with Hadoop, Hive, Spark and connector JARs all on the classpath, that second case is the common one, and it is very answerable.

-verbose:class is a JVM flag, not a Spark one. It makes the JVM print every class it loads and the source it loaded from, which turns the guess into a lookup:

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --conf spark.driver.extraJavaOptions="-verbose:class" \
  --conf spark.executor.extraJavaOptions="-verbose:class" \
  --class com.rangareddy.pipeline.TripsIngest \
  s3a://lakehouse-prod/artifacts/trips-ingest-2.4.1.jar

Then search the container log for the class in the stack trace. The path after source: is the JAR that won. Once you can name that JAR the fix follows directly: shade the dependency, set spark.driver.userClassPathFirst, or remove the duplicate.

Two similarly named things, both useful:

• --verbose is a spark-submit flag. It prints the resolved Spark configuration, the classpath and the parsed arguments before launch. Use it on every escalation; it costs nothing and answers “was my config even applied?”.
• -verbose:class is a JVM flag passed through extraJavaOptions. It prints class-loader activity for the life of the JVM.

-verbose:class buys you the loader’s ground truth at the cost of a very large log. Turn it on for one reproduction, take the answer, turn it off. On a busy executor it can add hundreds of megabytes, which is a fine trade for one run.

java.lang.StackOverflowError

The failure looks like this, on either the driver or an executor:

java.lang.StackOverflowError
  at scala.collection.immutable.List.foreach(List.scala:431)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:...)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:...)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:...)

A wall of repeating frames from TreeNode, Catalyst, or an Avro or Parquet schema walker is the signature, and it is good news: it usually means your code is fine. What it describes is a recursive walk over a structure deeper than the thread’s stack, from hundreds of columns, a deeply nested struct, a plan built by chaining union or withColumn in a loop, or a long chain of predicates.

Find the side that threw it first. A stack trace in the driver log means the driver’s plan walk overflowed, and the knob is the driver’s. A trace in an executor log means the knob is the executor’s. If the logs do not make it obvious, set both:

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --conf spark.driver.extraJavaOptions="-Xss4m" \
  --conf spark.executor.extraJavaOptions="-Xss4m" \
  --class com.rangareddy.pipeline.TripsIngest \
  s3a://lakehouse-prod/artifacts/trips-ingest-2.4.1.jar

-Xss sets the stack size per thread, so budget for it: an executor with many task threads pays the increase on each one, out of off-heap memory that is not counted against spark.executor.memory. Raise spark.executor.memoryOverhead alongside it and YARN stays happy.

Escalate in steps: 4m, 8m, 16m, 32m. Most plans are comfortable well before the top of that range. If you find yourself needing hundreds of megabytes of stack, the plan itself is the better thing to simplify, and there are three clean ways to do it: collapse a union chain into one unionByName over a collected sequence, checkpoint the DataFrame to truncate its lineage, or flatten the nested schema.

Routing Spark through an HTTP proxy

When executors have to reach an external endpoint, a schema registry, a cloud metadata service, a REST catalog, and the network only allows it through a proxy, the proxy settings are standard JVM system properties. Spark has no config of its own for them, which is why they go through extraJavaOptions.

Per application:

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --conf spark.driver.extraJavaOptions="-Dhttp.proxyHost=proxy.corp.internal -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.corp.internal -Dhttps.proxyPort=8443 -Dhttp.nonProxyHosts=localhost|127.0.0.1|*.corp.internal" \
  --conf spark.executor.extraJavaOptions="-Dhttp.proxyHost=proxy.corp.internal -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.corp.internal -Dhttps.proxyPort=8443 -Dhttp.nonProxyHosts=localhost|127.0.0.1|*.corp.internal" \
  --class com.rangareddy.pipeline.TripsIngest \
  s3a://lakehouse-prod/artifacts/trips-ingest-2.4.1.jar

http.nonProxyHosts is the part worth including from the start. It keeps in-cluster traffic off the proxy, so calls to the NameNode or a local metastore continue to go direct.

For every application on the cluster, put the same values in $SPARK_HOME/conf/spark-defaults.conf:

spark.driver.extraJavaOptions   -Dhttp.proxyHost=proxy.corp.internal -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.corp.internal -Dhttps.proxyPort=8443 -Dhttp.nonProxyHosts=localhost|127.0.0.1|*.corp.internal
spark.executor.extraJavaOptions -Dhttp.proxyHost=proxy.corp.internal -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.corp.internal -Dhttps.proxyPort=8443 -Dhttp.nonProxyHosts=localhost|127.0.0.1|*.corp.internal

Note that extraJavaOptions in spark-defaults.conf is a single string, and a --conf spark.driver.extraJavaOptions=... on the command line replaces it rather than appending. Spark has a purpose-built answer for this: spark.driver.defaultJavaOptions. Put cluster-wide settings such as the proxy there, leave extraJavaOptions free for users, and Spark prepends the defaults to whatever a user passes. Both layers then apply together.

Production tips

• Always pass --verbose. It costs nothing and confirms up front that the config you set is in the resolved configuration.
• Put cluster-wide JVM policy in defaultJavaOptions. Spark prepends it to whatever a user passes in extraJavaOptions, so both layers apply.
• Raise spark.executor.memoryOverhead when you raise -Xss. Thread stacks are off-heap and YARN counts them.
• Copy the shipped log4j2.properties.template instead of writing a Log4j 2 config from scratch or adapting a Log4j 1.x one.
• Keep the executor-side config filename bare (log4j2-debug.properties), and the client-mode driver-side path absolute.
• Scope DEBUG to the packages you care about rather than the root logger. On a large cluster this keeps the output readable and the disks healthy.
• Turn -verbose:class off once you have your answer. It is a diagnostic rather than a permanent setting.
• On YARN, send file appenders to ${sys:spark.yarn.app.container.log.dir} so aggregation picks them up.

Confirming each change took effect

Three of these four apply without printing anything, so it is worth knowing the one-second check for each. All of them are quick:

For the two logging rows the check is the same and takes one glance: if your custom pattern layout is on the log lines, the file was read. Running with --verbose first makes all five of these visible before the job even starts.

Where to look instead

These four are diagnostics and last-mile plumbing, and knowing their boundary saves time.

For a slow job rather than a broken one, the productive settings live in spark.sql.*: shuffle partition counts, join strategies, AQE and file sizing. The Spark UI’s SQL tab will point you at the right one faster than any JVM flag. For memory pressure, spark.executor.memory and spark.memory.fraction are the knobs, and Spark helpfully rejects -Xmx here so you cannot set it in the wrong place.

For logging you want on permanently, edit $SPARK_CONF_DIR/log4j2.properties and let Spark upload it for every job, rather than threading --files through each one.

Conclusion

These four techniques belong in one post because the delivery mechanism is the real lesson. There are two JVMs, they take flags through two configs, the deploy mode decides where the driver’s copy of a file lives, and in client mode the driver has already started by the time your application code runs. Once that model is in your head, any future JVM flag is a lookup in the JVM documentation and a one-line config change.

The Log4j 2 boundary is the one version detail worth remembering alongside it. Spark 3.3.0 moved to Log4j 2, so on any current cluster the file is log4j2.properties and the flag is -Dlog4j.configurationFile=. Guidance written for Spark 3.2 and earlier will say otherwise, so start from the template Spark ships in conf/ and you will get the logging you asked for on the first run.

Next, if you are sizing the JVMs rather than instrumenting them, the Spark Configuration Generator turns a node count, core count and memory-per-node into executor settings, and the Spark Submit Command Formatter will break a command like the ones above into an editable table.

References

• Spark configuration reference for extraJavaOptions, defaultJavaOptions and the client-mode caveat
• Running Spark on YARN for custom Log4j 2 configs and spark.yarn.app.container.log.dir
• The log4j2.properties.template shipped in Spark 4.2.0, the right starting point for a custom config
• Log4j 2 properties-file syntax for the appender and logger property names
• Java HTTP proxy system properties for http.proxyHost, https.proxyPort and nonProxyHosts

TL;DR

• Pick a Spark version and the tool offers only the Iceberg releases that actually support it, read from each release’s own build, then only the Scala versions Iceberg publishes a runtime for.
• Spark 4.x is Scala 2.13 only, so no _2.12 Iceberg runtime exists for it. The tool will not let you build that coordinate.
• It emits the --packages coordinate, the IcebergSparkSessionExtensions config, and the catalog configs for Hive, Hadoop, REST or JDBC.
• SparkSessionCatalog is used for Hive so Iceberg and existing Hive tables coexist under one catalog name; every other type gets SparkCatalog.

Spark Submit Command generator using different Iceberg Catalog(s)

This tool is used to generate or build the Spark Submit Command using Iceberg Catalog(s).

References

• Iceberg Spark getting started for the runtime coordinate and session extensions
• Iceberg Spark configuration for catalog properties and the SparkCatalog versus SparkSessionCatalog choice
• CatalogUtil.java at apache-iceberg-1.11.0, the accepted type values
• Iceberg gradle.properties at apache-iceberg-1.11.0, the Spark and Scala support matrix this tool encodes

TL;DR

• parquet-cli is where Parquet tooling is maintained now. parquet-tools was renamed to parquet-tools-deprecated in Parquet 1.12.0 and left the build in 1.12.3.
• The command set is broader rather than renamed. rowcount, size, dump and merge are covered by meta, column-size, pages and rewrite, and there are new commands for bloom filters, size statistics and geospatial statistics.
• rewrite is the one to learn first. A single command merges files, prunes columns, masks columns and changes the compression codec, and it supersedes prune.
• rewrite merges by copying row groups intact, which is what makes it fast. Two 10-row files become one file with two row groups of 10, so use your table format’s compaction when you want re-chunking.
• After --prune-columns, confirm the result with meta or column-size, which read the physical schema. The stored parquet.avro.schema metadata is copied through unchanged, so an Avro-model reader reports the pruned field as null.

Where Parquet tooling lives now

parquet-tools was the standard way to look inside a Parquet file for years, and parquet-cli has taken over that role. You can trace the handover by diffing the root pom.xml across releases:

The replacement, parquet-cli, has been in the build the whole time and is still there in 1.18.0. The repository itself was also renamed: apache/parquet-mr is now apache/parquet-java.

The old jar is still on Maven Central, so a parquet-tools-1.11.2.jar you downloaded in 2022 keeps working on older files. parquet-cli is where five years of fixes and every newer format feature landed, including size statistics, geospatial statistics and the Parquet variant type, so it is the better choice for files written by a current Spark, Hudi or Iceberg.

This post is written against Parquet 1.18.0 on Java 17. Every transcript below is real output from that version.

Getting parquet-cli

The published parquet-cli runtime jar deliberately does not bundle Hadoop, which keeps it small and lets it pick up whatever Hadoop and connector versions your cluster already has. There are two easy ways to run it.

On a cluster, where Hadoop is already there

This is the easy case, and the one you will use most. hadoop jar puts the whole Hadoop client classpath in front of you:

wget -O parquet-cli.jar \
  https://repo1.maven.org/maven2/org/apache/parquet/parquet-cli/1.18.0/parquet-cli-1.18.0-runtime.jar

hadoop jar parquet-cli.jar org.apache.parquet.cli.Main \
  meta hdfs:///warehouse/hr/employees/part-00000-8f3a1c92.parquet

The same jar reads cloud object storage, as long as the matching connector is on the classpath:

hadoop jar parquet-cli.jar org.apache.parquet.cli.Main \
  meta s3a://lakehouse-prod/warehouse/hr/employees/part-00000-8f3a1c92.parquet

On a laptop, with no Hadoop install

Fetch the runtime jar plus the shaded Hadoop client and the few libraries parquet-cli expects to find. Include an SLF4J binding: parquet-cli writes its output through SLF4J, so the binding is what makes the output appear. The script below sets up a working wrapper in one go.

mkdir -p ~/parquet-cli && cd ~/parquet-cli

BASE=https://repo1.maven.org/maven2
curl -sLO $BASE/org/apache/parquet/parquet-cli/1.18.0/parquet-cli-1.18.0-runtime.jar
curl -sLO $BASE/org/apache/hadoop/hadoop-client-api/3.4.1/hadoop-client-api-3.4.1.jar
curl -sLO $BASE/org/apache/hadoop/hadoop-client-runtime/3.4.1/hadoop-client-runtime-3.4.1.jar
curl -sLO $BASE/com/google/guava/guava/33.4.0-jre/guava-33.4.0-jre.jar
curl -sLO $BASE/com/google/guava/failureaccess/1.0.2/failureaccess-1.0.2.jar
curl -sLO $BASE/commons-logging/commons-logging/1.3.5/commons-logging-1.3.5.jar
curl -sLO $BASE/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
# Without an SLF4J binding, parquet-cli runs and prints nothing.
curl -sLO $BASE/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar
curl -sLO $BASE/ch/qos/reload4j/reload4j/1.2.25/reload4j-1.2.25.jar

cat > log4j.properties <<'PROPS'
log4j.rootLogger=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%m%n
PROPS

cat > parquet <<'SH'
#!/usr/bin/env bash
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
exec java -cp "$DIR/*:$DIR" org.apache.parquet.cli.Main "$@"
SH
chmod +x parquet

Every example from here on uses that parquet wrapper.

Building a file to inspect

parquet-cli can create a Parquet file from CSV, which is convenient for reproducing a problem. Supply an explicit Avro schema so the column types are what you meant rather than what got inferred:

cat > employees.csv <<'CSV'
employee_id,first_name,last_name,email,phone_number,hire_date,salary,manager_id
1,Ranga,Reddy,rangareddy@yahoo.com,99509833,2007-06-21,2600,
2,Raja Sekhar,Reddy,raja@gmail.com,75050798,2008-01-13,2600,1
3,Vasundra,Reddy,vasu@gmail.com,91512344,2003-09-17,4400,1
4,Meena,P,meena@test.com,81535555,2004-02-17,13000,2
5,Manoj,Kumar,manu@rediff.com,60312366,2005-08-17,6000,3
6,Vinod,Kumar,vinod@zoho.com,71237777,2002-06-07,6500,3
7,Raja,Reddy,rajar@yahoo.co.in,91518888,2002-06-07,10000,4
8,Shiva,P,shiva@mymail.com,81512380,2002-06-07,12008,6
9,Reddy,Babu,babu@mail.com,91528181,2002-06-07,8300,7
10,Nishanth,Reddy,nish@nish.com,61512347,2003-06-17,24000,2
CSV

cat > employees.avsc <<'AVSC'
{
  "type": "record",
  "name": "employees",
  "namespace": "com.rangareddy.hr",
  "fields": [
    {"name": "employee_id",  "type": "int"},
    {"name": "first_name",   "type": "string"},
    {"name": "last_name",    "type": "string"},
    {"name": "email",        "type": "string"},
    {"name": "phone_number", "type": "long"},
    {"name": "hire_date",    "type": "string"},
    {"name": "salary",       "type": "int"},
    {"name": "manager_id",   "type": ["null", "int"], "default": null}
  ]
}
AVSC

./parquet convert-csv employees.csv -s employees.avsc -o employees.parquet --overwrite

One note on schemas: the CSV reader takes physical Avro types rather than logical ones, so keep dates as string in the .avsc when the source is CSV, or convert from Avro or JSON when you want date and decimal logical types.

The command map

This is the table to keep. The left column is what you typed with parquet-tools; the right column is what does that job now, and the bottom rows are capabilities the old tool never had.

The full list is registered in Main.java, which is the authoritative answer to “does this version have that command”.

Reading data

cat prints records as JSON, one per line. head is the same command bound to a 10-record limit, and -n overrides it:

./parquet cat -n 2 employees.parquet

{"employee_id": 1, "first_name": "Ranga", "last_name": "Reddy", "email": "rangareddy@yahoo.com", "phone_number": 99509833, "hire_date": "2007-06-21", "salary": 2600, "manager_id": null}
{"employee_id": 2, "first_name": "Raja Sekhar", "last_name": "Reddy", "email": "raja@gmail.com", "phone_number": 75050798, "hire_date": "2008-01-13", "salary": 2600, "manager_id": 1}

Unlike parquet-tools cat, there is no separate --json flag, because JSON is the only output format.

Reading the schema

schema gives you the Avro view, which is what most downstream readers see:

./parquet schema employees.parquet

{
  "type" : "record",
  "name" : "employees",
  "namespace" : "com.rangareddy.hr",
  "fields" : [ {
    "name" : "employee_id",
    "type" : "int"
  }, {
    "name" : "first_name",
    "type" : "string"
  }, {
    "name" : "manager_id",
    "type" : [ "null", "int" ],
    "default" : null
  } ]
}

The Parquet message type is a different thing, and when you are debugging a type-mismatch it is the one you want, because it shows physical types, required versus optional, and the logical-type annotations. It comes out of meta, covered next.

meta: the best first command

meta replaces three old commands at once. It reports the writer, the key-value metadata, the Parquet message type, and per-column statistics per row group:

./parquet meta employees.parquet

File path:  employees.parquet
Created by: parquet-mr version 1.18.0 (build 0209dfb48d153f2f3e49a9f12addebe15c0f1d77)
Properties:
  parquet.avro.schema: {"type":"record","name":"employees","namespace":"com.rangareddy.hr","fields":[...]}
    writer.model.name: avro
Schema:
message com.rangareddy.hr.employees {
  required int32 employee_id;
  required binary first_name (STRING);
  required binary last_name (STRING);
  required binary email (STRING);
  required int64 phone_number;
  required binary hire_date (STRING);
  required int32 salary;
  optional int32 manager_id;
}

Row group 0:  count: 10  89.50 B records  start: 4  total(compressed): 895 B total(uncompressed):882 B
--------------------------------------------------------------------------------
              type      encodings count     avg size   nulls   min / max
employee_id   INT32     G   _     10        6.70 B     0       "1" / "10"
first_name    BINARY    G   _     10        12.10 B    0       "Manoj" / "Vinod"
last_name     BINARY    G _ R     10        11.50 B    0       "Babu" / "Reddy"
email         BINARY    G   _     10        15.70 B    0       "babu@mail.com" / "vinod@zoho.com"
phone_number  INT64     G   _     10        9.60 B     0       "60312366" / "99509833"
hire_date     BINARY    G _ R     10        14.40 B    0       "2002-06-07" / "2008-01-13"
salary        INT32     G   _     10        8.20 B     0       "2600" / "24000"
manager_id    INT32     G _ R     10        11.30 B    1       "1" / "7"

Four things to read off this, in the order they usually matter:

Created by identifies the writer. On a mixed cluster this is how you find out that half your files came from an old engine. Note that the string still says parquet-mr even in 1.18.0, despite the repository rename.

count: per row group is the row count, so meta is also rowcount. Add them up across row groups for the file total.

The min / max column is why predicate pushdown works or does not. If a filter on salary is not pruning row groups, look here first: statistics that are absent, or present but useless because the column is unsorted and every row group spans the full range, explain it immediately.

The nulls column is a fast data-quality check. manager_id showing 1 null across 10 rows matches the CEO having no manager.

The encodings column is compact: G is a dictionary-encoded column chunk, _ means no dictionary fallback, and R indicates RLE. Use pages when you need the per-page detail.

Architecture: row groups, column chunks and pages

Every command below reports on one level of the same three-level hierarchy, so it is worth naming the levels before reading their output.

A Parquet file is a sequence of row groups, each a horizontal slice of the rows. Within a row group, each column’s values for those rows live in one column chunk, which is the unit that gets its own encoding, compression and min/max statistics. A column chunk is in turn a sequence of pages, which is the smallest unit the reader decompresses.

That structure is what makes the tooling map onto commands the way it does: meta reports the file and its row groups, column-size aggregates column chunks, and pages opens a single column chunk. It is also why predicate pushdown is a row-group-level and page-level decision: the engine compares your filter against the statistics at those levels and skips whole chunks or pages it can prove irrelevant.

column-size answers “which column is my file”, which is the first question when a table is unexpectedly large:

./parquet column-size employees.parquet

manager_id-> Size In Bytes: 113 Size In Ratio: 0.12625699
employee_id-> Size In Bytes: 67 Size In Ratio: 0.074860334
last_name-> Size In Bytes: 115 Size In Ratio: 0.12849163
phone_number-> Size In Bytes: 96 Size In Ratio: 0.10726257
hire_date-> Size In Bytes: 144 Size In Ratio: 0.16089386
salary-> Size In Bytes: 82 Size In Ratio: 0.09162011
first_name-> Size In Bytes: 121 Size In Ratio: 0.13519552
email-> Size In Bytes: 157 Size In Ratio: 0.17541899

Note the output is unordered, so sort it yourself on a wide table. On a real fact table this is where you discover that one JSON blob column is 70% of your storage.

pages drops to page level within a column chunk, which is where you go when compression or encoding is the question:

./parquet pages employees.parquet -c salary

Column: salary
--------------------------------------------------------------------------------
  page   type  enc  count   avg size   size       rows     nulls   min / max
  0-0    data  G _  10      5.90 B     59 B

size-stats reports the Parquet size statistics added to the format more recently, including the unencoded byte size and the repetition and definition level histograms that let an engine estimate decode cost without reading the data. Columns with no entry simply have no size statistics written:

./parquet size-stats employees.parquet

File path: employees.parquet

Row group 0
--------------------------------------------------------------------------------
column         unencoded bytes rep level histogram   def level histogram
[employee_id]  -               -                     -
[first_name]   61 B            -                     -
[last_name]    41 B            -                     -
[email]        150 B           -                     -
[phone_number] -               -                     -
[hire_date]    100 B           -                     -
[salary]       -               -                     -
[manager_id]   -               -                     -

rewrite: merge, prune, mask, recompress

rewrite is the command worth real attention: it is the only one that writes data, and it absorbed several older tools into one interface. Its own help text is explicit that prune is deprecated and will be removed in 2.0.0 in favour of it.

Merging several files and switching the codec at the same time:

./parquet rewrite \
  -i employees.parquet,employees_b.parquet \
  -o employees_merged.parquet \
  -c ZSTD \
  --overwrite

How merging handles row groups

Here is the behaviour worth understanding before you rely on it. Reading the merged file back:

./parquet meta employees_merged.parquet

Row group 0:  count: 10  87.00 B records  start: 4    total(compressed): 870 B total(uncompressed):880 B
Row group 1:  count: 10  87.00 B records  start: 874  total(compressed): 870 B total(uncompressed):880 B

Two 10-row inputs became one file with two row groups of 10 rows rather than one row group of 20. rewrite copies row groups intact, which is exactly why it is fast: it moves column chunks without decoding them. That makes it ideal for changing a codec, dropping a column or consolidating a handful of files cheaply.

For re-chunking, reach for the tool built for it: your table format’s own compaction, such as Hudi clustering or Iceberg rewrite_data_files, or a rewrite through an engine. rewrite gives you a cheap physical merge; compaction gives you better row-group sizing.

Verifying a column prune

When you prune a column, it is worth knowing which command to verify with. Prune a column and nullify another:

./parquet rewrite \
  -i employees.parquet \
  -o employees_masked.parquet \
  --prune-columns phone_number \
  --mask-mode nullify --mask-columns manager_id \
  --overwrite

The Parquet message type is correctly pruned to seven columns:

message com.rangareddy.hr.employees {
  required int32 employee_id;
  required binary first_name (STRING);
  required binary last_name (STRING);
  required binary email (STRING);
  required binary hire_date (STRING);
  required int32 salary;
  optional int32 manager_id;
}

The parquet.avro.schema entry in the file’s key-value metadata still declares phone_number, because rewrite copies key-value metadata through untouched. An Avro-model reader follows that schema, finds no column, and returns null:

./parquet head -n 1 employees_masked.parquet

{"employee_id": 1, "first_name": "Ranga", "last_name": "Reddy", "email": "rangareddy@yahoo.com", "phone_number": null, "hire_date": "2007-06-21", "salary": 2600, "manager_id": null}

The prune worked: phone_number is physically gone and column-size lists seven columns. The record-model view simply reports the field from the stored Avro schema. So for a GDPR deletion, verify with meta or column-size, which read the physical Parquet schema, and you get a clear answer.

Masking optional columns

Nullify needs somewhere to put the null, so it applies to optional columns. On a required column it stops and tells you clearly:

java.io.IOException: Required column [email] cannot be nullified
	at org.apache.parquet.hadoop.rewrite.ParquetRewriter.processBlock(ParquetRewriter.java:533)

To scrub a non-nullable column, prune it instead, or rewrite the data through an engine that can change required to optional first.

Production tips

• Start with meta. Writer, row counts, statistics and the message type in one command answer most questions.
• Use meta for row counts, and add the per-row-group count: values. There is no rowcount.
• Sort column-size yourself. The output order is not by size.
• Check min / max in meta first when a filter is not pruning as much as you expect. It usually answers the question immediately.
• Use rewrite for cheap physical changes and your table format’s compaction for row-group sizing. rewrite preserves row-group boundaries.
• After --prune-columns, verify with meta or column-size. They read the physical schema, while cat and head follow the stored Avro schema.
• On a cluster, prefer hadoop jar so the Hadoop and connector classpath is already correct.
• Pin the version in your notes. Main.java at the release tag is the only reliable list of which commands your jar has.

Where parquet-cli fits best

parquet-cli is the right tool whenever the question is about one file: is the statistic there, what wrote it, why is this column so large, what encodings did it choose. For that it is unbeatable, and it needs nothing but the file.

Table-level questions have their own tooling, and pairing the two is the productive combination. A file under a Hudi, Iceberg or Delta table path is one version of one file group, so the table format is what knows whether it is live, which delete files or deletion vectors apply, what is pending compaction and how the schema has evolved. Use Hudi’s CLI and metadata table or Iceberg’s metadata tables and CALL procedures for those, then drop to parquet-cli for the file in front of you.

Conclusion

The move from parquet-tools to parquet-cli is often described as a rename, and it is better than that. The command set genuinely grew: dump became four focused commands, rewrite took over from both merge and prune, and there are new commands for bloom filters, size statistics and geospatial statistics that the old tool never had. Two commands carry most of the daily work. meta answers the writer, row-count, schema and statistics questions in one shot, and rewrite handles every physical change to a file.

The two rewrite behaviours in this post are the ones worth carrying forward, and both follow from a single sensible design decision. Because rewrite copies column chunks rather than decoding them, it is fast, it preserves row-group boundaries, and it passes key-value metadata through untouched. That tells you where it fits: excellent for recompressing, pruning and consolidating, and paired with your table format’s compaction when you want row groups resized. It also tells you which command verifies a prune, since meta and column-size read the physical schema.

If you arrived here searching for parquet-tools, the old jar still runs and there is no urgency. When you do switch, the command map above is the whole migration, and you get the newer format features for free.

References

• Apache Parquet Java on GitHub, formerly apache/parquet-mr
• Main.java at the 1.18.0 tag, the registered command list for the release you are running
• The 1.12.3 root pom.xml, the release where the parquet-tools module disappears
• parquet-cli on Maven Central for the runtime jar
• Parquet file format specification for row groups, column chunks, pages and statistics

TL;DR

• Paste a spark-submit command written on one line and get it back split across lines with a trailing backslash per option, or minified back to one line.
• Each Spark option is resolved to the configuration property it actually sets, so --num-executors shows as spark.executor.instances and you can compare a command against spark-defaults.conf directly.
• Your application’s own arguments are kept separate from Spark’s, so a job that takes --input and --output no longer has them mistaken for Spark options.
• Quoted values survive the round trip intact, including an extraJavaOptions string containing spaces.

Spark Submit Command Formatter/Minifier

Format a spark-submit command across lines for review, or minify it back onto one line for a scheduler that wants a single string. Both directions preserve the command exactly.

The useful part is the breakdown underneath. A long spark-submit line hides duplicated or contradictory settings, and the same setting can arrive as either a flag or a --conf, which makes them hard to compare by eye. The parameter table resolves every option to its configuration property, so --executor-memory 18g and --conf spark.executor.memory=18g line up as the same row.

The option-to-property mapping follows the OptionAssigner list in SparkSubmit.scala and the flag names in SparkSubmitOptionParser.java at the Spark v4.2.0 tag, so --principal and --keytab resolve to spark.kerberos.principal and spark.kerberos.keytab rather than the pre-Spark-3 spark.yarn.* names.

References

• Submitting applications for the spark-submit argument reference
• SparkSubmit.scala at v4.2.0, the OptionAssigner list this tool’s flag-to-property mapping follows
• Spark configuration reference for what each --conf key means and its precedence
• Spark Configuration Generator to work out the executor sizes before formatting the command
• Spark JVM troubleshooting playbook for the extraJavaOptions gotchas a long command tends to hide

TL;DR

• Enter total nodes, cores per node and memory per node; the tool returns executor count, executor cores, executor memory and memory overhead.
• It offers three shapes: Tiny (one core per executor), Fat (one executor per node using every core) and Balanced (five cores per executor).
• The Balanced calculation reserves one core per node for the OS and node manager, subtracts one executor for the YARN ApplicationMaster, and sets spark.executor.memoryOverhead to 10% of the per-executor memory.
• Five cores per executor is a convention, not a derived number. Treat every output as a starting point and confirm it against your own stage metrics in the Spark UI.

Spark Configuration Generator

Spark Configuration Generator tool will generate the spark configuration based on hardware configuration.

References

• Spark configuration reference for the executor memory and core properties this tool emits
• Running Spark on YARN for the ApplicationMaster overhead and container sizing rules
• Spark tuning guide for memory management and the data-locality trade-offs behind the defaults
• Spark JVM troubleshooting playbook for the JVM flags that go alongside these sizes

TL;DR

• Kerberos is the authentication standard across the Hadoop ecosystem, so setting up a KDC is a prerequisite for securing HDFS, YARN, Hive and Spark.
• Install krb5-workstation on every node and krb5-server on one, then configure kdc.conf (KDC behaviour) and krb5.conf (client behaviour) to agree on the realm.
• kdb5_util create -s builds the principal database; the ACL file decides who may administer it; kadmin.local creates the first admin.
• Services authenticate with keytabs rather than passwords, which is why the last step is exporting one and testing kinit -kt from a client machine.

Kerberos

Kerberos is a secure authentication method developed by MIT that allows two services located in a non-secured network to authenticate themselves in a secure way. Kerberos, which is based on a ticketing system, serves as both Authentication Server and as Ticket Granting Server (TGS).

Kerberos has become the standard authentication method within the Hadoop ecosystem. For this reason, most Big Data technologies have adopted it as their authentication method.

Install & Configure Kerberos Server & Client in Linux

Let’s see how we can install, setup and configure Kerberos in a Cluster.

We will install Kerberos Server in one machine and Kerberos client in rest of the machines.

Step 1: Install Kerberos Client

We need to install the Kerberos client on every node in the cluster.

sudo dnf install krb5-workstation krb5-libs

On RHEL 7 and older, yum replaces dnf. On Debian and Ubuntu the client package is krb5-user instead.

Step2: Install Kerberos Server

The Kerberos server usually goes on the master node, though that is a convention rather than a rule; any server in the cluster will do.

sudo dnf install krb5-server

On Debian and Ubuntu the equivalents are krb5-kdc and krb5-admin-server, and their configuration lives in a different directory, so follow the Ubuntu server guide linked at the end for those paths.

Step 3: Configure Kerberos

The configuration lives in two files, one for the KDC and one for clients.

3.1 kdc.conf changes

Login Kerberos Server Installed machine

$ vi /var/kerberos/krb5kdc/kdc.conf

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 EXAMPLE.COM = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }
 EXAMPLE.COM = {
   renew_lifetime = 7d
 }

In the above kdc.conf file we choosed realm is EXAMPLE.COM

3.2 krb5.conf changes

$ vi /etc/krb5.conf

{
    [logging]
        default = FILE:/var/log/krb5libs.log
        kdc = FILE:/var/log/krb5libs.log
        admin_server = FILE:/var/log/kadmind.log

    [libdefaults]
        default_realm = EXAMPLE.COM
        dns_lookup_kdc = false
        dns_lookup_realm = false
        ticket_lifetime = 24h
        renew_lifetime = 7d
        forwardable = true
        default_tgs_enctypes = aes256-cts aes128-cts des3-hmac-sha1 des-hmac-sha1 des-cbc-crc
        default_tkt_enctypes = aes256-cts aes128-cts des3-hmac-sha1 des-hmac-sha1 des-cbc-crc
        permitted_enctypes = aes256-cts aes128-cts des3-hmac-sha1 des-hmac-sha1 des-cbc-crc
        udp_preference_limit = 1
        kdc_timeout = 3000
    [realms]
        EXAMPLE.COM = {
            kdc = node1.example.com
            admin_server = node1.example.com
        }
    [domain_realm]
}

Step 4: Create Kerberos KDC Database

In this step , we will create a KDC – Key Distribution Centre database. This database is used by the Kerberos server. So it is a crucial point in our installation steps.

$ kdb5 util create -r EXAMPLE.COM -s

It will ask for setting up a Master Password . Follow as asked and note down the password. This password is needed for any KDC database related activities like restart or any DB changes later etc.

Step 5: ACL changes

$ vi /var/kerberos/krb5kdc/kadm5.acl

Modify with your Realm name. In our case as , it is –

*/admin@EXAMPLE.COM *

Step 6: Add Admin for KDC

Note this Step MUST BE Executed only in the KDC Server machine – NOT in any Kerberos client machines.

$ kadmin.local

This will bring you to kadmin.local prompt. In that prompt, use the highlighted command. Note you have to use your own Realm name.

kadmin.local: addprinc   root/admin@EXAMPLE.COM

To see list of all principals created –

kadmin.local : listprincs

Step 7: Restart the Kerberos Admin & KDC Server

Note these steps MUST be done in KDC Server machine.

Start both services and enable them so the KDC comes back after a reboot:

sudo systemctl enable --now krb5kdc
sudo systemctl enable --now kadmin

Confirm both came up before moving on:

systemctl status krb5kdc kadmin

We are done with the Setup. We will test it from Kerberos as well Client servers.

Testing 1: Test Kerberos from Server

Test the Kerberos installation. Use the below command –

Check if any Ticket exists

$ klist

If no tickets exist in the cache, create a new one

$ kinit root/admin

Check again if you have any ticket

$ klist

Hopefully now you can see tickets listed here.

If you want to destroy any ticket, use

$ kdestroy

Testing 2 : Test Kerberos from Client machine

In previous step, we tested Kerberos from Kerberos server itself.

In this step, we will test Kerberos from the client machine. This step is important because in most cases you will use the client machines as a user. And if the user tries to access any services in the network , it will need Kerberos authentication. So let’s try this .

1. Create a non-admin user

So , we will use a non-admin user . Use below commands in KDC server to create a normal user (i.e. user with no admin access).

$ kadmin.local

In kadmin.local prompt use –

kadmin.local: addprinc rangareddy@EXAMPLE.COM

so we have created a normal user rangareddy.

2. Create a keytab file for the user

We will create a keytab file for the user rangareddy

$ kadmin.local

In kadmin.local prompt, use below

kadmin.local: xst -norandkey -k /tmp/rangareddy.keytab rangareddy@EXAMPLE.COM

It will create a keytab file rangareddy.keytab in /tmp directory for the rangareddy.

3. Test Kerberos from client machine

In previous step, we created the rangareddy.keytab file in KDC SERVER machine.

Copy the keytab file to the client machine.

Lets place it in /root/rangareddy.keytab in client machine.

Now in the client machine, open command prompt

Create a kerberos ticket

$ kinit -kt /root/rangareddy.keytab rangareddy@EXAMPLE.COM

Check if ticket created

$ klist

References

• MIT Kerberos documentation for kdc.conf, krb5.conf and kadmin reference
• Kerberos V5 System Administrator’s Guide for realm and database administration
• Hadoop in secure mode for how Hadoop services consume the keytabs created here
• Spark security for delegation tokens and --principal / --keytab on a Kerberized cluster
• Ubuntu server Kerberos guide for the Debian and Ubuntu package names and configuration paths