All posts

Spark Submit Command generator using Iceberg Catalog

Pick a catalog type (Hive, Hadoop, REST or JDBC) plus Spark, Iceberg and Scala versions, and get the spark-shell command with the right iceberg-spark-runtime coordinates and catalog configs. Every version pair comes from the Iceberg release's own build rather than a hand-kept table.

Interactive tool Spark

TL;DR

  • Pick a Spark version and the tool offers only the Iceberg releases that actually support it, then only the Scala versions Iceberg publishes a runtime for.
  • Spark 4.x is Scala 2.13 only, so no _2.12 Iceberg runtime exists for it. The tool will not let you build that coordinate.
  • It emits the --packages coordinate, the IcebergSparkSessionExtensions config, and the catalog configs for Hive, Hadoop, REST or JDBC.
  • SparkSessionCatalog is used for Hive so Iceberg and existing Hive tables coexist under one catalog name; every other type gets SparkCatalog.

Spark Submit Command generator using different Iceberg Catalog(s)

This tool is used to generate or build the Spark Submit Command using Iceberg Catalog(s).

Spark Submit Command generator using Iceberg Catalog

Spark Submit Command

References

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee