Spark Submit Command generator using Iceberg Catalog
Pick a catalog type (Hive, Hadoop, REST or JDBC) plus Spark, Iceberg and Scala versions, and get the spark-shell command with the right iceberg-spark-runtime coordinates and catalog configs. Version pairs come from each Iceberg release's own build, up to Iceberg 1.11.0 on Spark 4.1.
TL;DR
- Pick a Spark version and the tool offers only the Iceberg releases that actually support it, read from each release’s own build, then only the Scala versions Iceberg publishes a runtime for.
- Spark 4.x is Scala 2.13 only, so no
_2.12Iceberg runtime exists for it. The tool will not let you build that coordinate.- It emits the
--packagescoordinate, theIcebergSparkSessionExtensionsconfig, and the catalog configs for Hive, Hadoop, REST or JDBC.SparkSessionCatalogis used for Hive so Iceberg and existing Hive tables coexist under one catalog name; every other type getsSparkCatalog.
Spark Submit Command generator using different Iceberg Catalog(s)
This tool is used to generate or build the Spark Submit Command using Iceberg Catalog(s).
References
- Iceberg Spark getting started for the runtime coordinate and session extensions
- Iceberg Spark configuration for catalog properties and the
SparkCatalogversusSparkSessionCatalogchoice CatalogUtil.javaat apache-iceberg-1.11.0, the acceptedtypevalues- Iceberg
gradle.propertiesat apache-iceberg-1.11.0, the Spark and Scala support matrix this tool encodes
Found this useful?
These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.