Apache Hudi on Spark: the complete cheat sheet

• 1. Introduction, key features and architecture
• 2. Getting started: shell and CRUD
• 3. Table types: Copy-on-Write and Merge-on-Read
• 4. File layout and internal columns
• 5. Critical writer properties
• 6. The timeline
• 7. Supported data types
• 8. Write operations
• 9. Key generators
• 10. Query types
• 11. Indexes
• 12. The metadata table
• 13. Concurrency control
• 14. Schema evolution
• 15. Table services
• 16. Bootstrapping an existing dataset
• 17. SQL procedures
• 18. Savepoints and restore
• 19. Catalog integration
• 20. Writer interfaces
• 21. Hudi Streamer
• 22. Tuning
• 23. Diagnosing a table
• 24. Best practices
• Conclusion
• References

TL;DR

• This is a reference, not a tutorial. Every section is a lookup: the config key, its default, and the one sentence that tells you whether you want it.
• The three properties frozen at table creation are the table type, the record key and the key generator. Everything else a later job can change, so get those three right first.
• Two merge-mode keys exist and they are not duplicates. hoodie.record.merge.mode is the table property; hoodie.write.record.merge.mode is the writer-side option. Both arrived in 1.0.0.
• Hudi layers four concurrency controls: snapshot isolation across all processes, MVCC between writers and table services, and OCC or NBCC between writers.
• Defaults worth memorising: hoodie.index.type resolves to SIMPLE on Spark, hoodie.compact.inline is false, hoodie.clean.commits.retained is 10, and hoodie.parquet.max.file.size is 120 MB.

Most Hudi questions are not conceptual. You know you want an upsert keyed by order_id, you know roughly what compaction does, and what you actually need is the exact spelling of a config key and whether its default is going to surprise you. That is what this page is for.

It is organised for lookup rather than for reading front to back. Each section stands alone, leads with a table of keys and defaults, and shows the PySpark and Spark SQL forms side by side where both exist. The table of contents above is the fastest way in.

Written against the latest Hudi release, 1.2.0 as of now, with Spark bundles for Spark 3.3 through 4.1. Diagrams captioned Source: Apache Hudi documentation are taken from the project’s own docs, © The Apache Software Foundation, and are used under the Apache License 2.0; the rest are my own. Every config key, default, enum value and class name below was read from the release-1.2.0 tag rather than recalled, and source links point at that tag so what you click matches what you read. Where a claim is specific to a version, the version is named.

1. Introduction, key features and architecture

Apache Hudi (Hadoop Upserts Deletes and Incrementals) is a transactional data lakehouse platform that brings ACID semantics, record-level mutations and incremental processing to data lakes. It is built on an immutable, versioned timeline of file slices stored in open formats (Apache Parquet, Apache Avro), so batch and streaming workloads run over the same physical dataset without copying data.

The timeline is the centre of the design: an ordered log of every event that has happened, or is happening, against a table. Each write produces a new instant, a commit on Copy-on-Write and a deltacommit on Merge-on-Read, recording the file slices and the active schema. A reader then picks a query type, which is simply a rule for which instants and file slices to resolve. That is why the five query types are projections over one timeline rather than five separate engines.

Hudi is a table format plus a set of services, not an engine. Writers, readers and catalogs plug into the same physical table, which is what lets one copy of the data serve batch ETL, streaming ingestion and interactive SQL.

2. Getting started: shell and CRUD

Each of the three write operations below lands exactly one completed instant on the timeline, and the snapshot read that follows returns the state that instant produced.

export SPARK_VERSION=3.5
export HUDI_VERSION=1.2.0
export SCALA_VERSION=2.12

pyspark --master "local[2]" \
  --packages org.apache.hudi:hudi-spark$SPARK_VERSION-bundle_$SCALA_VERSION:$HUDI_VERSION \
  --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
  --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
  --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
  --conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar'

Swap pyspark for spark-sql and the same four --conf flags give you the SQL shell used throughout this page.

base_path = 'file:///tmp/hudi_crud_demo'
table_name = 'hudi_crud_table'

hudi_options = {
    'hoodie.table.name': table_name,
    'hoodie.datasource.write.recordkey.field': 'id',
    'hoodie.datasource.write.precombine.field': 'dt',
    'hoodie.datasource.write.partitionpath.field': 'dt',
    'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.SimpleKeyGenerator'
}
cols = ['id', 'name', 'age', 'balance', 'dt']

# CREATE / INSERT
inserts_df = spark.createDataFrame(
    [(1, 'Alice', 30, 1000.50, '2024-01-15'), (2, 'Bob', 25, 2500.00, '2024-01-16')], cols)

inserts_df.write.format('hudi').options(**hudi_options) \
    .option('hoodie.datasource.write.operation', 'insert').mode('overwrite').save(base_path)

# READ (snapshot)
spark.read.format('hudi').load(base_path).show()

# UPDATE / UPSERT
updates_df = spark.createDataFrame(
    [(1, 'Alice Updated', 31, 1500.75, '2024-01-15'), (3, 'Charlie', 28, 3000.0, '2024-01-17')], cols)

updates_df.write.format('hudi').options(**hudi_options) \
    .option('hoodie.datasource.write.operation', 'upsert').mode('append').save(base_path)

# DELETE (hard delete by key)
deletes_df = spark.createDataFrame([(2, '', 0, 0.0, '2024-01-16')], cols)

deletes_df.write.format('hudi').options(**hudi_options) \
    .option('hoodie.datasource.write.operation', 'delete').mode('append').save(base_path)

The same lifecycle in Spark SQL:

CREATE TABLE hudi_crud_table (
  id BIGINT, name STRING, age INT, balance DOUBLE, dt STRING
) USING hudi
PARTITIONED BY (dt)
LOCATION 'file:///tmp/hudi_crud_demo_sql'
TBLPROPERTIES (primaryKey = 'id', orderingFields = 'dt');

INSERT INTO hudi_crud_table (id, name, age, balance, dt) VALUES
  (1, 'Alice', 30, 1000.50, '2024-01-15'),
  (2, 'Bob',   25, 2500.00, '2024-01-16');

SELECT * FROM hudi_crud_table;

UPDATE hudi_crud_table SET balance = 1500.75, age = 31 WHERE id = 1;

DELETE FROM hudi_crud_table WHERE id = 2;

Note: hoodie.datasource.write.precombine.field is deprecated in 1.2.0 and still works, because DataSourceOptions.ORDERING_FIELDS resolves to it. The table property is now hoodie.table.ordering.fields, spelled orderingFields in SQL, with hoodie.table.precombine.field kept as an alias.

3. Table types: Copy-on-Write and Merge-on-Read

The table type is set once at create time and fixes the write-cost against read-cost trade-off.

Choosing a table type:
  Streaming or CDC ingestion, frequent updates?  -> MoR
  Batch loads, read-heavy BI, simple operations? -> CoW
  Unsure? -> Start with CoW; move hot write paths to MoR when write latency hurts.

Copy-on-Write stores data in columnar files only (Parquet). Updates rewrite the affected base files synchronously, producing new file slices while readers keep snapshot isolation.

Merge-on-Read combines columnar and row-based formats (Parquet and Avro). Updates append to Avro log files, merged with the base files during snapshot reads or at compaction.

Source: Apache Hudi documentation.

Source: Apache Hudi documentation.

Dimension by dimension:

hudi_options = {
    'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',   # or MERGE_ON_READ
}

In SQL the same choice is type = 'cow' or type = 'mor' in TBLPROPERTIES.

4. File layout and internal columns

A table splits into partitions, each partition holds file groups (the unit of update), and each file group holds file slices: successive versions of the same records, each a base Parquet file plus the delta logs written against it.

A Merge-on-Read file group accumulates log files against one base file, and compaction folds them into the next slice. Before:

Source: Apache Hudi documentation.

And after compaction, where the merged result becomes the base file of a new slice:

Source: Apache Hudi documentation.

5. Critical writer properties

Record key, partition path and key generator are fixed at table creation and persisted in .hoodie/hoodie.properties. Changing them later means rewriting the table, so the useful split is between what is frozen and what the next job is free to change.

hudi_options = {
    'hoodie.table.name': 'orders',
    'hoodie.datasource.write.table.type': 'MERGE_ON_READ',      # fixed at creation
    'hoodie.datasource.write.recordkey.field': 'order_id',      # record identity
    'hoodie.datasource.write.precombine.field': 'updated_at',   # latest version wins
    'hoodie.datasource.write.partitionpath.field': 'order_date',
    'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.SimpleKeyGenerator',
    'hoodie.datasource.write.operation': 'upsert',
}

Note: Record key uniqueness is scoped per partition with a local index and table-wide with a global index. Pick the index type to match how your keys behave, not the other way round.

The two merge-mode keys are not duplicates, and this catches people. Merge semantics in 1.x are set by RecordMergeMode, which is exposed twice: hoodie.record.merge.mode is the table property in HoodieTableConfig, and hoodie.write.record.merge.mode is the writer option in HoodieWriteConfig. Both arrived in 1.0.0 and both take EVENT_TIME_ORDERING, COMMIT_TIME_ORDERING or CUSTOM. Neither carries a static default: the documented behaviour is EVENT_TIME_ORDERING when an ordering field is set and COMMIT_TIME_ORDERING when it is not. This replaces the legacy payload classes such as OverwriteWithLatestAvroPayload.

The two built-in modes differ only in which column decides the winner. Event-time ordering compares the ordering field, so a late-arriving record with an older value loses:

Source: Apache Hudi documentation.

Commit-time ordering compares the instant instead, so the newest write always wins:

Source: Apache Hudi documentation.

6. The timeline

The timeline under .hoodie/timeline/ (table version 8 and later; directly under .hoodie/ on 0.x) is the ordered log of every action on a table. An instant is identified by its instant time, an action and a state.

Instant time is a monotonic yyyyMMddHHmmssSSS timestamp that totally orders all actions. It is the handle incremental reads and time-travel queries use.

Every action moves through requested, then inflight, then completed. Only completed instants are visible to queries, which is what makes a commit atomic.

Source: Apache Hudi documentation.

Once instants age out of the active timeline they are archived rather than deleted. Hudi 1.x stores that history as an LSM tree, so the archived timeline stays queryable without keeping every instant in the active set:

Source: Apache Hudi documentation.

7. Supported data types

New in 1.2.0, and the reason to be on it if you need them:

1.2.0 also adds a Lance base file format for unstructured data.

Type promotion is widening and lossless: int -> long -> float -> double, and decimal(p,s) to larger precision or scale. Narrowing is rejected. Not supported: CHAR, VARCHAR, NUMERIC, NULL, OBJECT.

8. Write operations

The write operation controls index involvement, deduplication and file sizing, and the lane it falls into decides whether the write pays for an index lookup at all.

The reason the mutable lane exists at all is the shape of the write. A batch overwrite rewrites partitions wholesale; an incremental upsert touches only the file groups holding the keys in the batch:

Source: Apache Hudi documentation.

# Bulk insert: immutable, for initial large loads
hudi_options_bulk = {
    'hoodie.datasource.write.operation': 'bulk_insert',
    'hoodie.bulkinsert.shuffle.parallelism': '200',
    'hoodie.bulkinsert.sort.mode': 'GLOBAL_SORT',
}

INSERT INTO orders VALUES (...);                          -- insert, see the note
INSERT OVERWRITE orders PARTITION (order_date) ...;       -- insert_overwrite
INSERT OVERWRITE TABLE orders ...;                        -- insert_overwrite_table
UPDATE orders SET status = 'shipped' WHERE order_id = 1;  -- upsert
MERGE INTO orders USING updates ON ...;                   -- upsert
DELETE FROM orders WHERE order_id = 2;                    -- delete

Note: Spark SQL INSERT INTO does not honour hoodie.datasource.write.operation. It is governed by hoodie.spark.sql.insert.into.operation (0.14.0 and later; insert by default, or bulk_insert or upsert), with duplicate handling through hoodie.datasource.insert.dup.policy.

9. Key generators

A key generator derives the record key and partition path from named columns through hoodie.datasource.write.keygenerator.class.

A single-field record key is the bare column value; a multi-field key is built as field:value pairs joined by commas. CustomKeyGenerator accepts field:Type specs (SIMPLE or TIMESTAMP) on the partition path only and delegates the record key to SimpleKeyGenerator or ComplexKeyGenerator, so the row above uses recordkey.field=id with partitionpath.field=city:SIMPLE,dept:SIMPLE.

hudi_options = {
    'hoodie.datasource.write.recordkey.field': 'user_id,event_type',
    'hoodie.datasource.write.partitionpath.field': 'country,date',
    'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.ComplexKeyGenerator',
}
# user_id=101, event_type=click, country=US, date=2024-01-15
#   record key -> "user_id:101,event_type:click"
#   partition  -> "US/2024-01-15"

10. Query types

The same table reads five ways. All five are projections over one timeline, differing only in which instants and file slices they resolve.

Source: Apache Hudi documentation.

base_path = 'file:///tmp/hudi_table'

df_snapshot = spark.read.format('hudi').load(base_path)

df_ro = spark.read.format('hudi') \
    .option('hoodie.datasource.query.type', 'read_optimized').load(base_path)

df_incr = spark.read.format('hudi') \
    .option('hoodie.datasource.query.type', 'incremental') \
    .option('hoodie.datasource.read.begin.instanttime', '20240101000000000') \
    .option('hoodie.datasource.read.end.instanttime', '20240102000000000').load(base_path)

df_cdc = spark.read.format('hudi') \
    .option('hoodie.datasource.query.type', 'incremental') \
    .option('hoodie.datasource.query.incremental.format', 'cdc') \
    .option('hoodie.datasource.read.begin.instanttime', '20240101000000000').load(base_path)

df_tt = spark.read.format('hudi').option('as.of.instant', '20240101000000000').load(base_path)

-- Snapshot, the default
SELECT * FROM hudi_table;

-- Read-optimized. The _ro suffix is the Hive-sync name; hudi_query is the
-- table-valued function equivalent and takes 'snapshot' or 'read_optimized'.
SELECT * FROM hudi_table_ro;
SELECT count(*) FROM hudi_query('hudi_table', 'read_optimized');

-- Incremental, and its CDC form
SELECT * FROM hudi_table_changes('hudi_table', 'latest_state', '20240101000000000');
SELECT * FROM hudi_table_changes('hudi_table', 'cdc', '20240101000000000');

-- Time travel: an instant string, or a timestamp in
-- yyyy-MM-dd HH:mm:ss.SSS or yyyy-MM-dd form
SELECT * FROM hudi_table TIMESTAMP AS OF '20240101000000000';
SELECT * FROM hudi_table TIMESTAMP AS OF '2024-01-01 00:00:00';

11. Indexes

Hudi’s writer-side index maps a record key (plus, for non-global indexes, its partition path) to a file group. That mapping is what lets a Copy-on-Write upsert avoid joining against the whole table to find files to rewrite, and bounds how many change records a Merge-on-Read base file merges against.

hoodie.index.type carries no default of its own. HoodieIndexConfig picks one by engine: SIMPLE on Spark and Java, INMEMORY on Flink.

Choosing an index:
  Keys unique only within a partition, writer always knows the partition?
    -> RECORD_LEVEL_INDEX (large tables) or SIMPLE / BLOOM (small to medium)
  Keys unique table-wide, or records move across partitions?
    -> GLOBAL_RECORD_LEVEL_INDEX (large tables) or GLOBAL_SIMPLE / GLOBAL_BLOOM
  Very high write throughput with stable keys? -> BUCKET

A local index enforces uniqueness only within a partition, so a write probes just that partition’s files: cheap and scalable, but the writer must always supply the same partition path for a key. A global index enforces uniqueness across the table and can find or move a key in any partition, at the cost of a table-wide lookup that grows with the table.

Source: Apache Hudi documentation.

HoodieIndex.IndexType has exactly ten values in 1.2.0:

HBASE was an eleventh in the 0.x line and is gone in 1.x, so a 0.x config carrying it needs changing before you upgrade.

# Bucket index: deterministic hashing, no lookup
hudi_options_bucket = {
    'hoodie.index.type': 'BUCKET',
    'hoodie.index.bucket.engine': 'SIMPLE',        # or CONSISTENT_HASHING
    'hoodie.bucket.index.num.buckets': '256',
    'hoodie.bucket.index.hash.field': 'user_id',
}

# Global bloom
hudi_options_global = {
    'hoodie.index.type': 'GLOBAL_BLOOM',
    'hoodie.bloom.index.filter.type': 'DYNAMIC_V0',
    'hoodie.bloom.index.update.partition.path': 'true',
}

# Record-level index, backed by the metadata table
hudi_options_record = {
    'hoodie.index.type': 'GLOBAL_RECORD_LEVEL_INDEX',
    'hoodie.metadata.enable': 'true',
    'hoodie.metadata.global.record.level.index.enable': 'true',
    # partition-scoped keys instead:
    #   'hoodie.index.type': 'RECORD_LEVEL_INDEX'
    #   'hoodie.metadata.record.level.index.enable': 'true'
}

Note: The fixed-bucket BUCKET index (engine SIMPLE) can leave skewed partitions with oversized buckets. The CONSISTENT_HASHING engine, which is Merge-on-Read only, resizes buckets to counter that.

Note: The two record-index scopes keep separate sizing defaults, tuned to what each stores: 10 to 10000 file groups for the global index, 1 to 10 for the partition-scoped one. Let each use its own and the sizing looks after itself.

12. The metadata table

The metadata table is a single internal Merge-on-Read Hudi table under .hoodie/metadata/, one partition per index, that replaces object-store LIST calls and powers data skipping and point lookups.

Source: Apache Hudi documentation.

hudi_options = {
    'hoodie.metadata.enable': 'true',                            # core, file listings
    'hoodie.metadata.index.column.stats.enable': 'true',         # data skipping
    'hoodie.metadata.index.column.stats.column.list': 'user_id,event_type,date',
    'hoodie.metadata.index.bloom.filter.enable': 'true',
    'hoodie.metadata.global.record.level.index.enable': 'true',
}

-- Secondary and expression indexes are DDL, not writer options
CREATE INDEX idx_email ON hudi_table (email);
CREATE INDEX idx_day ON hudi_table USING column_stats(ts)
  OPTIONS(expr='from_unixtime', format='yyyy-MM-dd');
DROP INDEX idx_email ON hudi_table;

-- Inspect what the metadata table holds
SELECT type, key FROM hudi_metadata('hudi_table') LIMIT 20;
CALL show_metadata_table_partitions(table => 'hudi_table');

Note: hoodie.metadata.record.index.enable (0.14.0) survives only as an alias of hoodie.metadata.global.record.level.index.enable, so upgrades are uneventful. Use the new name when you want the partition-scoped index, since the alias resolves to the global one.

13. Concurrency control

Hudi separates three kinds of process acting on a table, writers, table services and readers, and layers four concurrency controls across them.

The first two are always on and need no configuration. The last two are the choice you make, through hoodie.write.concurrency.mode plus a lock provider.

# Optimistic concurrency with a DynamoDB lock
hudi_options = {
    'hoodie.write.concurrency.mode': 'optimistic_concurrency_control',
    'hoodie.clean.policy.failed.writes': 'LAZY',
    'hoodie.write.lock.provider': 'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider',
    'hoodie.write.lock.dynamodb.table': 'hudi_locks',
    'hoodie.write.lock.dynamodb.partition_key': 'lock_key',
    'hoodie.write.lock.dynamodb.region': 'us-east-1',
    'hoodie.write.lock.wait_time_ms': '60000',
    'hoodie.write.lock.num_retries': '10',
}

# Non-blocking concurrency, Merge-on-Read only
hudi_options_nbcc = {
    'hoodie.write.concurrency.mode': 'non_blocking_concurrency_control',
    'hoodie.datasource.write.table.type': 'MERGE_ON_READ',
}

Note: Optimistic concurrency needs an external lock provider; non-blocking concurrency does not. The documentation scopes NBCC to Merge-on-Read tables using the simple or partition-level bucket index, and not to clustering against an ingestion writer, where you still use the optimistic mode.

14. Schema evolution

Backwards-compatible changes work at write time out of the box. The full DDL surface needs the experimental schema-on-read mode.

With schema-on-read enabled, Hudi assigns every column a permanent internal ID and persists the versioned schema history under .hoodie/.schema/. Readers resolve columns by ID rather than by name, which is why a rename is a metadata-only change, a drop is a tombstone on the ID, and an add allocates a fresh ID. Existing data files are never rewritten.

SET hoodie.schema.on.read.enable = true;

ALTER TABLE trips ADD COLUMNS (driver STRING AFTER rider);
ALTER TABLE trips ALTER COLUMN fare TYPE double;
ALTER TABLE trips RENAME COLUMN rider TO passenger;
ALTER TABLE trips DROP COLUMN driver;

Note: Schema-on-read is a one-way door. The documentation states that once it is enabled it cannot be disabled again, because the table will already have accepted changes that depend on it. Turn it on when you need rename, drop or reorder, and not before.

15. Table services

Self-managing background jobs keep file sizes, storage and the timeline healthy. Each runs inline or asynchronously, and they chain: compaction and clustering produce new file slices, cleaning reclaims the superseded ones, and archival trims the timeline entries left behind.

Clustering is the one worth seeing, because it changes layout without changing content: small files are rewritten into larger, sorted ones.

Source: Apache Hudi documentation.

compaction_opts = {
    'hoodie.compact.inline': 'true',                  # default false
    'hoodie.compact.inline.max.delta.commits': '5',   # the default
    'hoodie.compaction.target.io': '512000',          # MB per run, default 500 GB
}

clustering_opts = {
    'hoodie.clustering.inline': 'true',
    'hoodie.clustering.inline.max.commits': '4',
    'hoodie.clustering.plan.strategy.sort.columns': 'user_id,date',
}

maintenance_opts = {
    'hoodie.clean.automatic': 'true',                 # the default
    'hoodie.clean.policy': 'KEEP_LATEST_COMMITS',     # the default
    'hoodie.clean.commits.retained': '10',            # the default
    'hoodie.archive.automatic': 'true',
    'hoodie.keep.min.commits': '20',
    'hoodie.keep.max.commits': '30',
    'hoodie.partition.ttl.inline': 'true',
    'hoodie.partition.ttl.strategy.days.retain': '90',
}

Note: hoodie.clean.policy and hoodie.clean.commits.retained are the canonical keys; the older hoodie.cleaner. spellings are registered alternatives and still resolve.

Note: Async table services inside one writer process coordinate on their own. Running compaction or clustering as a separate job alongside a live writer is multi-writer territory: configure a lock provider and hoodie.write.concurrency.mode first.

16. Bootstrapping an existing dataset

Convert a large existing Parquet dataset into a Hudi table, with or without rewriting the underlying data.

CALL run_bootstrap(
  table => 'bootstrapped_table',
  table_type => 'COPY_ON_WRITE',
  bootstrap_path => 's3a://lakehouse-prod/raw/orders_parquet',
  base_path => 's3a://lakehouse-prod/warehouse/orders',
  rowKey_field => 'order_id',
  partition_path_field => 'order_date'
);

bootstrap_options = {
    'hoodie.table.name': 'bootstrapped_table',
    'hoodie.bootstrap.base.path': 's3a://lakehouse-prod/raw/orders_parquet',
    'hoodie.datasource.write.recordkey.field': 'order_id',
    'hoodie.datasource.write.precombine.field': 'updated_at',
    'hoodie.datasource.write.partitionpath.field': 'order_date',
    'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.SimpleKeyGenerator',
    'hoodie.bootstrap.mode.selector':
        'org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector',
    'hoodie.bootstrap.mode.selector.regex.mode': 'METADATA_ONLY',
    'hoodie.bootstrap.mode.selector.regex': '.*',
}

spark.range(1).write.format('hudi').options(**bootstrap_options) \
    .option('hoodie.datasource.write.operation', 'bootstrap') \
    .mode('append').save('s3a://lakehouse-prod/warehouse/orders')

17. SQL procedures

Day-two administration from Spark SQL with CALL <procedure>(named => args), returned as a result set.

CALL show_commits(table => 'hudi_table', limit => 10);

-- op takes 'run', 'schedule' or 'scheduleandexecute'
CALL run_compaction(op => 'run', table => 'hudi_table');

-- order_strategy takes 'linear', 'z-order' or 'hilbert'
CALL run_clustering(
  table => 'hudi_table',
  order => 'user_id,event_type',
  order_strategy => 'z-order'
);

CALL show_metadata_table_partitions(table => 'hudi_table');

18. Savepoints and restore

A savepoint pins the file versions of a commit so cleaning and archival cannot remove them; restore rewinds the table to that instant. Together they are Hudi’s point-in-time recovery, which is a different guarantee from time travel: time travel reads an old state, restore makes it current again.

-- 1. Before risky maintenance: a backfill, a migration
CALL create_savepoint(table => 'hudi_table', commit_time => '20260713100000000');

-- 2. What restore points exist
CALL show_savepoints(table => 'hudi_table');

-- 3. Rewind. Stop every writer and async table service first
CALL rollback_to_savepoint(table => 'hudi_table', instant_time => '20260713100000000');

-- 4. Release it once the table is healthy again
CALL delete_savepoint(table => 'hudi_table', instant_time => '20260713100000000');

Note: rollback_to_savepoint is destructive. Every commit after the savepoint is removed permanently, and all writers and async table services must be stopped before restoring. Savepointed file versions are exempt from cleaning, so long-lived savepoints grow storage; delete them once the risk has passed.

19. Catalog integration

Publish table metadata to external catalogs so engines can discover and query the tables.

Hive sync registers a Merge-on-Read table under two names, and HiveSyncTool spells the suffixes out: _rt serves snapshot reads and _ro serves read-optimized reads. When an engine looks a commit behind, check which of the two the catalog handed it.

hudi_options_hive = {
    'hoodie.datasource.hive_sync.enable': 'true',
    'hoodie.datasource.hive_sync.mode': 'hms',        # or 'jdbc', 'hiveql'
    'hoodie.datasource.hive_sync.metastore.uris': 'thrift://metastore.internal:9083',
    'hoodie.datasource.hive_sync.database': 'analytics',
    'hoodie.datasource.hive_sync.table': 'orders',
    'hoodie.datasource.hive_sync.partition_fields': 'order_date',
    'hoodie.datasource.hive_sync.partition_extractor_class':
        'org.apache.hudi.hive.MultiPartKeysValueExtractor',
}

hudi_options_glue = {
    'hoodie.datasource.meta.sync.enable': 'true',
    'hoodie.meta.sync.client.tool.class': 'org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool',
    'hoodie.datasource.write.hive_style_partitioning': 'true',
    'hoodie.datasource.hive_sync.database': 'analytics',
    'hoodie.datasource.hive_sync.table': 'orders',
}

Note: Apache XTable is a format-interoperability tool, writing Iceberg and Delta metadata alongside Hudi, rather than a catalog. Pair it with whatever catalog the target engine reads.

20. Writer interfaces

The same table accepts writes through four Spark entry points, sharing the writer configs above and differing only in how the pipeline is driven.

stream_df.writeStream.format('hudi') \
    .options(**hudi_options) \
    .option('checkpointLocation', 's3a://lakehouse-prod/checkpoints/orders') \
    .outputMode('append') \
    .start('s3a://lakehouse-prod/warehouse/orders')

21. Hudi Streamer

HoodieStreamer, formerly DeltaStreamer, is a self-contained Spark job for one-shot or --continuous ingestion: source, then schema provider, then transformers, then the upsert, checkpointed for exactly-once resume.

All of these live under hudi-utilities, and the set is pluggable through org.apache.hudi.utilities.sources.Source. Operations are UPSERT (the default), INSERT, BULK_INSERT and DELETE; HoodieMultiTableStreamer runs many tables from one job.

HUDI_SPARK_BUNDLE=/tmp/hudi-spark3.5-bundle_2.12-1.2.0.jar
HUDI_UTILITIES_SLIM_BUNDLE=/tmp/hudi-utilities-slim-bundle_2.12-1.2.0.jar

spark-submit \
  --jars $HUDI_SPARK_BUNDLE,$HUDI_UTILITIES_SLIM_BUNDLE \
  --class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_SLIM_BUNDLE \
  --table-type COPY_ON_WRITE \
  --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
  --source-ordering-field ts \
  --target-base-path s3a://lakehouse-prod/warehouse/events \
  --target-table events \
  --props /etc/hudi/kafka-source.properties \
  --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
  --continuous --min-sync-interval-seconds 60

hoodie.datasource.write.recordkey.field=event_id
hoodie.datasource.write.precombine.field=ts
hoodie.datasource.write.partitionpath.field=event_date
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator
hoodie.streamer.source.kafka.topic=events_topic
bootstrap.servers=kafka-broker1:9092,kafka-broker2:9092
group.id=hudi_ingestion_group
auto.offset.reset=latest
enable.auto.commit=false
hoodie.datasource.hive_sync.enable=true
hoodie.datasource.hive_sync.database=analytics
hoodie.datasource.hive_sync.table=events
hoodie.datasource.hive_sync.partition_fields=event_date

22. Tuning

The knobs that most affect write and read performance. They fall into three stages, and the middle one is where a write-side decision becomes a read-side cost.

Treat every number above as a starting point and measure on your own data. File sizing and parallelism in particular depend on your record width and cluster shape, which no default can know.

23. Diagnosing a table

-- The one-query health check: unscheduled log bytes per file group
SELECT File_ID, Partition_Path, Log_File_Count, Log_File_Unscheduled
FROM hudi_filesystem_view('hudi_table')
ORDER BY Log_File_Unscheduled DESC
LIMIT 20;

-- Is compaction actually completing?
CALL show_compaction(table => 'hudi_table', limit => 10);

The failure worth planning for is the quiet one. Nothing raises an error when compaction stops; snapshot reads simply merge a little more every hour, which is why the query above belongs on a schedule rather than in an incident.

24. Best practices

• Use a record-level index on large tables. Exact key-to-file lookups avoid bloom false-positive scans at billion-row scale.
• Resist over-partitioning. Thousands of tiny partitions mean small files and metadata pressure. Prefer coarse time buckets.
• Point BI at snapshot queries. Latest committed state with no change-tracking overhead.
• Point ETL at incremental queries. Process what changed since the last run instead of rescanning.
• Enable CDC when a consumer needs before and after images, since it persists extra data with every commit.
• Tune the cleaner deliberately. Retained commits bound storage growth and how far back time travel and rollback can reach.
• Tune compaction from write frequency. Fewer delta commits between compactions means fresher reads and more write amplification.
• Enable schema-on-read only when you need rename, drop or reorder, since it is experimental and cannot be turned off afterwards.
• When a config looks ignored, read what the table actually recorded. .hoodie/hoodie.properties holds the frozen table contract, and reading it settles most arguments about why a writer option did not take effect.

Conclusion

A reference page is only worth keeping open if you trust it, so here is what to re-check when you move off the release this was written against. Read the release notes for table-version changes first, because a table version bump is what changes on-disk layout and reader compatibility. Then re-check the defaults in the tuning table, since file sizing and retention defaults are the ones that move quietly between releases. Config key renames are the third thing, and Hudi is good about keeping the old spelling as a registered alternative, so an upgrade rarely breaks; it just leaves you writing a name that is no longer the canonical one.

The three decisions worth more than the rest of this page put together are the table type, the record key and the index. The first two are frozen at creation and the third governs whether your write cost tracks your batch or your table. Everything else here is a knob you can turn later.

References

• Hudi configuration reference for every hoodie.* key with its default and since-version
• Hudi SQL DDL and SQL DML for the Spark SQL surface used throughout this page
• Hudi concurrency control for the four controls, the lock providers and the multi-writer setup
• HoodieIndex.java at release-1.2.0 for the authoritative IndexType list
• Apache Hudi: architecture, features and what makes it different for the guide this page is the reference companion to