Apache Hudi architecture: what actually happens when you upsert a row

• Architecture: the four primitives
• What does a table look like on disk?
• How does a key find its file group?
• Which version of a key wins?
• Which handle actually writes?
• What does the timeline record?
• What is inside a log file?
• How is a failed write undone?
• What does a reader have to reconcile?
• How does the schema change under a live table?
• Who keeps the table healthy?
• How do concurrent writers coexist?
• When is a simpler choice better?
• Common misconceptions
• Production tips
• Frequently asked questions
• Conclusion
• References
• Trademarks

TL;DR

• A Hudi table is four things: a timeline of instants, file groups holding file slices, an index mapping record keys to file groups, and a metadata table that keeps planning off object storage.
• FileSlice is literally three fields: a base instant time, a base file, and a sorted set of log files. Almost every behaviour that looks strange follows from that shape.
• Three write handles cover every write. HoodieCreateHandle starts a new file group, HoodieMergeHandle rewrites a base file, and HoodieAppendHandle appends a log block. Which one runs is the Copy-on-Write against Merge-on-Read decision.
• The index is what makes an upsert a lookup rather than a join, and it is the one component with no equivalent in Iceberg or Delta.
• Readers only ever see completed instants, which is where atomicity comes from. A half-finished write is on the timeline as .inflight and invisible to queries.

You can use Hudi for a year thinking of it as Parquet with upserts bolted on. Then compaction falls behind, or an upsert that used to take two minutes takes twenty, and the mental model runs out. There is no obvious place to look because the parts that are doing the work, the index and the timeline, are not the parts you configured.

This is the map. It follows one MERGE INTO from the statement to the bytes on storage, naming the class responsible at each step. By the end you should be able to look at a Hudi table directory and say what state it is in, and look at a slow write and say which component is doing the work.

Assumed knowledge: comfort with Spark and SQL. No Hudi internals knowledge is assumed, and every term is defined on first use.

Architecture: the four primitives

Almost everything in Hudi is composed from four things. Learn these and the rest reads as combination.

flowchart LR
  W["writer<br/>Spark, Flink,<br/>Hudi Streamer"] --> T["a Hudi table"]

  T --> TL["timeline<br/>what happened, and when"]
  T --> IX["index<br/>record key to file group"]
  T --> MD["metadata table<br/>listings and indexes"]
  T --> FG["file groups<br/>holding file slices"]

  TL --> Q["a query"]
  IX --> Q
  MD --> Q
  FG --> Q

  Q --> Q1["snapshot"]
  Q --> Q2["read optimized"]
  Q --> Q3["incremental"]

FileSlice is worth reading rather than describing, because the class is three fields:

private final String baseInstantTime;
private HoodieBaseFile baseFile;
private final TreeSet<HoodieLogFile> logFiles;

A base instant, one base file, and a sorted set of log files. A HoodieFileGroup is then a HoodieFileGroupId, which is just a partition path and a file ID, plus a TreeMap of slices keyed by instant. That is the entire storage model, and the rest of this post is consequences of it.

What does a table look like on disk?

s3a://lakehouse-prod/warehouse/trips/
├── .hoodie/
│   ├── hoodie.properties                       <- the frozen table contract
│   ├── timeline/
│   │   ├── 20260916090000123.deltacommit.requested
│   │   ├── 20260916090000123.deltacommit.inflight
│   │   ├── 20260916090000123_20260916090004881.deltacommit
│   │   └── history/                            <- archived instants, an LSM tree
│   ├── metadata/                               <- an internal Hudi table
│   │   ├── files/
│   │   ├── column_stats/
│   │   └── record_index/
│   └── .index_defs/index.json
└── city_id=sf/
    ├── 8f3a1c92-4f1e-4c77-9a2b-4b1d2e3f4a5b-0_0-24-1893_20260916090000123.parquet
    └── .8f3a1c92-4f1e-4c77-9a2b-4b1d2e3f4a5b-0_20260916090000123.log.1_0-31-2104

Two details carry a lot. The leading UUID on both files is the file ID, so those two files are one file group. And log files begin with a dot, which makes them hidden: a casual ls on a Merge-on-Read partition looks exactly like Copy-on-Write.

hoodie.properties is the table contract. Record key, partition path and key generator are written there at creation and cannot be changed without rewriting the table, which is why the first three decisions matter more than the rest.

How does a key find its file group?

This is the step with no equivalent in Iceberg or Delta, and it is why Hudi behaves differently on write-heavy workloads.

Before Hudi can apply an update it must answer, for every incoming record, whether that key already exists and which file group holds it. That step is called tagging. HoodieIndex.IndexType has ten values in 1.2.0, and the choice between them is entirely a choice about how tagging happens:

hoodie.index.type has no default of its own. HoodieIndexConfig picks one by engine: SIMPLE on Spark and Java, INMEMORY on Flink. A Spark writer that never sets it is joining against the table on every write, and that cost scales with the table rather than the batch. An upsert that slows down as the table grows while the batch stays the same size has that shape.

The distinction running through the list is scope. A global index enforces that a key is unique table-wide and can move a record between partitions when its partition value changes. A partition-scoped index allows the same key in several partitions. That is a data-model decision, not a performance one, and your schema already answers it.

Which version of a key wins?

Every post about Hudi says “upsert” and then moves on. The question it skips is the one that decides your data: when two records share a key, which survives? The answer is a table property, fixed at creation, and it is worth knowing before you write anything.

hoodie.table.ordering.fields names the comparison field. The documentation states the rule exactly: “when two records have the same key value, the largest value for the ordering field determined by Object.compareTo(..), is picked”. Several fields compare left to right. In 1.2.0 this key carries hoodie.table.precombine.field as a registered alternative, so the older spelling still works and resolves to the same property.

hoodie.record.merge.mode then picks the strategy, and RecordMergeMode has three values:

There is no fixed default. The config resolves to COMMIT_TIME_ORDERING when no ordering field is set and EVENT_TIME_ORDERING when one is, so naming an ordering field silently changes the merge semantics of the table.

The difference is visible in three lines. Write a record with ordering value 300, then write the same key with ordering value 100 in a later commit:

-- EVENT_TIME_ORDERING, because an ordering field is set
-- the second write loses, even though it committed later
SELECT fare_amount, updated_at FROM trips WHERE trip_id = 'trip-1';
-- 10.0   300

-- COMMIT_TIME_ORDERING, set explicitly with no ordering field
-- the second write wins, because it committed later
SELECT fare_amount, updated_at FROM trips WHERE trip_id = 'trip-1';
-- 99.0   100

That is the whole late-arriving-data story. Event-time ordering makes an out-of-order replay harmless, because an old record cannot overwrite a newer one. Commit-time ordering makes the last writer authoritative, which is what you want when the source has no reliable event time.

The table records its own answer, so you can always check rather than assume:

-- the catalog view: orderingFields, preCombineField, primaryKey, type
SHOW TBLPROPERTIES trips;

hoodie.properties under the table root is the authoritative record, and on a Merge-on-Read table created with an ordering field it reads:

hoodie.table.type=MERGE_ON_READ
hoodie.table.ordering.fields=updated_at
hoodie.record.merge.mode=EVENT_TIME_ORDERING
hoodie.table.version=9

Note that nothing set hoodie.record.merge.mode there. It was derived from the presence of an ordering field, which is the silent switch described above.

Which handle actually writes?

Once tagging says where a record belongs, one of three handles runs. They are the clearest statement of what the table types mean, because the table type is really just a choice of handle.

flowchart TB
  IN["tagged records"] --> Q1{"key already<br/>in a file group?"}
  Q1 -->|"no"| CH["<b>HoodieCreateHandle</b><br/>write a new base file,<br/>starting a new file group"]
  Q1 -->|"yes"| Q2{"table type?"}
  Q2 -->|"Copy-on-Write"| MH["<b>HoodieMergeHandle</b><br/>read the base file, merge,<br/>write a new base file"]
  Q2 -->|"Merge-on-Read"| AH["<b>HoodieAppendHandle</b><br/>append a log block<br/>beside the base file"]
  CH --> C["commit instant<br/>on the timeline"]
  MH --> C
  AH --> C
  C --> MT["update the<br/>metadata table"]

The source describes each in a line. HoodieMergeHandle is “called to read the base file, the incoming records, merge the records and write the final base file”. HoodieAppendHandle is an “IO Operation to append data onto an existing file”. That is the whole Copy-on-Write against Merge-on-Read trade, stated as two classes:

Copy-on-Write buys cheap, simple reads at the cost of rewriting a whole base file per update. Merge-on-Read buys low write latency at the cost of read-side merge work and a compaction job somebody operates.

Change one row in a 120 MB base file and Copy-on-Write writes 120 MB. That is fine when writes are occasional. It stops being fine when a change stream delivers every minute.

What does the timeline record?

The timeline is Hudi’s authoritative record of everything that has happened. Each entry is an instant: an action, a time, and a state.

flowchart LR
  R["20260916090000123<br/>.deltacommit.requested"] --> I["20260916090000123<br/>.deltacommit.inflight"]
  I --> C["20260916090000123_20260916090004881<br/>.deltacommit"]
  I --> X["rollback<br/>on failure"]

Readers only ever see the completed form. A half-finished write sits on the timeline as .inflight and is invisible to queries, which is exactly where atomicity comes from: there is no partial state for a reader to observe, because visibility is a filename.

1.2.0 defines twelve action types:

Instants age out of the active timeline into timeline/history/, stored as an LSM tree so the archive stays queryable without keeping every instant hot.

What is inside a log file?

On Merge-on-Read this is where the updates actually live, and it explains several behaviours that look arbitrary from outside.

A Hudi log file is not a Parquet file. It is a container of blocks separated by a six-byte marker:

byte[] MAGIC = new byte[] {'#', 'H', 'U', 'D', 'I', '#'};

HoodieLogBlockType has seven values in 1.2.0:

Two of those explain a lot. COMMAND_BLOCK is how rollback works without deleting anything: a failed write leaves its blocks in place and Hudi appends a command block naming the instant to invalidate, so append-only storage stays append-only. CORRUPT_BLOCK is deliberate: a truncated write on object storage leaves a partial block, and rather than failing the read, the reader classifies it and scans forward to the next marker.

Block headers carry what a reader needs. SCHEMA in the header is why schema evolution survives the read path: a block written under an older schema carries that schema with it.

How is a failed write undone?

A Spark job can die halfway through writing a file group, on object storage that has no transactions. Hudi handles this by writing down what it is about to do before doing it.

Every file a write is about to produce gets a marker first, under .hoodie/.temp/<instant>/. A marker records the file and its IOType, which is CREATE, MERGE, or APPEND for tables at version 6 and below. That is precisely the information a rollback needs: whether to delete a newly created file or undo a merge into an existing one.

hoodie.write.markers.type defaults to TIMELINE_SERVER_BASED, which batches marker entries through the timeline server into a few files rather than creating one small file per data file. DIRECT writes a marker file per data file, and the documentation notes it is used as a fallback on HDFS or when the timeline server is disabled, and always for Spark structured streaming.

The sequence for a failed write:

1. The instant sits on the timeline as .inflight, so no reader can see it.
2. Markers name every file the write touched.
3. A rollback instant uses the markers to remove or invalidate exactly those files.
4. On Merge-on-Read, invalidating an appended log block does not mean deleting bytes: Hudi appends a COMMAND_BLOCK naming the instant to ignore, so append-only storage stays append-only.

Markers are transient by design. After a successful commit .hoodie/.temp/ is empty, so finding leftover marker directories is itself the signal that a write died and no rollback has run yet.

What does a reader have to reconcile?

A file slice does not have one answer. It has three — snapshot merges the base file with its logs, read_optimized reads the base file alone as of the last compaction, and an incremental query returns the records changed between two instants. The freshness and cost of each are compared in Hudi Merge-on-Read, where the distinction does the most work.

The three table-valued functions below resolve trips against the current database, so a session needs the Hudi bundle, the Hudi SQL extensions and a database selected:

spark-sql \
  --packages org.apache.hudi:hudi-spark3.5-bundle_2.12:1.2.0 \
  --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
  --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog

USE lakehouse_prod;

-- Everything committed, including what is still in log files
SELECT count(*) FROM hudi_query('trips', 'snapshot');

-- Base files only: no merge, but misses everything since the last compaction
SELECT count(*) FROM hudi_query('trips', 'read_optimized');

-- What changed after an instant
SELECT trip_id, city_id, fare_amount, updated_at
FROM hudi_table_changes('trips', 'latest_state', '20260916090000123');

The gap between the first two counts is exactly the set of writes not yet compacted, which turns the pair into a free health check. On Copy-on-Write they are always identical, because there are no logs to skip.

For the physical picture, hudi_filesystem_view reports the real state per file group, and Log_File_Unscheduled is the column that matters: log bytes no compaction has even been planned for.

SELECT File_ID, Partition_Path, Base_Instant_Time,
       Log_File_Count, Log_File_Scheduled, Log_File_Unscheduled
FROM hudi_filesystem_view('trips')
ORDER BY Log_File_Unscheduled DESC
LIMIT 20;

How does the schema change under a live table?

Two mechanisms, and they answer different questions.

On the write path, a schema wider than the table’s is merged in. The guards are off by default and worth knowing: hoodie.avro.schema.validate is false, so an incompatible write is not blocked by a compatibility check, and hoodie.datasource.write.schema.allow.auto.evolution.column.drop is false, which means a write that omits a column does not silently drop it from the table. That default is the safe one, and the reason a write missing a column fails rather than quietly narrowing your schema.

On the read path, every log block carries its own schema in its header. A block written last month under an older schema is readable today without rewriting it, because the reader takes the schema from the block rather than assuming the table’s current one. That is why schema evolution costs nothing at write time on Merge-on-Read.

Full schema evolution, the kind that renames and reorders columns, is opt-in: hoodie.schema.on.read.enable defaults to false.

Time travel uses the same machinery, since the timeline already records what the table looked like at each instant:

-- the table as of a specific instant
SELECT count(*) FROM hudi_query('trips', 'snapshot');

# or through the reader option, which takes an instant time
(spark.read.format("hudi")
    .option("as.of.instant", "20260916090000123")
    .load("s3a://lakehouse-prod/warehouse/trips")
    .count())

Who keeps the table healthy?

Hudi ships maintenance as part of the platform rather than as procedures you must remember. Each service can run inline with writes or asynchronously.

The one default to know: hoodie.compact.inline is false, so a Spark writer on the defaults never compacts as part of the write. Something else has to, whether an async service, a scheduled offline job, or turning inline compaction on. A Merge-on-Read table whose logs grow without bound is a table where that choice was never made.

The failure is quiet rather than loud. Nothing errors when compaction stops; snapshot queries simply merge a little more every hour.

How do concurrent writers coexist?

Hudi separates three kinds of process, writers, table services and readers, and layers four concurrency controls across them:

The first two are always on. The last two are chosen through hoodie.write.concurrency.mode, which defaults to SINGLE_WRITER. Non-blocking concurrency control is the one with the least equivalent elsewhere: two writers append to the same file group and the conflict is resolved by the reader and the compactor using commit completion time, rather than one writer aborting. The documentation scopes it to Merge-on-Read tables on a bucket index.

When is a simpler choice better?

Hudi assumes records have identity and change over time. Where that holds, everything above earns its keep. Where it does not, there are better answers.

Append-only tables need no index and no merge. bulk_insert skips tagging, and if you never correct a row, plain partitioned Parquet or Iceberg is less machinery.

Read-heavy tables written a few times a day do well on Copy-on-Write with inline services, which keeps every read a plain columnar scan.

Teams without capacity to operate table services should start on Copy-on-Write with inline compaction. The table type is a per-table decision, so moving to Merge-on-Read later is open once somebody owns the job.

Common misconceptions

“rollback_to_instant restores the table to that instant.” It unwinds the tip of the timeline, one commit. Rolling back the newest instant succeeds; naming an older one fails with a HoodieRollbackException. Since show_commits returns instants newest first, sorting them and passing the front of the list hands the procedure exactly the argument it cannot process. Going further back is repeated rollbacks, or a savepoint and a restore.

“A savepoint blocks rollback.” It does not. Creating a savepoint and then rolling back the newest instant still succeeds, and rolling back an older instant fails whether a savepoint exists or not. The rule is about position on the timeline, not about savepoints.

“Merge-on-Read is slower to read, so Copy-on-Write is the safe default.” The cost moves rather than disappearing. Copy-on-Write pays it on every write by rewriting whole base files, and Merge-on-Read pays it on the read until compaction catches up. The question is which side of your workload can absorb it, not which is faster.

“An empty result from a metadata function means the table is empty.” Some of them need arguments you may not have supplied. hudi_filesystem_view takes a partition as its second argument, and called with only a table name it returns zero rows rather than an error, which reads exactly like a table with no files. Pass the partition, or the empty string for a non-partitioned table.

Production tips

• Set hoodie.index.type explicitly. An absent value resolves to SIMPLE on Spark, whose cost scales with the table rather than the batch.
• Decide index scope from your data model, not from benchmarks. Global against partition-scoped changes correctness when partition values mutate.
• Decide who runs compaction before production. hoodie.compact.inline is false by default, so absent a decision nobody does.
• Alert on Log_File_Unscheduled, not on query duration. It climbs long before anyone notices a slow read.
• Leave the metadata table on. It is the default, the metadata-backed indexes need it, and planning without it lists object storage.
• Point heavy dashboards at read_optimized when last-compaction freshness is acceptable.
• Treat cleaning as part of your erasure story. A deleted row survives in older file slices until retention expires them.

Frequently asked questions

Why does my Merge-on-Read partition look like Copy-on-Write? Log files start with a dot, so most listings hide them. Use hudi_filesystem_view rather than a directory listing.

Can I change the record key later? Not without rewriting the table. Record key, partition path and key generator go into hoodie.properties at creation and are the table’s contract.

What happens if a compaction job fails halfway? Nothing a reader sees. Compaction is an instant like any other, so an incomplete run sits at .inflight and the base file it was going to replace is untouched. HoodieCompactor takes --retry-last-failed-job to roll back and re-execute the last failed plan.

Does enabling a record index speed up my queries? No. Tagging is a write-path concern. Query-side skipping comes from column_stats and partition_stats, which are separate partitions of the metadata table with separate configs.

How do engines other than Spark and Flink read this? Trino, Presto and Hive read through connectors. Beyond that, Apache XTable converts Hudi metadata into Iceberg or Delta metadata over the same Parquet files, so an engine speaking either can read a table Hudi writes.

Conclusion

Back to the upsert that got slower while the batch stayed the same size. That symptom has a structural cause rather than a tuning knob: an unset index, so every write joins against a table that keeps growing. The fix is a decision about tagging, and you can only reason about it if you know tagging exists.

The idea that organises everything else is that Hudi gave records identity. The index exists because keys are addressable. Incremental queries exist because the timeline knows which records changed in each instant. Non-blocking concurrency is possible because file groups localise where a change lands. Formats that describe a table only as a set of files cannot offer those without first adding the concept Hudi started with.

That choice has a cost worth stating plainly. An index has to be maintained, table services have to be operated, and a Merge-on-Read table nobody compacts degrades every day without raising an error. Hudi asks more of its operators than a format designed for append-mostly batch writes, and repays it where small batches of changes arrive continuously against a large table. If your writes are not keyed, you are paying for machinery you will not use.

References

• Hudi overview and table types for the timeline, file groups and the query types each table type supports
• Hudi configuration reference for every hoodie.* key with its default and since-version
• FileSlice.java, the three fields the storage model rests on
• HoodieIndex.java for the authoritative IndexType list
• Apache Hudi on Spark: the complete cheat sheet for the config surface this post explains

Trademarks

Apache Hudi, Apache Spark, Apache Flink, Apache Parquet, Apache Avro, Apache Iceberg, Apache HBase, Apache XTable (incubating), Apache Hive and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Delta Lake is a trademark of the Linux Foundation.