Apache Hudi architecture: what actually happens when you upsert a row

• Architecture: the four primitives
• What does a table look like on disk?
• How does a key find its file group?
• Which handle actually writes?
• What does the timeline record?
• What is inside a log file?
• What does a reader have to reconcile?
• Who keeps the table healthy?
• How do concurrent writers coexist?
• When is a simpler choice better?
• Production tips
• Frequently asked questions
• Conclusion
• References
• Trademarks

TL;DR

• A Hudi table is four things: a timeline of instants, file groups holding file slices, an index mapping record keys to file groups, and a metadata table that keeps planning off object storage.
• FileSlice is literally three fields: a base instant time, a base file, and a sorted set of log files. Almost every behaviour that looks strange follows from that shape.
• Three write handles cover every write. HoodieCreateHandle starts a new file group, HoodieMergeHandle rewrites a base file, and HoodieAppendHandle appends a log block. Which one runs is the Copy-on-Write against Merge-on-Read decision.
• The index is what makes an upsert a lookup rather than a join, and it is the one component with no equivalent in Iceberg or Delta.
• Readers only ever see completed instants, which is where atomicity comes from. A half-finished write is on the timeline as .inflight and invisible to queries.

You can use Hudi for a year thinking of it as Parquet with upserts bolted on. Then compaction falls behind, or an upsert that used to take two minutes takes twenty, and the mental model runs out. There is no obvious place to look because the parts that are doing the work, the index and the timeline, are not the parts you configured.

This is the map. It follows one MERGE INTO from the statement to the bytes on storage, naming the class responsible at each step. By the end you should be able to look at a Hudi table directory and say what state it is in, and look at a slow write and say which component is doing the work.

Written against the latest Hudi release, 1.2.0 as of now. Every class name, config key and default below was read from the release-1.2.0 tag rather than recalled. Assumed knowledge: comfort with Spark and SQL. No Hudi internals knowledge is assumed, and every term is defined on first use.

Architecture: the four primitives

Almost everything in Hudi is composed from four things. Learn these and the rest reads as combination.

flowchart LR
  W["writer<br/>Spark, Flink,<br/>Hudi Streamer"] --> T["a Hudi table"]

  T --> TL["timeline<br/>what happened, and when"]
  T --> IX["index<br/>record key to file group"]
  T --> MD["metadata table<br/>listings and indexes"]
  T --> FG["file groups<br/>holding file slices"]

  TL --> Q["a query"]
  IX --> Q
  MD --> Q
  FG --> Q

  Q --> Q1["snapshot"]
  Q --> Q2["read optimized"]
  Q --> Q3["incremental"]

FileSlice is worth reading rather than describing, because the class is three fields:

private final String baseInstantTime;
private HoodieBaseFile baseFile;
private final TreeSet<HoodieLogFile> logFiles;

A base instant, one base file, and a sorted set of log files. A HoodieFileGroup is then a HoodieFileGroupId, which is just a partition path and a file ID, plus a TreeMap of slices keyed by instant. That is the entire storage model, and most of the behaviour people find surprising falls straight out of it.

What does a table look like on disk?

s3a://lakehouse-prod/warehouse/trips/
├── .hoodie/
│   ├── hoodie.properties                       <- the frozen table contract
│   ├── timeline/
│   │   ├── 20260916090000123.deltacommit.requested
│   │   ├── 20260916090000123.deltacommit.inflight
│   │   ├── 20260916090000123_20260916090004881.deltacommit
│   │   └── history/                            <- archived instants, an LSM tree
│   ├── metadata/                               <- an internal Hudi table
│   │   ├── files/
│   │   ├── column_stats/
│   │   └── record_index/
│   └── .index_defs/index.json
└── city_id=sf/
    ├── 8f3a1c92-4f1e-4c77-9a2b-4b1d2e3f4a5b-0_0-24-1893_20260916090000123.parquet
    └── .8f3a1c92-4f1e-4c77-9a2b-4b1d2e3f4a5b-0_20260916090000123.log.1_0-31-2104

Two details carry a lot. The leading UUID on both files is the file ID, so those two files are one file group. And log files begin with a dot, which makes them hidden: a casual ls on a Merge-on-Read partition looks exactly like Copy-on-Write.

hoodie.properties is the table contract. Record key, partition path and key generator are written there at creation and cannot be changed without rewriting the table, which is why the first three decisions matter more than the rest.

How does a key find its file group?

This is the step with no equivalent in Iceberg or Delta, and it is why Hudi behaves differently on write-heavy workloads.

Before Hudi can apply an update it must answer, for every incoming record, whether that key already exists and which file group holds it. That step is called tagging. HoodieIndex.IndexType has ten values in 1.2.0, and the choice between them is entirely a choice about how tagging happens:

hoodie.index.type has no default of its own. HoodieIndexConfig picks one by engine: SIMPLE on Spark and Java, INMEMORY on Flink. A Spark writer that never sets it is joining against the table on every write, and that cost scales with the table rather than the batch, which is the single most common reason a Hudi upsert gets slower as the table grows while the batch stays the same size.

The distinction running through the list is scope. A global index enforces that a key is unique table-wide and can move a record between partitions when its partition value changes. A partition-scoped index allows the same key in several partitions. That is a data-model decision, not a performance one, and your schema already answers it.

Which handle actually writes?

Once tagging says where a record belongs, one of three handles runs. They are the clearest statement of what the table types mean, because the table type is really just a choice of handle.

flowchart TB
  IN["tagged records"] --> Q1{"key already<br/>in a file group?"}
  Q1 -->|"no"| CH["<b>HoodieCreateHandle</b><br/>write a new base file,<br/>starting a new file group"]
  Q1 -->|"yes"| Q2{"table type?"}
  Q2 -->|"Copy-on-Write"| MH["<b>HoodieMergeHandle</b><br/>read the base file, merge,<br/>write a new base file"]
  Q2 -->|"Merge-on-Read"| AH["<b>HoodieAppendHandle</b><br/>append a log block<br/>beside the base file"]
  CH --> C["commit instant<br/>on the timeline"]
  MH --> C
  AH --> C
  C --> MT["update the<br/>metadata table"]

The source describes each in a line. HoodieMergeHandle is “called to read the base file, the incoming records, merge the records and write the final base file”. HoodieAppendHandle is an “IO Operation to append data onto an existing file”. That is the whole Copy-on-Write against Merge-on-Read trade, stated as two classes:

Copy-on-Write buys cheap, simple reads at the cost of rewriting a whole base file per update. Merge-on-Read buys low write latency at the cost of read-side merge work and a compaction job somebody operates.

Change one row in a 120 MB base file and Copy-on-Write writes 120 MB. That is fine when writes are occasional. It stops being fine when a change stream delivers every minute.

What does the timeline record?

The timeline is Hudi’s authoritative record of everything that has happened. Each entry is an instant: an action, a time, and a state.

flowchart LR
  R["20260916090000123<br/>.deltacommit.requested"] --> I["20260916090000123<br/>.deltacommit.inflight"]
  I --> C["20260916090000123_20260916090004881<br/>.deltacommit"]
  I --> X["rollback<br/>on failure"]

Readers only ever see the completed form. A half-finished write sits on the timeline as .inflight and is invisible to queries, which is exactly where atomicity comes from: there is no partial state for a reader to observe, because visibility is a filename.

1.2.0 defines twelve action types:

Instants age out of the active timeline into timeline/history/, stored as an LSM tree so the archive stays queryable without keeping every instant hot.

What is inside a log file?

On Merge-on-Read this is where the updates actually live, and it explains several behaviours that look arbitrary from outside.

A Hudi log file is not a Parquet file. It is a container of blocks separated by a six-byte marker:

byte[] MAGIC = new byte[] {'#', 'H', 'U', 'D', 'I', '#'};

HoodieLogBlockType has seven values in 1.2.0:

Two of those explain a lot. COMMAND_BLOCK is how rollback works without deleting anything: a failed write leaves its blocks in place and Hudi appends a command block naming the instant to invalidate, so append-only storage stays append-only. CORRUPT_BLOCK is deliberate: a truncated write on object storage leaves a partial block, and rather than failing the read, the reader classifies it and scans forward to the next marker.

Block headers carry what a reader needs. SCHEMA in the header is why schema evolution survives the read path: a block written under an older schema carries that schema with it.

What does a reader have to reconcile?

A file slice does not have one answer. It has three, and Hudi asks you which you want:

The three table-valued functions below resolve trips against the current database, so a session needs the Hudi bundle, the Hudi SQL extensions and a database selected:

spark-sql \
  --packages org.apache.hudi:hudi-spark3.5-bundle_2.12:1.2.0 \
  --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
  --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog

USE lakehouse_prod;

-- Everything committed, including what is still in log files
SELECT count(*) FROM hudi_query('trips', 'snapshot');

-- Base files only: no merge, but misses everything since the last compaction
SELECT count(*) FROM hudi_query('trips', 'read_optimized');

-- What changed after an instant
SELECT trip_id, city_id, fare_amount, updated_at
FROM hudi_table_changes('trips', 'latest_state', '20260916090000123');

The gap between the first two counts is exactly the set of writes not yet compacted, which turns the pair into a free health check. On Copy-on-Write they are always identical, because there are no logs to skip.

For the physical picture, hudi_filesystem_view reports the real state per file group, and Log_File_Unscheduled is the column that matters: log bytes no compaction has even been planned for.

SELECT File_ID, Partition_Path, Base_Instant_Time,
       Log_File_Count, Log_File_Scheduled, Log_File_Unscheduled
FROM hudi_filesystem_view('trips')
ORDER BY Log_File_Unscheduled DESC
LIMIT 20;

Who keeps the table healthy?

Hudi ships maintenance as part of the platform rather than as procedures you must remember. Each service can run inline with writes or asynchronously.

The one default to know: hoodie.compact.inline is false, so a Spark writer on the defaults never compacts as part of the write. Something else has to, whether an async service, a scheduled offline job, or turning inline compaction on. A Merge-on-Read table whose logs grow forever is almost always a table where nobody made that choice.

The failure is quiet rather than loud. Nothing errors when compaction stops; snapshot queries simply merge a little more every hour.

How do concurrent writers coexist?

Hudi separates three kinds of process, writers, table services and readers, and layers four concurrency controls across them:

The first two are always on. The last two are chosen through hoodie.write.concurrency.mode, which defaults to SINGLE_WRITER. Non-blocking concurrency control is the one with the least equivalent elsewhere: two writers append to the same file group and the conflict is resolved by the reader and the compactor using commit completion time, rather than one writer aborting. The documentation scopes it to Merge-on-Read tables on a bucket index.

When is a simpler choice better?

Hudi assumes records have identity and change over time. Where that holds, everything above earns its keep. Where it does not, there are better answers.

Append-only tables need no index and no merge. bulk_insert skips tagging, and if you never correct a row, plain partitioned Parquet or Iceberg is less machinery.

Read-heavy tables written a few times a day do well on Copy-on-Write with inline services, which keeps every read a plain columnar scan.

Teams without capacity to operate table services should start on Copy-on-Write with inline compaction. The table type is a per-table decision, so moving to Merge-on-Read later is open once somebody owns the job.

Production tips

• Set hoodie.index.type explicitly. An absent value resolves to SIMPLE on Spark, whose cost scales with the table rather than the batch.
• Decide index scope from your data model, not from benchmarks. Global against partition-scoped changes correctness when partition values mutate.
• Decide who runs compaction before production. hoodie.compact.inline is false by default, so absent a decision nobody does.
• Alert on Log_File_Unscheduled, not on query duration. It climbs long before anyone notices a slow read.
• Leave the metadata table on. It is the default, the metadata-backed indexes need it, and planning without it lists object storage.
• Point heavy dashboards at read_optimized when last-compaction freshness is acceptable.
• Treat cleaning as part of your erasure story. A deleted row survives in older file slices until retention expires them.

Frequently asked questions

Why does my Merge-on-Read partition look like Copy-on-Write? Log files start with a dot, so most listings hide them. Use hudi_filesystem_view rather than a directory listing.

Can I change the record key later? Not without rewriting the table. Record key, partition path and key generator go into hoodie.properties at creation and are the table’s contract.

What happens if a compaction job fails halfway? Nothing a reader sees. Compaction is an instant like any other, so an incomplete run sits at .inflight and the base file it was going to replace is untouched. HoodieCompactor takes --retry-last-failed-job to roll back and re-execute the last failed plan.

Does enabling a record index speed up my queries? No. Tagging is a write-path concern. Query-side skipping comes from column_stats and partition_stats, which are separate partitions of the metadata table with separate configs.

How do engines other than Spark and Flink read this? Trino, Presto and Hive read through connectors. Beyond that, Apache XTable converts Hudi metadata into Iceberg or Delta metadata over the same Parquet files, so an engine speaking either can read a table Hudi writes.

Conclusion

Back to the upsert that got slower while the batch stayed the same size. That symptom has one likely cause and it is not a tuning knob: an unset index, so every write joins against a table that keeps growing. The fix is a decision about tagging, and you can only reason about it if you know tagging exists.

The idea that organises everything else is that Hudi gave records identity. The index exists because keys are addressable. Incremental queries exist because the timeline knows which records changed in each instant. Non-blocking concurrency is possible because file groups localise where a change lands. Formats that describe a table only as a set of files cannot offer those without first adding the concept Hudi started with.

That choice has a cost worth stating plainly. An index has to be maintained, table services have to be operated, and a Merge-on-Read table nobody compacts degrades every day without raising an error. Hudi asks more of its operators than a format designed for append-mostly batch writes, and repays it where small batches of changes arrive continuously against a large table. If your writes are not keyed, you are paying for machinery you will not use.

References

• Hudi overview and table types for the timeline, file groups and the query types each table type supports
• Hudi configuration reference for every hoodie.* key with its default and since-version
• FileSlice.java at release-1.2.0, the three fields the storage model rests on
• HoodieIndex.java at release-1.2.0 for the authoritative IndexType list
• Apache Hudi on Spark: the complete cheat sheet for the config surface this post explains

Trademarks

Apache Hudi, Apache Spark, Apache Flink, Apache Parquet, Apache Avro, Apache Iceberg, Apache HBase, Apache XTable (incubating), Apache Hive and Apache are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Delta Lake is a trademark of the Linux Foundation.