Guides

The posts on this site in the order worth reading them, grouped by what you are trying to learn: how Spark executes a query, how the table formats store a table, and which tool to reach for.

The category and archive pages list everything by topic and by date. This page is different: it puts the posts in a reading order, and says what each one gives you, so you can start somewhere sensible rather than at whatever was published most recently.

Each path is built so that every post assumes only what the ones above it already explained.

How does Spark actually execute a query?

Seven posts that build on each other, from the runtime up to the optimizer and back down to what fails in production. If you read one, read the first.

  1. Apache Spark architecture is the foundation: who schedules what, how a job becomes stages and tasks, where a shuffle is written, and which component to suspect when a job misbehaves. Everything else here assumes it.
  2. Inside Spark’s Catalyst optimizer takes one query apart rule by rule: the four trees, what a rule is in the source, and how to switch one off and watch the plan change.
  3. Adaptive query execution covers what the engine revises once it has measured the data, and, just as usefully, the parts of the plan it never revisits.
  4. Apache Spark joins in depth is the decision that dominates most jobs: five strategies, why there are five, and how to make each one appear on demand.
  5. Spark memory management accounts for every region of an executor’s memory and the arithmetic that sizes it, including the one hard floor you cannot tune below.
  6. Data skew in Apache Spark is the most common way all of the above goes wrong in production, with six fixes and the measurements that tell you which one you need.
  7. Every Apache Spark release puts the whole thing on a timeline: what each version changed, which defaults moved underneath you, and what a given upgrade is actually worth.

Which Spark reference should I keep open?

These are lookup material rather than a reading order. Open them beside your editor.

How do the table formats store a table?

Start by choosing, then go deep on the one you chose.

  1. Open table formats in practice is the comparison: what actually differs between Iceberg, Hudi and Delta Lake, rather than what the marketing pages differ on.
  2. Apache Hudi: an introduction then Apache Hudi architecture for what happens when you upsert a row, down to bytes on object storage.
  3. Apache Iceberg architecture for what a commit actually swaps, and why the catalog is part of the table.
  4. Apache XTable incremental sync for keeping one table readable as more than one format without re-converting everything each time.

The matching cheat sheets are Hudi, Iceberg and XTable.

How do I run a table format on Spark?

The posts where the two halves of the site meet.

Which tool do I reach for?

Small, self-contained utilities and command-line notes.

Everything, newest first