Spark, Iceberg and Hudi internals — explained by the engineer who debugs them in production

Hi, I’m Ranga Reddy.

Deep-dives, troubleshooting playbooks and interactive tools for Apache Spark, Iceberg, Hudi and Kafka — from an Apache Hudi contributor and Cloudera's former Spark–Iceberg backline engineer.

Read the latest About me

Start here

Most read first, then Spark, Iceberg, Hudi and the rest. 64 posts in total.

Spark

Spark on Kubernetes on your Mac: kind, spark-submit, the Spark Operator and Iceberg

Sep 30, 2026 24 min read

Spark

Apache Spark architecture: what actually happens when you call an action

Sep 27, 2026 28 min read

Spark

Apache Spark: the complete cheat sheet

Sep 25, 2026 33 min read

Spark

Spark Submit Command Formatter tool

Sep 11, 2026 Interactive tool

Spark

Spark Configuration Generator tool

Sep 9, 2026 Interactive tool

Spark

Inside Spark's Catalyst optimizer: how a query gets rewritten, rule by rule

Sep 3, 2026 35 min read

Spark

Adaptive query execution in Apache Spark: what it re-plans, and what it leaves alone

Sep 1, 2026 13 min read

Spark

Apache Spark joins in depth: the five strategies and how one gets picked

Aug 30, 2026 33 min read

Spark

Spark memory management: every region, and the arithmetic that sizes it

Aug 28, 2026 27 min read

Spark

Spark shuffle internals: the three writers, and which one your job gets

Jul 27, 2026 44 min read

Spark

Data skew in Apache Spark: how to see it, and nine ways to fix it

Jul 25, 2026 49 min read

Spark

Spark performance tuning: seventeen techniques, and how to prove each one worked

Jul 23, 2026 37 min read

Spark

Native Spark execution: Apache Gluten and DataFusion Comet compared

Jul 21, 2026 27 min read

Spark

Every Apache Spark release, and the problem each one was built to solve

Jul 19, 2026 190 min read

Spark

Spark Connect: the driver becomes a server, and your application becomes a client

Jul 17, 2026 32 min read

Spark

Structured Streaming internals: the checkpoint, the state store, and what a restart cannot change

Jul 15, 2026 12 min read

Spark

Spark Submit Command generator using Iceberg Catalog

Jun 29, 2026 Interactive tool

Spark

A Spark JVM playbook: custom logging, class loading, stack size and proxies

Jun 17, 2026 15 min read

Spark

Logging a Spark stream: the job you cannot restart to debug

Jun 15, 2026 27 min read

Spark

Top Apache Spark interview questions, with answers

Jun 11, 2026 206 min read

Iceberg

Apache Iceberg 1.12.0: what's new, breaking changes, and upgrade guide

Oct 7, 2026 17 min read

Iceberg

What Hive tables could not do, and why Iceberg exists

Sep 19, 2026 10 min read

Iceberg

A local Iceberg playground: Spark, MinIO and a REST catalog in one compose file

Sep 15, 2026 10 min read

Iceberg

The three tiers of an Iceberg table: catalog, metadata, data

Sep 7, 2026 9 min read

Iceberg

Apache Iceberg on Spark: the complete cheat sheet

Sep 5, 2026 40 min read

Iceberg

Apache Iceberg architecture: what actually happens when you commit

Aug 16, 2026 25 min read

Iceberg

Iceberg catalogs: the one pointer that makes a table a table

Aug 14, 2026 8 min read

Iceberg

Life of a write query in Iceberg: three files, one pointer swap, and what ACID means here

Aug 12, 2026 21 min read

Iceberg

Life of a read query in Iceberg: four prunes before a byte of data is read

Aug 10, 2026 18 min read

Iceberg

Iceberg format versions: what v1, v2, v3 and v4 actually change

Aug 8, 2026 22 min read

Iceberg

Schema evolution in Iceberg: why a rename costs nothing

Aug 6, 2026 26 min read

Iceberg

Hidden partitioning and partition evolution: changing layout without rewriting history

Aug 4, 2026 19 min read

Iceberg

Loading data into Iceberg: appends, overwrites and row-level SQL

Aug 2, 2026 27 min read

Iceberg

Streaming into Iceberg: availableNow, checkpoints, and the snapshot count nobody watches

Jul 13, 2026 26 min read

Iceberg

Copy-on-write or merge-on-read: choosing how an Iceberg table pays for a delete

Jul 11, 2026 25 min read

Iceberg

CDC into Iceberg: the delete files nobody counted

Jul 9, 2026 12 min read

Iceberg

Time travel and rollback in Iceberg: undoing a bad write in one statement

Jul 7, 2026 27 min read

Iceberg

Apache Iceberg through Spark Connect: the full surface, and the extension that breaks time travel

Jul 5, 2026 19 min read

Iceberg

Running Iceberg on Apache Polaris: one compose file, two storage backends

Jul 1, 2026 15 min read

Iceberg

Branching, tagging and write-audit-publish in Iceberg: CI for a table

Jun 28, 2026 20 min read

Iceberg

Iceberg views: a versioned view definition, and the cross-engine promise tested

Jun 24, 2026 21 min read

Iceberg

Migrating Hive tables to Iceberg: snapshot first, migrate second

Jun 23, 2026 31 min read

Iceberg

Sorting and clustering in Iceberg: what min/max statistics can and cannot skip

Jun 22, 2026 22 min read

Iceberg

Running Iceberg in production: file sizes, object-storage paths, and the jobs you must schedule

Jun 21, 2026 11 min read

Hudi

Apache Hudi: an introduction to its architecture, features and what makes it different

Sep 17, 2026 30 min read

Hudi

Apache Hudi architecture: what actually happens when you upsert a row

Aug 24, 2026 21 min read

Hudi

The Hudi index: how one lookup decides the cost of every upsert

Aug 22, 2026 16 min read

Hudi

Hudi Merge-on-Read: the write you skipped, and the read that pays for it

Aug 20, 2026 18 min read

Hudi

Apache Hudi on Spark: the complete cheat sheet

Aug 18, 2026 35 min read

Hudi

Apache Hudi through Spark Connect: running the full table surface from a thin client

Jul 3, 2026 26 min read

Hudi

Hudi key generators: the one decision you cannot change later

Jun 30, 2026 9 min read

Hudi

Concurrency control in Hudi: the lock provider that protects nothing

Jun 26, 2026 11 min read

Hudi

Incremental processing in Hudi: reading only what changed, and the timestamp that betrays you

Jun 20, 2026 11 min read

Hudi

Spark and Hudi logging: making a write explain itself

Jun 19, 2026 11 min read

Interview

Data engineer interview questions by company, deduplicated and answered

Oct 21, 2026 50 min read

Modeling

Data Modeling for Data Engineers: From Foundations to Lakehouse Design

Oct 1, 2026 48 min read

Lakehouse

Open table formats in practice: what actually differs between Iceberg, Hudi and Delta Lake

Sep 23, 2026 27 min read

Lakehouse

Run every demo on this blog locally: one compose stack for Hudi, Iceberg and Delta

Sep 21, 2026 19 min read

Tools

Inspecting Parquet files from the command line with parquet-cli

Sep 13, 2026 19 min read

Lakehouse

Apache XTable incremental sync: how to keep your conversions fast

Jul 31, 2026 12 min read

Lakehouse

Apache XTable cheat sheet: one table, every format

Jul 29, 2026 20 min read

Lakehouse

Table maintenance for Iceberg and Hudi: compaction, expiry, and the space you did not reclaim

Jun 27, 2026 13 min read

Lakehouse

One table, three formats: Iceberg, Delta and Hudi interoperability

Jun 25, 2026 9 min read

Linux

Kerberos Setup in Linux

Jun 13, 2026 6 min read