Tags

115 tags across 69 posts.

AQE2 ATS1 Architecture4 Arrow1 Branching1 CDC2 Catalog4 Catalogs1 Catalyst2 Checkpoints1 Cleaning1 Clustering1 Codegen1 Comet1 Commits1 Compaction5 Compatibility2 Concurrency3 Configuration1 Defaults1 Deletes3 Delta5 Design1 Docker2 Evolution1 Flink1 Generator1 GitHub1 Gluten1 Governance1 History1 Hive2 Hudi17 Iceberg33 Incremental1 Indexing3 Ingestion1 Internals9 Interoperability2 Interview1 Jobs1 Joins3 Kafka1 KafkaConnect1 Kerberos1 KeyGenerator1 Kubernetes1 Lakehouse11 Lineage1 Linux1 Locking1 Log4j2 Logging3 MCP1 Maintenance1 Manifests1 Memory1 Merge1 Metadata4 Migration2 MinIO2 Modeling1 Operator1 Optimizer1 Overwrite1 Paimon1 Parquet2 Partitioning2 Performance6 Pipelines1 Polaris1 Production3 Pruning2 PySpark4 Query1 REST1 Recovery1 Reference5 Release1 Releases1 Remote1 SQL3 Scenarios1 Scheduling1 Schema1 Search1 Security1 Serialization1 Shuffle2 Skew1 Snapshots3 Sorting1 Spark31 SparkConnect3 State1 Storage1 Streaming10 Tagging1 Timeline3 Tools2 Transactions1 Transforms1 Trino1 Troubleshoot2 Tuning5 Upgrade1 Upserts1 Utilities3 Velox1 Views1 WAP1 Warehouse1 Watermarks1 XTable4 gRPC1

AQE 2

  1. Adaptive query execution in Apache Spark: what it re-plans, and what it leaves alone
    Spark
  2. Spark performance tuning: seventeen techniques, and how to prove each one worked
    Spark

ATS 1

  1. 50 job websites for remote and tech roles, and how to use them well
    Career

Architecture 4

  1. Apache Spark architecture: what actually happens when you call an action
    Spark
  2. The three tiers of an Iceberg table: catalog, metadata, data
    Iceberg
  3. Apache Hudi architecture: what actually happens when you upsert a row
    Hudi
  4. Apache Iceberg architecture: what actually happens when you commit
    Iceberg

Arrow 1

  1. Spark Connect: the driver becomes a server, and your application becomes a client
    Spark

Branching 1

  1. Branching, tagging and write-audit-publish in Iceberg: CI for a table
    Iceberg

CDC 2

  1. CDC into Iceberg: the delete files nobody counted
    Iceberg
  2. Incremental processing in Hudi: reading only what changed, and the timestamp that betrays you
    Hudi

Catalog 4

  1. A local Iceberg playground: Spark, MinIO and a REST catalog in one compose file
    Iceberg
  2. Apache Iceberg architecture: what actually happens when you commit
    Iceberg
  3. Iceberg catalogs: the one pointer that makes a table a table
    Iceberg
  4. Running Iceberg on Apache Polaris: one compose file, two storage backends
    Iceberg

Catalogs 1

  1. Iceberg views: a versioned view definition, and the cross-engine promise tested
    Iceberg

Catalyst 2

  1. Inside Spark's Catalyst optimizer: how a query gets rewritten, rule by rule
    Spark
  2. Apache Spark joins in depth: the five strategies and how one gets picked
    Spark

Checkpoints 1

  1. Streaming into Iceberg: availableNow, checkpoints, and the snapshot count nobody watches
    Iceberg

Cleaning 1

  1. Table maintenance for Iceberg and Hudi: compaction, expiry, and the space you did not reclaim
    Lakehouse

Clustering 1

  1. Sorting and clustering in Iceberg: what min/max statistics can and cannot skip
    Iceberg

Codegen 1

  1. Inside Spark's Catalyst optimizer: how a query gets rewritten, rule by rule
    Spark

Comet 1

  1. Native Spark execution: Apache Gluten and DataFusion Comet compared
    Spark

Commits 1

  1. Life of a write query in Iceberg: three files, one pointer swap, and what ACID means here
    Iceberg

Compaction 5

  1. Apache Hudi architecture: what actually happens when you upsert a row
    Hudi
  2. Hudi Merge-on-Read: the write you skipped, and the read that pays for it
    Hudi
  3. Copy-on-write or merge-on-read: choosing how an Iceberg table pays for a delete
    Iceberg
  4. CDC into Iceberg: the delete files nobody counted
    Iceberg
  5. Table maintenance for Iceberg and Hudi: compaction, expiry, and the space you did not reclaim
    Lakehouse

Compatibility 2

  1. Iceberg format versions: what v1, v2, v3 and v4 actually change
    Iceberg
  2. Schema evolution in Iceberg: why a rename costs nothing
    Iceberg

Concurrency 3

  1. Hudi Merge-on-Read: the write you skipped, and the read that pays for it
    Hudi
  2. Life of a write query in Iceberg: three files, one pointer swap, and what ACID means here
    Iceberg
  3. Concurrency control in Hudi: the lock provider that protects nothing
    Hudi

Configuration 1

  1. Hudi key generators: the one decision you cannot change later
    Hudi

Defaults 1

  1. Every Apache Spark release, and the problem each one was built to solve
    Spark

Deletes 3

  1. Iceberg format versions: what v1, v2, v3 and v4 actually change
    Iceberg
  2. Copy-on-write or merge-on-read: choosing how an Iceberg table pays for a delete
    Iceberg
  3. CDC into Iceberg: the delete files nobody counted
    Iceberg

Delta 5

  1. Open table formats in practice: what actually differs between Iceberg, Hudi and Delta Lake
    Lakehouse
  2. Run every demo on this blog locally: one compose stack for Hudi, Iceberg and Delta
    Lakehouse
  3. Apache XTable incremental sync: how to keep your conversions fast
    Lakehouse
  4. Apache XTable cheat sheet: one table, every format
    Lakehouse
  5. One table, three formats: Iceberg, Delta and Hudi interoperability
    Lakehouse

Design 1

  1. Top Data engineer interview questions
    Interview

Docker 2

  1. Run every demo on this blog locally: one compose stack for Hudi, Iceberg and Delta
    Lakehouse
  2. A local Iceberg playground: Spark, MinIO and a REST catalog in one compose file
    Iceberg

Evolution 1

  1. Schema evolution in Iceberg: why a rename costs nothing
    Iceberg

Generator 1

  1. Spark Configuration Generator tool
    Spark

GitHub 1

  1. Six URL rewrites that turn any GitHub repository into an editor, a prompt, a diagram or an MCP server
    Tools

Gluten 1

  1. Native Spark execution: Apache Gluten and DataFusion Comet compared
    Spark

Governance 1

  1. Iceberg catalogs: the one pointer that makes a table a table
    Iceberg

History 1

  1. Time travel and rollback in Iceberg: undoing a bad write in one statement
    Iceberg

Hive 2

  1. What Hive tables could not do, and why Iceberg exists
    Iceberg
  2. Migrating Hive tables to Iceberg: snapshot first, migrate second
    Iceberg

Hudi 17

  1. One Hudi write, two table formats: committing Iceberg metadata inside the Hudi commit
    Hudi
  2. Open table formats in practice: what actually differs between Iceberg, Hudi and Delta Lake
    Lakehouse
  3. Run every demo on this blog locally: one compose stack for Hudi, Iceberg and Delta
    Lakehouse
  4. Apache Hudi: an introduction to its architecture, features and what makes it different
    Hudi
  5. Apache Hudi architecture: what actually happens when you upsert a row
    Hudi
  6. The Hudi index: how one lookup decides the cost of every upsert
    Hudi
  7. Hudi Merge-on-Read: the write you skipped, and the read that pays for it
    Hudi
  8. Apache Hudi on Spark: the complete cheat sheet
    Hudi
  9. Apache XTable incremental sync: how to keep your conversions fast
    Lakehouse
  10. Apache XTable cheat sheet: one table, every format
    Lakehouse
  11. Apache Hudi through Spark Connect: running the full table surface from a thin client
    Hudi
  12. Hudi key generators: the one decision you cannot change later
    Hudi
  13. Table maintenance for Iceberg and Hudi: compaction, expiry, and the space you did not reclaim
    Lakehouse
  14. Concurrency control in Hudi: the lock provider that protects nothing
    Hudi
  15. One table, three formats: Iceberg, Delta and Hudi interoperability
    Lakehouse
  16. Incremental processing in Hudi: reading only what changed, and the timestamp that betrays you
    Hudi
  17. Spark and Hudi logging: making a write explain itself
    Hudi

Iceberg 33

  1. One Hudi write, two table formats: committing Iceberg metadata inside the Hudi commit
    Hudi
  2. Apache Iceberg 1.12.0: what's new, breaking changes, and upgrade guide
    Iceberg
  3. Spark on Kubernetes on your Mac: kind, spark-submit, the Spark Operator and Iceberg
    Spark
  4. Open table formats in practice: what actually differs between Iceberg, Hudi and Delta Lake
    Lakehouse
  5. Run every demo on this blog locally: one compose stack for Hudi, Iceberg and Delta
    Lakehouse
  6. What Hive tables could not do, and why Iceberg exists
    Iceberg
  7. A local Iceberg playground: Spark, MinIO and a REST catalog in one compose file
    Iceberg
  8. The three tiers of an Iceberg table: catalog, metadata, data
    Iceberg
  9. Apache Iceberg on Spark: the complete cheat sheet
    Iceberg
  10. Apache Iceberg architecture: what actually happens when you commit
    Iceberg
  11. Iceberg catalogs: the one pointer that makes a table a table
    Iceberg
  12. Life of a write query in Iceberg: three files, one pointer swap, and what ACID means here
    Iceberg
  13. Life of a read query in Iceberg: four prunes before a byte of data is read
    Iceberg
  14. Iceberg format versions: what v1, v2, v3 and v4 actually change
    Iceberg
  15. Schema evolution in Iceberg: why a rename costs nothing
    Iceberg
  16. Hidden partitioning and partition evolution: changing layout without rewriting history
    Iceberg
  17. Loading data into Iceberg: appends, overwrites and row-level SQL
    Iceberg
  18. Apache XTable incremental sync: how to keep your conversions fast
    Lakehouse
  19. Apache XTable cheat sheet: one table, every format
    Lakehouse
  20. Streaming into Iceberg: availableNow, checkpoints, and the snapshot count nobody watches
    Iceberg
  21. Copy-on-write or merge-on-read: choosing how an Iceberg table pays for a delete
    Iceberg
  22. CDC into Iceberg: the delete files nobody counted
    Iceberg
  23. Time travel and rollback in Iceberg: undoing a bad write in one statement
    Iceberg
  24. Apache Iceberg through Spark Connect: the full surface, and the extension that breaks time travel
    Iceberg
  25. Running Iceberg on Apache Polaris: one compose file, two storage backends
    Iceberg
  26. Spark Submit Command generator using Iceberg Catalog
    Spark
  27. Branching, tagging and write-audit-publish in Iceberg: CI for a table
    Iceberg
  28. Table maintenance for Iceberg and Hudi: compaction, expiry, and the space you did not reclaim
    Lakehouse
  29. One table, three formats: Iceberg, Delta and Hudi interoperability
    Lakehouse
  30. Iceberg views: a versioned view definition, and the cross-engine promise tested
    Iceberg
  31. Migrating Hive tables to Iceberg: snapshot first, migrate second
    Iceberg
  32. Sorting and clustering in Iceberg: what min/max statistics can and cannot skip
    Iceberg
  33. Running Iceberg in production: file sizes, object-storage paths, and the jobs you must schedule
    Iceberg

Incremental 1

  1. Incremental processing in Hudi: reading only what changed, and the timestamp that betrays you
    Hudi

Indexing 3

  1. Apache Hudi: an introduction to its architecture, features and what makes it different
    Hudi
  2. Apache Hudi architecture: what actually happens when you upsert a row
    Hudi
  3. The Hudi index: how one lookup decides the cost of every upsert
    Hudi

Ingestion 1

  1. Loading data into Iceberg: appends, overwrites and row-level SQL
    Iceberg

Internals 9

  1. Apache Spark architecture: what actually happens when you call an action
    Spark
  2. Inside Spark's Catalyst optimizer: how a query gets rewritten, rule by rule
    Spark
  3. Adaptive query execution in Apache Spark: what it re-plans, and what it leaves alone
    Spark
  4. Apache Spark joins in depth: the five strategies and how one gets picked
    Spark
  5. Spark memory management: every region, and the arithmetic that sizes it
    Spark
  6. Apache Hudi architecture: what actually happens when you upsert a row
    Hudi
  7. Apache Iceberg architecture: what actually happens when you commit
    Iceberg
  8. Spark shuffle internals: the three writers, and which one your job gets
    Spark
  9. Every Apache Spark release, and the problem each one was built to solve
    Spark

Interoperability 2

  1. One Hudi write, two table formats: committing Iceberg metadata inside the Hudi commit
    Hudi
  2. Iceberg views: a versioned view definition, and the cross-engine promise tested
    Iceberg

Interview 1

  1. Top Apache Spark interview questions, with answers
    Interview

Jobs 1

  1. 50 job websites for remote and tech roles, and how to use them well
    Career

Joins 3

  1. Apache Spark joins in depth: the five strategies and how one gets picked
    Spark
  2. Data skew in Apache Spark: how to see it, and nine ways to fix it
    Spark
  3. Spark performance tuning: seventeen techniques, and how to prove each one worked
    Spark

Kafka 1

  1. System Design for Data Engineers: A Framework, the Building Blocks, and Worked Designs
    Interview

KafkaConnect 1

  1. Apache Iceberg 1.12.0: what's new, breaking changes, and upgrade guide
    Iceberg

Kerberos 1

  1. Kerberos Setup in Linux
    Linux

KeyGenerator 1

  1. Hudi key generators: the one decision you cannot change later
    Hudi

Kubernetes 1

  1. Spark on Kubernetes on your Mac: kind, spark-submit, the Spark Operator and Iceberg
    Spark

Lakehouse 11

  1. System Design for Data Engineers: A Framework, the Building Blocks, and Worked Designs
    Interview
  2. Apache Iceberg 1.12.0: what's new, breaking changes, and upgrade guide
    Iceberg
  3. Data Modeling for Data Engineers: From Foundations to Lakehouse Design
    Interview
  4. Open table formats in practice: what actually differs between Iceberg, Hudi and Delta Lake
    Lakehouse
  5. What Hive tables could not do, and why Iceberg exists
    Iceberg
  6. Apache Hudi: an introduction to its architecture, features and what makes it different
    Hudi
  7. Apache Iceberg on Spark: the complete cheat sheet
    Iceberg
  8. Apache Hudi on Spark: the complete cheat sheet
    Hudi
  9. Apache XTable incremental sync: how to keep your conversions fast
    Lakehouse
  10. Apache Iceberg through Spark Connect: the full surface, and the extension that breaks time travel
    Iceberg
  11. Apache Hudi through Spark Connect: running the full table surface from a thin client
    Hudi

Lineage 1

  1. Iceberg format versions: what v1, v2, v3 and v4 actually change
    Iceberg

Linux 1

  1. Kerberos Setup in Linux
    Linux

Locking 1

  1. Concurrency control in Hudi: the lock provider that protects nothing
    Hudi

Log4j 2

  1. Spark and Hudi logging: making a write explain itself
    Hudi
  2. Logging a Spark stream: the job you cannot restart to debug
    Spark

Logging 3

  1. Spark and Hudi logging: making a write explain itself
    Hudi
  2. A Spark JVM playbook: custom logging, class loading, stack size and proxies
    Spark
  3. Logging a Spark stream: the job you cannot restart to debug
    Spark

MCP 1

  1. Six URL rewrites that turn any GitHub repository into an editor, a prompt, a diagram or an MCP server
    Tools

Maintenance 1

  1. Sorting and clustering in Iceberg: what min/max statistics can and cannot skip
    Iceberg

Manifests 1

  1. The three tiers of an Iceberg table: catalog, metadata, data
    Iceberg

Memory 1

  1. Spark memory management: every region, and the arithmetic that sizes it
    Spark

Merge 1

  1. Loading data into Iceberg: appends, overwrites and row-level SQL
    Iceberg

Metadata 4

  1. What Hive tables could not do, and why Iceberg exists
    Iceberg
  2. The three tiers of an Iceberg table: catalog, metadata, data
    Iceberg
  3. The Hudi index: how one lookup decides the cost of every upsert
    Hudi
  4. Life of a read query in Iceberg: four prunes before a byte of data is read
    Iceberg

Migration 2

  1. Every Apache Spark release, and the problem each one was built to solve
    Spark
  2. Migrating Hive tables to Iceberg: snapshot first, migrate second
    Iceberg

MinIO 2

  1. A local Iceberg playground: Spark, MinIO and a REST catalog in one compose file
    Iceberg
  2. Running Iceberg on Apache Polaris: one compose file, two storage backends
    Iceberg

Modeling 1

  1. Data Modeling for Data Engineers: From Foundations to Lakehouse Design
    Interview

Operator 1

  1. Spark on Kubernetes on your Mac: kind, spark-submit, the Spark Operator and Iceberg
    Spark

Optimizer 1

  1. Inside Spark's Catalyst optimizer: how a query gets rewritten, rule by rule
    Spark

Overwrite 1

  1. Loading data into Iceberg: appends, overwrites and row-level SQL
    Iceberg

Paimon 1

  1. Apache XTable cheat sheet: one table, every format
    Lakehouse

Parquet 2

  1. Inspecting Parquet files from the command line with parquet-cli
    Tools
  2. Migrating Hive tables to Iceberg: snapshot first, migrate second
    Iceberg

Partitioning 2

  1. Hidden partitioning and partition evolution: changing layout without rewriting history
    Iceberg
  2. Hudi key generators: the one decision you cannot change later
    Hudi

Performance 6

  1. Adaptive query execution in Apache Spark: what it re-plans, and what it leaves alone
    Spark
  2. Apache Spark joins in depth: the five strategies and how one gets picked
    Spark
  3. Spark memory management: every region, and the arithmetic that sizes it
    Spark
  4. Data skew in Apache Spark: how to see it, and nine ways to fix it
    Spark
  5. Spark performance tuning: seventeen techniques, and how to prove each one worked
    Spark
  6. Sorting and clustering in Iceberg: what min/max statistics can and cannot skip
    Iceberg

Pipelines 1

  1. System Design for Data Engineers: A Framework, the Building Blocks, and Worked Designs
    Interview

Polaris 1

  1. Running Iceberg on Apache Polaris: one compose file, two storage backends
    Iceberg

Production 3

  1. Branching, tagging and write-audit-publish in Iceberg: CI for a table
    Iceberg
  2. Concurrency control in Hudi: the lock provider that protects nothing
    Hudi
  3. Running Iceberg in production: file sizes, object-storage paths, and the jobs you must schedule
    Iceberg

Pruning 2

  1. Life of a read query in Iceberg: four prunes before a byte of data is read
    Iceberg
  2. Hidden partitioning and partition evolution: changing layout without rewriting history
    Iceberg

PySpark 4

  1. Spark 4 in four releases: the features worth knowing and what they cost
    Spark
  2. Apache Iceberg on Spark: the complete cheat sheet
    Iceberg
  3. Apache Hudi on Spark: the complete cheat sheet
    Hudi
  4. Spark Connect: the driver becomes a server, and your application becomes a client
    Spark

Query 1

  1. Life of a read query in Iceberg: four prunes before a byte of data is read
    Iceberg

REST 1

  1. Iceberg catalogs: the one pointer that makes a table a table
    Iceberg

Recovery 1

  1. Time travel and rollback in Iceberg: undoing a bad write in one statement
    Iceberg

Reference 5

  1. Apache Spark: the complete cheat sheet
    Spark
  2. Apache Iceberg on Spark: the complete cheat sheet
    Iceberg
  3. Apache Hudi on Spark: the complete cheat sheet
    Hudi
  4. Apache XTable cheat sheet: one table, every format
    Lakehouse
  5. Top Apache Spark interview questions, with answers
    Interview

Release 1

  1. Apache Iceberg 1.12.0: what's new, breaking changes, and upgrade guide
    Iceberg

Releases 1

  1. Every Apache Spark release, and the problem each one was built to solve
    Spark

Remote 1

  1. 50 job websites for remote and tech roles, and how to use them well
    Career

SQL 3

  1. Spark 4 in four releases: the features worth knowing and what they cost
    Spark
  2. Top Data engineer interview questions
    Interview
  3. Apache Spark: the complete cheat sheet
    Spark

Scenarios 1

  1. Top Apache Spark interview questions, with answers
    Interview

Scheduling 1

  1. Apache Spark architecture: what actually happens when you call an action
    Spark

Schema 1

  1. Schema evolution in Iceberg: why a rename costs nothing
    Iceberg

Security 1

  1. Kerberos Setup in Linux
    Linux

Serialization 1

  1. Spark shuffle internals: the three writers, and which one your job gets
    Spark

Shuffle 2

  1. Apache Spark architecture: what actually happens when you call an action
    Spark
  2. Spark shuffle internals: the three writers, and which one your job gets
    Spark

Skew 1

  1. Data skew in Apache Spark: how to see it, and nine ways to fix it
    Spark

Snapshots 3

  1. Apache Iceberg architecture: what actually happens when you commit
    Iceberg
  2. Streaming into Iceberg: availableNow, checkpoints, and the snapshot count nobody watches
    Iceberg
  3. Time travel and rollback in Iceberg: undoing a bad write in one statement
    Iceberg

Sorting 1

  1. Sorting and clustering in Iceberg: what min/max statistics can and cannot skip
    Iceberg

Spark 31

  1. Spark 4 in four releases: the features worth knowing and what they cost
    Spark
  2. Top Data engineer interview questions
    Interview
  3. Apache Iceberg 1.12.0: what's new, breaking changes, and upgrade guide
    Iceberg
  4. Data Modeling for Data Engineers: From Foundations to Lakehouse Design
    Interview
  5. Spark on Kubernetes on your Mac: kind, spark-submit, the Spark Operator and Iceberg
    Spark
  6. Apache Spark architecture: what actually happens when you call an action
    Spark
  7. Apache Spark: the complete cheat sheet
    Spark
  8. Open table formats in practice: what actually differs between Iceberg, Hudi and Delta Lake
    Lakehouse
  9. Apache Hudi: an introduction to its architecture, features and what makes it different
    Hudi
  10. Spark Submit Command Formatter tool
    Spark
  11. Spark Configuration Generator tool
    Spark
  12. Apache Iceberg on Spark: the complete cheat sheet
    Iceberg
  13. Inside Spark's Catalyst optimizer: how a query gets rewritten, rule by rule
    Spark
  14. Adaptive query execution in Apache Spark: what it re-plans, and what it leaves alone
    Spark
  15. Apache Spark joins in depth: the five strategies and how one gets picked
    Spark
  16. Spark memory management: every region, and the arithmetic that sizes it
    Spark
  17. Apache Hudi on Spark: the complete cheat sheet
    Hudi
  18. Spark shuffle internals: the three writers, and which one your job gets
    Spark
  19. Data skew in Apache Spark: how to see it, and nine ways to fix it
    Spark
  20. Spark performance tuning: seventeen techniques, and how to prove each one worked
    Spark
  21. Native Spark execution: Apache Gluten and DataFusion Comet compared
    Spark
  22. Every Apache Spark release, and the problem each one was built to solve
    Spark
  23. Structured Streaming internals: the checkpoint, the state store, and what a restart cannot change
    Spark
  24. Apache Iceberg through Spark Connect: the full surface, and the extension that breaks time travel
    Iceberg
  25. Apache Hudi through Spark Connect: running the full table surface from a thin client
    Hudi
  26. Running Iceberg on Apache Polaris: one compose file, two storage backends
    Iceberg
  27. Spark Submit Command generator using Iceberg Catalog
    Spark
  28. Spark and Hudi logging: making a write explain itself
    Hudi
  29. A Spark JVM playbook: custom logging, class loading, stack size and proxies
    Spark
  30. Logging a Spark stream: the job you cannot restart to debug
    Spark
  31. Top Apache Spark interview questions, with answers
    Interview

SparkConnect 3

  1. Spark Connect: the driver becomes a server, and your application becomes a client
    Spark
  2. Apache Iceberg through Spark Connect: the full surface, and the extension that breaks time travel
    Iceberg
  3. Apache Hudi through Spark Connect: running the full table surface from a thin client
    Hudi

State 1

  1. Structured Streaming internals: the checkpoint, the state store, and what a restart cannot change
    Spark

Storage 1

  1. Running Iceberg in production: file sizes, object-storage paths, and the jobs you must schedule
    Iceberg

Streaming 10

  1. Spark 4 in four releases: the features worth knowing and what they cost
    Spark
  2. System Design for Data Engineers: A Framework, the Building Blocks, and Worked Designs
    Interview
  3. Top Data engineer interview questions
    Interview
  4. Apache Spark: the complete cheat sheet
    Spark
  5. Apache Hudi: an introduction to its architecture, features and what makes it different
    Hudi
  6. Structured Streaming internals: the checkpoint, the state store, and what a restart cannot change
    Spark
  7. Streaming into Iceberg: availableNow, checkpoints, and the snapshot count nobody watches
    Iceberg
  8. Copy-on-write or merge-on-read: choosing how an Iceberg table pays for a delete
    Iceberg
  9. Incremental processing in Hudi: reading only what changed, and the timestamp that betrays you
    Hudi
  10. Logging a Spark stream: the job you cannot restart to debug
    Spark

Tagging 1

  1. Branching, tagging and write-audit-publish in Iceberg: CI for a table
    Iceberg

Timeline 3

  1. Hudi Merge-on-Read: the write you skipped, and the read that pays for it
    Hudi
  2. Concurrency control in Hudi: the lock provider that protects nothing
    Hudi
  3. Incremental processing in Hudi: reading only what changed, and the timestamp that betrays you
    Hudi

Tools 2

  1. Six URL rewrites that turn any GitHub repository into an editor, a prompt, a diagram or an MCP server
    Tools
  2. Inspecting Parquet files from the command line with parquet-cli
    Tools

Transactions 1

  1. Life of a write query in Iceberg: three files, one pointer swap, and what ACID means here
    Iceberg

Transforms 1

  1. Hidden partitioning and partition evolution: changing layout without rewriting history
    Iceberg

Trino 1

  1. Iceberg views: a versioned view definition, and the cross-engine promise tested
    Iceberg

Troubleshoot 2

  1. Spark and Hudi logging: making a write explain itself
    Hudi
  2. A Spark JVM playbook: custom logging, class loading, stack size and proxies
    Spark

Tuning 5

  1. Apache Spark: the complete cheat sheet
    Spark
  2. Adaptive query execution in Apache Spark: what it re-plans, and what it leaves alone
    Spark
  3. Spark memory management: every region, and the arithmetic that sizes it
    Spark
  4. Data skew in Apache Spark: how to see it, and nine ways to fix it
    Spark
  5. Running Iceberg in production: file sizes, object-storage paths, and the jobs you must schedule
    Iceberg

Upgrade 1

  1. Apache Iceberg 1.12.0: what's new, breaking changes, and upgrade guide
    Iceberg

Upserts 1

  1. The Hudi index: how one lookup decides the cost of every upsert
    Hudi

Utilities 3

  1. Spark Submit Command Formatter tool
    Spark
  2. Spark Configuration Generator tool
    Spark
  3. Spark Submit Command generator using Iceberg Catalog
    Spark

Velox 1

  1. Native Spark execution: Apache Gluten and DataFusion Comet compared
    Spark

Views 1

  1. Iceberg views: a versioned view definition, and the cross-engine promise tested
    Iceberg

WAP 1

  1. Branching, tagging and write-audit-publish in Iceberg: CI for a table
    Iceberg

Warehouse 1

  1. Data Modeling for Data Engineers: From Foundations to Lakehouse Design
    Interview

Watermarks 1

  1. Structured Streaming internals: the checkpoint, the state store, and what a restart cannot change
    Spark

XTable 4

  1. One Hudi write, two table formats: committing Iceberg metadata inside the Hudi commit
    Hudi
  2. Apache XTable incremental sync: how to keep your conversions fast
    Lakehouse
  3. Apache XTable cheat sheet: one table, every format
    Lakehouse
  4. One table, three formats: Iceberg, Delta and Hudi interoperability
    Lakehouse

gRPC 1

  1. Spark Connect: the driver becomes a server, and your application becomes a client
    Spark