Archive

68 posts across 1 years, ordered from basic to advanced.

202668

  1. Spark 4, one minor release at a time: what each feature is, its syntax, and a real run
    Spark #Spark #SQL #Streaming #PySpark
  2. One Hudi write, two table formats: committing Iceberg metadata inside the Hudi commit
    Hudi #Hudi #Iceberg #XTable #Interoperability
  3. System Design for Data Engineers: A Framework, the Building Blocks, and Worked Designs
    Interview #Pipelines #Streaming #Lakehouse #Kafka
  4. 50 job websites for remote and tech roles, and how to use them well
    Career #Remote #Jobs #Search #ATS
  5. Top Data engineer interview questions
    Interview #SQL #Spark #Streaming #Design
  6. Apache Iceberg 1.12.0: what's new, breaking changes, and upgrade guide
    Iceberg #Iceberg #Lakehouse #Spark #Flink #KafkaConnect #Release #Upgrade
  7. Data Modeling for Data Engineers: From Foundations to Lakehouse Design
    Interview #Modeling #Warehouse #Lakehouse #Spark
  8. Spark on Kubernetes on your Mac: kind, spark-submit, the Spark Operator and Iceberg
    Spark #Spark #Kubernetes #Operator #Iceberg
  9. Apache Spark architecture: what actually happens when you call an action
    Spark #Spark #Architecture #Internals #Shuffle #Scheduling
  10. Apache Spark: the complete cheat sheet
    Spark #Spark #SQL #Tuning #Streaming #Reference
  11. Open table formats in practice: what actually differs between Iceberg, Hudi and Delta Lake
    Lakehouse #Iceberg #Hudi #Delta #Lakehouse #Spark
  12. Run every demo on this blog locally: one compose stack for Hudi, Iceberg and Delta
    Lakehouse #Docker #Iceberg #Hudi #Delta
  13. What Hive tables could not do, and why Iceberg exists
    Iceberg #Iceberg #Hive #Metadata #Lakehouse
  14. Apache Hudi: an introduction to its architecture, features and what makes it different
    Hudi #Hudi #Lakehouse #Spark #Indexing #Streaming
  15. A local Iceberg playground: Spark, MinIO and a REST catalog in one compose file
    Iceberg #Iceberg #MinIO #Docker #Catalog
  16. Inspecting Parquet files from the command line with parquet-cli
    Tools #Tools #Parquet
  17. Spark Submit Command Formatter tool
    Tool Spark #Spark #Utilities
  18. Spark Configuration Generator tool
    Tool Spark #Spark #Utilities #Generator
  19. The three tiers of an Iceberg table: catalog, metadata, data
    Iceberg #Iceberg #Metadata #Manifests #Architecture
  20. Apache Iceberg on Spark: the complete cheat sheet
    Iceberg #Iceberg #Lakehouse #Spark #PySpark #Reference
  21. Inside Spark's Catalyst optimizer: how a query gets rewritten, rule by rule
    Spark #Spark #Catalyst #Optimizer #Internals #Codegen
  22. Adaptive query execution in Apache Spark: what it re-plans, and what it leaves alone
    Spark #Spark #AQE #Performance #Tuning #Internals
  23. Apache Spark joins in depth: the five strategies and how one gets picked
    Spark #Spark #Joins #Catalyst #Performance #Internals
  24. Spark memory management: every region, and the arithmetic that sizes it
    Spark #Spark #Memory #Tuning #Internals #Performance
  25. Apache Hudi architecture: what actually happens when you upsert a row
    Hudi #Hudi #Architecture #Internals #Compaction #Indexing
  26. The Hudi index: how one lookup decides the cost of every upsert
    Hudi #Hudi #Indexing #Metadata #Upserts
  27. Hudi Merge-on-Read: the write you skipped, and the read that pays for it
    Hudi #Hudi #Compaction #Timeline #Concurrency
  28. Apache Hudi on Spark: the complete cheat sheet
    Hudi #Hudi #Lakehouse #Spark #PySpark #Reference
  29. Apache Iceberg architecture: what actually happens when you commit
    Iceberg #Iceberg #Architecture #Internals #Snapshots #Catalog
  30. Iceberg catalogs: the one pointer that makes a table a table
    Iceberg #Iceberg #Catalog #REST #Governance
  31. Life of a write query in Iceberg: three files, one pointer swap, and what ACID means here
    Iceberg #Iceberg #Commits #Concurrency #Transactions
  32. Life of a read query in Iceberg: four prunes before a byte of data is read
    Iceberg #Iceberg #Pruning #Metadata #Query
  33. Iceberg format versions: what v1, v2, v3 and v4 actually change
    Iceberg #Iceberg #Deletes #Lineage #Compatibility
  34. Schema evolution in Iceberg: why a rename costs nothing
    Iceberg #Iceberg #Schema #Evolution #Compatibility
  35. Hidden partitioning and partition evolution: changing layout without rewriting history
    Iceberg #Iceberg #Partitioning #Pruning #Transforms
  36. Loading data into Iceberg: appends, overwrites and row-level SQL
    Iceberg #Iceberg #Ingestion #Merge #Overwrite
  37. Apache XTable incremental sync: how to keep your conversions fast
    Lakehouse #XTable #Hudi #Iceberg #Delta #Lakehouse
  38. Apache XTable cheat sheet: one table, every format
    Lakehouse #XTable #Hudi #Iceberg #Delta #Paimon #Reference
  39. Spark shuffle internals: the three writers, and which one your job gets
    Spark #Spark #Shuffle #Serialization #Internals
  40. Data skew in Apache Spark: how to see it, and nine ways to fix it
    Spark #Spark #Skew #Performance #Joins #Tuning
  41. Spark performance tuning: seventeen techniques, and how to prove each one worked
    Spark #Spark #Performance #Joins #AQE
  42. Native Spark execution: Apache Gluten and DataFusion Comet compared
    Spark #Spark #Gluten #Comet #Velox
  43. Every Apache Spark release, and the problem each one was built to solve
    Spark #Spark #Releases #Migration #Defaults #Internals
  44. Spark Connect: the driver becomes a server, and your application becomes a client
    Spark #SparkConnect #gRPC #Arrow #PySpark
  45. Structured Streaming internals: the checkpoint, the state store, and what a restart cannot change
    Spark #Spark #Streaming #State #Watermarks
  46. Streaming into Iceberg: availableNow, checkpoints, and the snapshot count nobody watches
    Iceberg #Iceberg #Streaming #Checkpoints #Snapshots
  47. Copy-on-write or merge-on-read: choosing how an Iceberg table pays for a delete
    Iceberg #Iceberg #Deletes #Compaction #Streaming
  48. CDC into Iceberg: the delete files nobody counted
    Iceberg #Iceberg #CDC #Deletes #Compaction
  49. Time travel and rollback in Iceberg: undoing a bad write in one statement
    Iceberg #Iceberg #Snapshots #Recovery #History
  50. Apache Iceberg through Spark Connect: the full surface, and the extension that breaks time travel
    Iceberg #Spark #SparkConnect #Iceberg #Lakehouse
  51. Apache Hudi through Spark Connect: running the full table surface from a thin client
    Hudi #Spark #SparkConnect #Hudi #Lakehouse
  52. Running Iceberg on Apache Polaris: one compose file, two storage backends
    Iceberg #Iceberg #Polaris #Catalog #Spark #MinIO
  53. Hudi key generators: the one decision you cannot change later
    Hudi #Hudi #KeyGenerator #Partitioning #Configuration
  54. Spark Submit Command generator using Iceberg Catalog
    Tool Spark #Spark #Utilities #Iceberg
  55. Branching, tagging and write-audit-publish in Iceberg: CI for a table
    Iceberg #Iceberg #Branching #Tagging #WAP #Production
  56. Table maintenance for Iceberg and Hudi: compaction, expiry, and the space you did not reclaim
    Lakehouse #Iceberg #Hudi #Compaction #Cleaning
  57. Concurrency control in Hudi: the lock provider that protects nothing
    Hudi #Hudi #Concurrency #Locking #Timeline #Production
  58. One table, three formats: Iceberg, Delta and Hudi interoperability
    Lakehouse #Iceberg #Delta #Hudi #XTable
  59. Iceberg views: a versioned view definition, and the cross-engine promise tested
    Iceberg #Iceberg #Views #Catalogs #Trino #Interoperability
  60. Migrating Hive tables to Iceberg: snapshot first, migrate second
    Iceberg #Iceberg #Migration #Hive #Parquet
  61. Sorting and clustering in Iceberg: what min/max statistics can and cannot skip
    Iceberg #Iceberg #Performance #Sorting #Clustering #Maintenance
  62. Running Iceberg in production: file sizes, object-storage paths, and the jobs you must schedule
    Iceberg #Iceberg #Production #Tuning #Storage
  63. Incremental processing in Hudi: reading only what changed, and the timestamp that betrays you
    Hudi #Hudi #Incremental #CDC #Timeline #Streaming
  64. Spark and Hudi logging: making a write explain itself
    Hudi #Hudi #Spark #Logging #Log4j #Troubleshoot
  65. A Spark JVM playbook: custom logging, class loading, stack size and proxies
    Spark #Spark #Troubleshoot #Logging
  66. Logging a Spark stream: the job you cannot restart to debug
    Spark #Spark #Streaming #Logging #Log4j
  67. Kerberos Setup in Linux
    Linux #Linux #Kerberos #Security
  68. Top Apache Spark interview questions, with answers
    Interview #Spark #Interview #Reference #Scenarios