Streaming into Iceberg: availableNow, checkpoints, and the snapshot count nobody watches

• What a streaming write actually commits
• The run, measured
• Why re-running is safe
• Choosing the trigger
• The cost that shows up in week three
• The trap that produces a healthy-looking no-op
• Row-level operations alongside a stream
• Common misconceptions
• A model worth keeping
• References
• Trademarks

TL;DR

• A streaming write commits one Iceberg snapshot per micro-batch, not per file and not per row. Three input files consumed in one availableNow run produced 1 snapshot; a fourth file arriving later produced a second.
• Trigger.AvailableNow drains everything currently available, then stops. It is the batch-shaped way to run a streaming job, and it is what you want in an orchestrator.
• Re-running the same query over the same input added 0 rows. The checkpoint, not the table, is what makes that safe.
• The trigger interval is therefore a metadata-growth decision. A one-minute trigger is 1,440 snapshots a day, each with its own manifest list.
• The streaming file source does not recurse into subdirectories. Files staged one directory deep are silently never read, and the query reports healthy.

What a streaming write actually commits

Spark’s Structured Streaming writes to Iceberg through a normal sink, and the unit of work is the micro-batch. Each batch that produces data ends in an Iceberg commit, which means a new snapshot, a new manifest list, and a new metadata file.

That single fact explains most of what goes right and wrong with streaming Iceberg tables. It is not “streaming” in the sense of rows trickling into a file. It is a sequence of small, atomic batch commits, and everything about the table’s long-term health follows from how many of them you make.

flowchart LR
    F[new files arrive] --> MB[micro-batch]
    MB --> W[write data files]
    W --> C["commit: snapshot + manifest list + metadata.json"]
    C --> CK[checkpoint records progress]
    CK --> MB

The run, measured

Three Parquet files staged in a directory, a table created ahead of time, and a query with availableNow:

schema = StructType([StructField("id", LongType()), StructField("v", StringType())])

q = (spark.readStream.schema(schema).parquet(src)
     .writeStream.format("iceberg").outputMode("append")
     .trigger(availableNow=True)
     .option("checkpointLocation", "/work/ckpt")
     .toTable("local.db.stream"))
q.awaitTermination()

rows after first availableNow run = 3000
snapshots                         = 1
snapshot ops                      = ['append']

Three files, 3,000 rows, one snapshot. availableNow did not create a snapshot per file; it consumed everything available and committed once. That is the behaviour you want for a scheduled job, and the opposite of what a short processing-time trigger gives you.

Running the identical query a second time, with no new input:

rows after re-run, no new files   = 3000

Nothing was added and nothing was duplicated. Then a fourth file appeared:

rows after a 4th file arrives     = 4000
snapshots now                     = 2
data files in table               = 3

A second snapshot, for a second run that found work. The snapshot count tracks runs that found data, not files and not rows.

Why re-running is safe

The checkpoint is doing that work, and it is worth knowing what it holds:

checkpoint dirs = ['commits', 'metadata', 'offsets', 'sources']

offsets records what each batch was told to read, before it ran. commits records which batches finished. sources tracks which files the file source has already seen. On restart, the engine compares the two logs, re-runs anything that started and did not finish, and skips everything already committed.

The table is not what makes this idempotent. Point the same query at a new checkpoint directory and it will happily re-ingest every file and double your rows. The checkpoint is the record of progress; the table is just where the output went.

Two consequences follow. A checkpoint belongs to one query against one table — reusing it across two different sinks corrupts both notions of progress. And deleting a checkpoint to “reset” a job is a data-duplication event, not a clean slate, unless you also truncate the target.

The full anatomy of that directory, including why the offset log pins configuration, is in Structured Streaming internals.

Choosing the trigger

This is the decision that determines whether the table stays healthy.

availableNow is the right default for most pipelines. It turns a streaming query into something an orchestrator can schedule: it starts, it catches up, it exits with a status. You get the incremental-processing semantics of streaming with the operational shape of a batch job, and you commit once per run rather than once per micro-batch.

Reach for a continuous processing-time trigger when latency genuinely matters, and then accept that you have signed up for the maintenance in the next section.

The cost that shows up in week three

Every commit writes a metadata file, a manifest list, and at least one manifest. At one snapshot a minute, a table accumulates 1,440 of each per day. Nothing fails. Queries just get slower, because planning a query means reading the current metadata file, its manifest list, and the manifests it points at.

The symptom is distinctive: query planning time grows while the data volume is flat, and the slowdown affects every query against the table equally, including trivial ones.

Two jobs keep it in check, and a streaming table needs both scheduled from the day it is created:

Compaction merges the many small files each micro-batch produced into fewer large ones. Without it, a table ingesting every minute accumulates a file per partition per minute.

Snapshot expiry removes old snapshots and the files only they reference. This is the one that controls storage, and compaction without expiry grows the bill while shrinking the file count. The mechanics of both are in table maintenance for Iceberg and Hudi.

There is a third knob worth knowing: write.metadata.previous-versions-max and write.metadata.delete-after-commit.enabled bound how many old metadata JSON files are kept, which is separate from snapshot expiry and is the specific fix for a metadata directory with tens of thousands of files in it.

The trap that produces a healthy-looking no-op

Staging the input files one directory deep produced this:

rows after availableNow run = 0
snapshots created           = 0
checkpoint dirs             = ['commits', 'metadata', 'offsets', 'sources']

The query started, created its checkpoint, ran, and terminated normally. It read nothing, because the streaming file source lists files in the given path and does not recurse into subdirectories. The files were in src/batch0/part-*.parquet and the source was watching src/.

Nothing in that output says “I found no input”. The checkpoint exists, the query exited zero, and only the row count reveals the problem. This was my own mistake while writing this post, and it is worth naming because a silent zero-row ingestion is exactly the failure that survives into production unnoticed.

Two defences. Stage input flat in the watched directory, which is what the measured run above does. And assert on the row count or the snapshot count after the run, rather than on the exit status.

Row-level operations alongside a stream

A streaming append is the simple case. Streaming upserts go through foreachBatch and MERGE INTO, because the sink itself only appends:

def upsert(batch_df, batch_id):
    batch_df.createOrReplaceTempView("changes")
    batch_df.sparkSession.sql("""
        MERGE INTO local.db.target t USING changes s ON t.id = s.id
        WHEN MATCHED THEN UPDATE SET *
        WHEN NOT MATCHED THEN INSERT *""")

query = (stream.writeStream.foreachBatch(upsert)
         .trigger(availableNow=True)
         .option("checkpointLocation", "/work/ckpt_merge").start())

Note that foreachBatch gives at-least-once by default: the same batch_id can run twice after a failure. A MERGE keyed on a primary key is naturally idempotent, which is why this pattern works — an append inside foreachBatch would not be.

The delete-file accumulation that this pattern causes on a merge-on-read table is its own problem, covered in copy-on-write or merge-on-read.

Common misconceptions

“Streaming writes row by row.” It writes micro-batches, each an atomic Iceberg commit.

“One snapshot per file.” One snapshot per micro-batch that produced data. Three files in one availableNow run gave one snapshot.

“Re-running will duplicate data.” Not with the same checkpoint. With a new checkpoint, it certainly will.

“availableNow is not really streaming.” It uses the streaming engine, offsets and checkpoint, and processes only what is new. It just stops when it catches up.

“Small files are the only streaming problem.” Metadata files accumulate on the same schedule and are bounded by different settings.

A model worth keeping

A streaming write into Iceberg is a loop of small batch commits. The trigger sets how often you commit; the checkpoint makes re-running safe; the snapshot count is the bill.

So choose availableNow unless latency requires otherwise, keep the checkpoint as carefully as you keep the table, and schedule compaction and expiry on the same day you create the table rather than the week queries get slow.

References

• Iceberg Spark structured streaming for the sink options and write behaviour
• Structured Streaming programming guide for triggers and checkpoint semantics
• Iceberg table properties for write.metadata.previous-versions-max and the metadata retention settings
• Structured Streaming internals for what the checkpoint directory holds
• Table maintenance for Iceberg and Hudi for the compaction and expiry this post assumes

Trademarks

Apache Iceberg, Apache Spark, Apache Parquet, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.