New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
A streaming job that ran fine for months just started failing every few minutes, and nobody touched the code.
What you'll have at the end
The scheduled compaction job identified as the real source of the failures, its scope narrowed to skip the partition still under active write, its retry budget raised as a backstop, and several days of scheduled runs completing with zero commit failures.
You need
A Spark Structured Streaming job appending into an Iceberg table partitioned by date, and a separate scheduled job that runs table compaction (rewrite_data_files) against the same table, with access to both jobs' own logs and the compaction job's schedule.
Not covered
Fixing genuine data conflicts from concurrent UPDATE, DELETE, or MERGE traffic landing on the same rows compaction is rewriting, or enabling partial-progress commits for a single very long compaction run, both separate tuning problems from a plain append job colliding with an unscoped compaction call.
Leans on
Copy-on-write or merge-on-read: pick the write mode for a table with constant updates
if the collision is UPDATEs, DELETEs, or a MERGE landing on the same rows compaction is rewriting rather than a plain append job, the fix is usually a write-mode change, not a scope filter or a retry setting.
A lakehouse query keeps slowing down as manifest bloat builds up
once commits stop failing, a table fed by frequent small batches usually still needs its manifests rewritten separately, since that's a different maintenance job from the one this recipe fixes.
Checked 26 Aug 2026
Part of the Data Lakehouse cookbook