New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
Every night the whole customers table reloads, and it's starting to run past the time the next job needs to start.
What you'll have at the end
A nightly job that only pulls rows changed since its last successful run
You need
A nightly job that already pulls a whole source table into the warehouse, plus access to that table's own schema, or someone who can tell you whether it has a reliably-updated timestamp column and whether its rows ever get hard deleted.
Not covered
Reading every insert, update, and delete straight off the source's own transaction log the instant it happens; this pattern accepts a nightly lag instead.
Reconcile every row between source and target after a load
build the actual weekly reconciliation job that catches the hard deletes this recipe routes around
Make a pipeline safe to rerun with an upsert
make the incremental pull's own load step safe to rerun if it fails partway through, before the watermark decides whether it succeeded
Checked 25 Aug 2026
Part of the ETL (Extract, Transform, Load) cookbook