New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
Your dimension table needs to keep old versions of a row instead of overwriting them, and nothing does that automatically.
What you'll have at the end
A batch job that closes the old row and inserts the new one, verified by querying both rows afterward
You need
A dimension table already shaped for Type 2 history (a surrogate key column, an effective-date range, and a current-row flag), reachable from a Spark job, plus a batch of the day's incoming source records for the same entities.
Not covered
Deciding which columns need Type 2 tracking in the first place, or handling a change that arrives several days late for an entity that already moved again since.
Preserve customer history with a Type 2 slowly changing dimension
start there first if the dimension is not shaped for Type 2 history yet; this recipe assumes that shape already exists and turns it into a running Spark job.
Deduplicate records, keep only the latest one
the same window-function trick for collapsing duplicate rows, useful any time a source sends more than one change for one entity outside this dimension job.
Checked 25 Aug 2026
Part of the PySpark cookbook