New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
Your job finishes green, but the null rate quietly jumped to 30 percent and nobody noticed for weeks.
What you'll have at the end
A validation step that raises and stops the job when row counts or null rates cross a set line
You need
A PySpark job that already reads today's batch into a DataFrame, runs its transforms, and is right up against the point where it writes the result out, plus one column in that data that should almost never be empty and a small place to store one number between runs, such as a row in a table or a value in a state file.
Not covered
Picking a dedicated data quality framework to declare these checks instead of writing them by hand, and deciding how the job pages or messages anyone once it stops.
Leans on
Give a big CSV import an explicit schema instead of a guess
start here first if rows are arriving with the wrong shape or types; this recipe assumes the shape is already right and only watches volume and completeness.
Write a pytest suite that catches a broken PySpark transformation
reach for this to catch a bug in your own transform code before it ships; this recipe catches bad data arriving from outside your code, not a mistake inside it.
Checked 25 Aug 2026
Part of the PySpark cookbook