New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
You know the job is slow. You don't know which stage, which task, or which join is the reason.
What you'll have at the end
The exact stage and the exact metric behind the slowdown, both read straight off the job's own execution plan instead of guessed from the code.
You need
A Spark batch run that already finished noticeably slower than normal, with event logging turned on so its execution record can still be reopened after the run exited.
Not covered
Fixing whichever cause turns out to be behind it, salting a skewed key, adding shuffle partitions, or forcing a broadcast, once the stage and the metric are named; each of those is a separate job in itself.
Leans on
One skewed key is stalling your Spark join
once this diagnosis actually points at skew instead of spill, the salting fix itself lives there
Drop a slow UDF for native functions
when the bottleneck stage turns out clean on shuffle and spill both, a slow per-row function inside that same stage is the next thing worth checking
Speed up a fact-to-dimension join with a broadcast join
when the plan shows a small table going through a shuffled join instead of a broadcast one, that's the fix
Checked 25 Aug 2026
Part of the PySpark cookbook