PySpark cookbook

Spark scans your file twice just to guess the column types, and sometimes it guesses wrong anyway.

The analytics team wants one row per item, but your API response nests objects inside arrays inside more objects, five levels deep.

Same customer, two rows, two different timestamps.

Your job finishes green, but the null rate quietly jumped to 30 percent and nobody noticed for weeks.

A join between your huge fact table and a tiny lookup table is shuffling every row of the big side across the network for nothing.

Why does one task run twenty minutes when the rest finish in one?

One job, three branches, the same expensive steps rerun three separate times.

Your custom Python function works, but it sends every row across the JVM boundary and the stage crawls.

100MB of data. 40,000 output files.

You know the job is slow. You don't know which stage, which task, or which join is the reason.

A ten-million-row export from the source database crawls in on one JDBC connection, one thread doing all the work, while the rest of your cluster sits there doing nothing.

One small edit. One silently broken table downstream.

Your dimension table needs to keep old versions of a row instead of overwriting them, and nothing does that automatically.