One Spark job is producing forty thousand tiny files

You need

A Spark job that already writes its result into a directory partitioned by a column with dozens or hundreds of distinct values, whose output folder holds far more files than the data size can explain.

Not covered

Compacting files a lakehouse table format already has sitting on disk, and the steady drip of small files a streaming job's own frequent micro-batches produce: both call for a scheduled compaction job running on a timer, not a one-time write fix.