The small files problem, caught early

Your new job writes one file per event, and six months later the data lake is crawling.

Beginner to intermediateRead 6 minDoing: a couple of hours6 steps

What you'll have at the end

An ingestion script that buffers events into batches of a set size before writing, with the before-and-after file count written down.

Before you start

You need

A job already receiving events one at a time and writing each one out as its own file in an object store or data lake path, in a language where you can hold a short list in memory between writes.

Not covered

Running a compaction job to merge files that already exist in the lake; this fixes the writer before those files ever get created.

Leans on

6 steps

Where to go next

Choose Parquet over plain CSV

once these batches land at a steady size, the file format you write them in is the next lever if reads are still slow

Checked 25 Aug 2026

Part of the Big Data cookbook