Stream a log file too big to fit in memory

The log file you need to search through is bigger than the memory your script is allowed to use.

IntermediateRead 6 minDoing: about 40 minutes5 steps

What you'll have at the end

A script that finds every line matching a pattern in a multi-gigabyte log file and writes them out, using a peak memory footprint well under a tenth of the file's own size.

Before you start

You need

A log file already sitting on disk, too large to comfortably read into memory in one go, and a plain string or short pattern you need to find every occurrence of.

Not covered

Watching a log file for new lines as it keeps growing, which needs different logic from searching one that has already finished growing.

Leans on

5 steps

Where to go next

Two heap snapshots and the leak they were hiding

when memory keeps climbing in a service that runs for days instead of one file being too big for a single pass.

Checked 25 Aug 2026

Part of the Node.js cookbook