Pull every page from an API that paginates

The API only hands back fifty rows at a time and you need all of them.

BeginnerRead 5 minDoing: an hour5 steps

▶ Start

What you'll have at the end

A script that pulls every page and stops with the right total row count

Before you start

You need

A working authenticated call to the API's list endpoint that already returns one page of records.

Not covered

What happens to the pulled rows once they land, a full reload, an incremental load, or an upsert, is a separate loading decision.

Leans on

Reading the API's own signal

Find how the API signals more pages
Fetch one page and read the next-page signal

Building a pull that survives real traffic

Loop the fetch until the signal ends
Wait out a rate limit and resume

Confirming every row arrived

Check the final count against the source

Where to go next

Choose full reload or incremental for a new source

once every row can be pulled, decide whether each run should reload everything or fetch only what changed since last time.

Make a pipeline safe to rerun with an upsert

if a pull dies partway through and gets rerun from scratch, use this to avoid loading the same rows twice.

Checked 25 Aug 2026

Part of the ETL (Extract, Transform, Load) cookbook