ETL (Extract, Transform, Load) cookbook

The API only hands back fifty rows at a time and you need all of them.

Why does the report show more orders than the store actually took?

Every night the whole customers table reloads, and it's starting to run past the time the next job needs to start.

The job died at row eight thousand, and rerunning it just doubled everything that already loaded.

Loading a million rows one INSERT at a time is going to take all night.

A source added a column last week and nobody told you until a report broke.

Ten thousand rows load fine until row four thousand has a bad date, and the whole job dies right there.

Somebody asked if the load actually got everything, and all you had was a guess.

The transform looks right on the ten rows you checked by eye, but you have no idea if it still works after the next change.

Analysts need the order data, and nobody wants raw customer emails sitting in a table everyone can query.

A bug in last month's transform means three weeks of numbers are wrong, and today's job can't stop while you fix it.