Background Job Processing Systems cookbook

10 recipes · 71 steps · about 57 minutes of reading

Hand a slow endpoint to a real task queue
FastAPI's background tasks vanish on every redeploy. I need a job that survives one.
BeginnerRead 5 minDoing: an afternoon6 steps
Make a background job idempotent
acks_late replays a crashed job, but nobody has checked what a second run does to a real charge.
Beginner to intermediateRead 4 minDoing: an afternoon6 steps
Exponential backoff stops a retry storm from hammering a flaky API
A payment API has one bad ten minutes, and a thousand jobs that all failed in the same second all retry in the same second.
IntermediateRead 5 minDoing: an afternoon7 steps
A missing dead letter queue means an exhausted job just vanishes
A job fails five times. Then it's just gone.
IntermediateRead 5 minDoing: an afternoon7 steps
Schedule a job that survives a redeploy
What happens to the nightly report the day that one server gets replaced?
IntermediateRead 5 minDoing: an afternoon6 steps
Stop a scheduled job from firing once per worker replica
Scaling a worker fleet from one replica to three quietly turned one nightly invoice email into three, and support caught it before anyone else did.
Intermediate to advancedRead 6 minDoing: an afternoon7 steps
Is your queue growing because you need more workers
Worker CPU sits near zero, the queue length keeps climbing, and nobody can say whether that means add workers or something else entirely.
Intermediate to advancedRead 7 minDoing: an afternoon8 steps
Catch the one poison message that retries forever and starves every worker
One malformed webhook payload retries every ten minutes forever, eating a worker slot every other job needs.
Intermediate to advancedRead 6 minDoing: an afternoon9 steps
Partway through a fan-out job, one crash wipes out everything
One task resizes every image in a batch. A single crash partway through loses the whole thing.
AdvancedRead 6 minDoing: an afternoon8 steps
Drain your workers on deploy without losing the job mid-flight
A deploy sends SIGTERM to every worker. Whatever task was mid-run either finishes or vanishes, depending on luck.
AdvancedRead 8 minDoing: an afternoon7 steps

Cookbook badge

Finish all 10 recipes to earn this.

In these roadmaps

Backend Developerstep 11 of 15

Background Job Processing Systems cookbook

Hand a slow endpoint to a real task queue

Make a background job idempotent

Exponential backoff stops a retry storm from hammering a flaky API

A missing dead letter queue means an exhausted job just vanishes

Schedule a job that survives a redeploy

Stop a scheduled job from firing once per worker replica

Is your queue growing because you need more workers

Catch the one poison message that retries forever and starves every worker

Partway through a fan-out job, one crash wipes out everything

Drain your workers on deploy without losing the job mid-flight

In these roadmaps

Background Job Processing Systems cookbook

Hand a slow endpoint to a real task queue

Make a background job idempotent

Exponential backoff stops a retry storm from hammering a flaky API

A missing dead letter queue means an exhausted job just vanishes

Schedule a job that survives a redeploy

Stop a scheduled job from firing once per worker replica

Is your queue growing because you need more workers

Catch the one poison message that retries forever and starves every worker

Partway through a fan-out job, one crash wipes out everything

Drain your workers on deploy without losing the job mid-flight

In these roadmaps