Background Job Processing Systems Cheat Sheet

Updated 2026-05-28

Next Topic: Bun JavaScript Runtime Cheat Sheet

🧠Study flashcards on this topic143 cards · spaced repetition→

Background job processing systems enable applications to offload time-consuming tasks (email sending, image processing, data exports, API calls) to separate worker processes that execute asynchronously from the main request-response cycle. These systems prevent user-facing operations from blocking on long-running work by placing jobs into persistent queues backed by Redis, PostgreSQL, RabbitMQ, or cloud services like AWS SQS. Understanding job queue architectures, retry strategies, priority scheduling, and monitoring patterns is essential for building scalable, resilient systems that handle millions of tasks daily while gracefully managing failures. A well-designed background job system transforms a slow, brittle application into one that feels fast and handles spikes in load without degradation — every modern web application processing significant workload relies on one.

What This Cheat Sheet Covers

This topic spans 15 focused tables and 160 indexed concepts, 143 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Job Queue Frameworks by LanguageTable 2: Message Brokers and Queue BackendsTable 3: Job Queue Architectural PatternsTable 4: Retry and Failure Handling StrategiesTable 5: Job Scheduling and TimingTable 6: Concurrency and Worker ManagementTable 7: Message Delivery Guarantees and SemanticsTable 8: Job Queue Monitoring and ObservabilityTable 9: Advanced Queue Features and PatternsTable 10: FIFO and Ordering GuaranteesTable 11: Job Queue Antipatterns and PitfallsTable 12: Cloud-Native Job ProcessingTable 13: Job Queue Performance OptimizationTable 14: Workflow Orchestration PlatformsTable 15: Real-World Use Cases

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Job Queue Frameworks by Language

The ecosystem of job queue libraries spans every major backend language; choosing the right one determines your operational model, performance ceiling, and infrastructure requirements. Most frameworks offer a spectrum from Redis-backed in-memory speed to database-backed ACID guarantees — with managed cloud platforms now offering a third path requiring no broker infrastructure at all.

Framework	Example	Description
Sidekiq	`class EmailJob` `include Sidekiq::Job` `def perform(user_id)` `User.find(user_id).send_welcome` `end` `end` `EmailJob.perform_async(123)`	• Ruby's most popular background job processor • multi-threaded (handles many jobs per process), uses Redis for storage, supports retries and scheduled jobs • powers Shopify, GitHub, and thousands of Rails apps.
BullMQ	`const queue = new Queue('emails');` `await queue.add('send', {` `userId: 123, type: 'welcome'` `});`	• Node.js and Bun job queue built on Redis with priorities, rate limiting, delayed jobs, parent-child workflows, and events • successor to Bull with better TypeScript support and performance.
Celery	`@app.task` `def send_email(user_id):` `user = User.objects.get(id=user_id)` `user.send_welcome()` `send_email.delay(123)`	• Python's distributed task queue standard • supports multiple brokers (RabbitMQ, Redis, SQS), result backends, periodic tasks with Celery Beat, and canvas workflows for complex orchestration.
Hangfire	`BackgroundJob.Enqueue(() =>` `SendEmail(123));` `RecurringJob.AddOrUpdate(` `"cleanup", () => Cleanup(),` `Cron.Daily);`	• .NET background job processor integrated with ASP.NET • uses SQL Server, PostgreSQL, or Redis for persistence, includes built-in dashboard, supports fire-and-forget, delayed, recurring, and continuation jobs.
Temporal	`@workflow.run` `async def order_workflow(order_id):` `await charge_payment(order_id)` `await ship_order(order_id)` `await send_notification(order_id)`	• Durable execution platform for workflows and long-running processes • uses event sourcing to capture state at every step, enabling automatic retries from exact failure point • handles multi-step sagas, human-in-the-loop, and distributed orchestration • supports Go, Java, Python, TypeScript, PHP.
Inngest	`inngest.createFunction(` `{ id: "send-email" },` `{ event: "user.created" },` `async ({ event }) => {` `await sendEmail(event.data);` `}` `);`	• Event-driven workflow platform for TypeScript/JavaScript • serverless-native with automatic retries, step functions, and durable execution • handles long-running workflows without managing infrastructure
Trigger.dev	`task("send-email", async (payload) => {` `await sendEmail(payload.userId);` `});`	• Open-source TypeScript background job platform with no timeouts — runs on long-lived compute, not serverless functions • automatic retries, real-time observability, elastic scaling, and support for AI/agent workflows • self-hostable or managed cloud.
Laravel Queue	`class SendEmailJob` `implements ShouldQueue {` `public function handle() {` `Mail::to($this->user)->send(...);` `}` `}` `SendEmailJob::dispatch($user);`	• PHP/Laravel's built-in async job system supporting Redis, SQS, database, and Beanstalkd drivers • job chaining, batching, rate limiting, and graceful failure handling built in • monitored via Laravel Horizon (Redis dashboard).
Active Job	`class EmailJob < ApplicationJob` `queue_as :emails` `def perform(user_id)` `User.find(user_id).send_email` `end` `end` `EmailJob.perform_later(123)`	• Rails' built-in abstraction layer for background jobs • doesn't process jobs itself but provides unified API for Sidekiq, Solid Queue, GoodJob, Resque, Delayed Job, and others • enables adapter-agnostic job code.
Asynq	`client := asynq.NewClient(...)` `task := asynq.NewTask("email:send",` `payload, asynq.MaxRetry(5))` `client.Enqueue(task)`	• Go task queue backed by Redis • supports weighted priority queues, strict priority queues, unique tasks, periodic tasks, task aggregation, and Prometheus metrics • includes Asynqmon web UI and CLI.
River	`riverClient.InsertTx(ctx, tx,` `SendEmailArgs{UserID: user.ID},` `nil)`	• Go background job library backed by PostgreSQL • transactional enqueueing — job is committed in same DB transaction as your business logic, guaranteeing no lost jobs • no external services required beyond Postgres.
JobRunr	`BackgroundJob.enqueue(` `() -> myService.sendEmail(userId));` `BackgroundJob.scheduleRecurrently(` `Cron.daily(), () -> report());`	• Java distributed background job library; uses any existing database (Postgres, SQL Server, Oracle) as storage • simple lambda-based API, real-time dashboard, automatic retries, and carbon-aware scheduling (v8+).
RQ (Redis Queue)	`from rq import Queue` `q = Queue(connection=redis_conn)` `job = q.enqueue(send_email, 123)`	• Lightweight Python job queue using Redis • simpler than Celery with fewer features; uses fork-based workers • includes dashboard via rq-dashboard; good for small to medium Python apps.
GoodJob	`class NotifyJob < ApplicationJob` `def perform(user_id)` `User.find(user_id).notify` `end` `end` `NotifyJob.perform_later(123)`	• Rails Active Job adapter using PostgreSQL for storage • multithread, multi-process execution; includes web dashboard and cron-style recurring jobs • no Redis required; ACID guarantees for job atomicity.
Solid Queue	`# config/queue.yml` `default:` `workers: 3` `queues: [default, mailers]`	• Rails-official Active Job backend using SQLite or PostgreSQL • ships with Rails 8; designed for simpler deployments without Redis • supports priorities, recurring jobs, and concurrency controls.
Resque	`class EmailJob` `@queue = :emails` `def self.perform(user_id)` `User.find(user_id).send_email` `end` `end` `Resque.enqueue(EmailJob, 123)`	• Redis-backed Ruby library using forked processes instead of threads • simpler than Sidekiq but less efficient (one job per process) • includes web UI; largely superseded by Sidekiq in modern Rails apps.
delayed_job	`class WelcomeMailer` `handle_asynchronously :deliver` `end`	• Rails background job system storing jobs in the database via ActiveRecord • single-threaded polling model; lower throughput than Redis-backed queues • no external dependencies; still used in smaller or legacy Rails apps.
Quartz.NET	`IJobDetail job = JobBuilder` `.Create<ReportJob>().Build();` `ITrigger trigger = TriggerBuilder` `.Create().WithCronSchedule(` `"0 0 3 * * ?").Build();`	• .NET job scheduling library for cron-style recurring jobs • stores schedules in SQL Server, PostgreSQL, or in-memory; thread pool execution with clustering support • often combined with Hangfire for full background processing.
Kue (deprecated)	`const queue = kue.createQueue();` `queue.create('email', {` `userId: 123` `}).save();`	• Node.js priority job queue backed by Redis with web UI • no longer actively maintained; BullMQ is the recommended replacement • still found in legacy Node apps.

Table 2: Message Brokers and Queue Backends

The broker or storage layer underneath your job framework determines throughput, durability, delivery guarantees, and operational complexity. Redis offers the fastest in-memory performance while PostgreSQL provides ACID consistency without extra services; managed cloud queues (SQS, Pub/Sub) remove infrastructure entirely at the cost of vendor lock-in.

System	Example	Description
Redis	`LPUSH jobs:email '{"user_id":123}'` `BRPOP jobs:email 0`	• In-memory data store most commonly used as job queue backend • extremely fast (microsecond latency) but requires persistence configuration (AOF/RDB) to prevent job loss on crashes • supports lists, sorted sets, and streams.

The rest of this section is part of the full cheat sheet

Table 3: Job Queue Architectural Patterns

Architectural patterns capture the recurring structural decisions that determine how work flows through a job system. Moving from the simplest (fire-and-forget) to the most sophisticated (distributed saga orchestration), each pattern solves a distinct class of problem — knowing when to use which one prevents both over-engineering and under-engineering.

Pattern	Example	Description
Fire-and-forget	`BackgroundJob.Enqueue(() =>` `ProcessOrder(orderId))`	• Enqueue job and immediately return; simplest pattern for tasks like sending emails, logging, or cache warming • job executed asynchronously by worker pool with no waiting for result.

The rest of this section is part of the full cheat sheet

Table 4: Retry and Failure Handling Strategies

Failures are inevitable in distributed systems — transient network errors, rate-limited APIs, and temporary outages are the norm, not the exception. Retry strategies determine whether failures are resolved gracefully or spiral into cascading outages; the key insight is that not all failures should be retried, and retry storms can be worse than the original failure.

Strategy	Example	Description
Exponential backoff	Retry delays: 1s, 2s, 4s, 8s, 16s `retry_delay = base_delay ` `(2 * attempt)`	• Progressively increase wait time between retries • reduces load on failing service, giving it time to recover; prevents thundering herd • standard pattern for transient failures (network blips, rate limits, temporary outages).

The rest of this section is part of the full cheat sheet

Table 1: Job Queue Frameworks by Language

Framework	Example	Description
Sidekiq	`class EmailJob` `include Sidekiq::Job` `def perform(user_id)` `User.find(user_id).send_welcome` `end` `end` `EmailJob.perform_async(123)`	• Ruby's most popular background job processor • multi-threaded (handles many jobs per process), uses Redis for storage, supports retries and scheduled jobs • powers Shopify, GitHub, and thousands of Rails apps.
BullMQ	`const queue = new Queue('emails');` `await queue.add('send', {` `userId: 123, type: 'welcome'` `});`	• Node.js and Bun job queue built on Redis with priorities, rate limiting, delayed jobs, parent-child workflows, and events • successor to Bull with better TypeScript support and performance.
Celery	`@app.task` `def send_email(user_id):` `user = User.objects.get(id=user_id)` `user.send_welcome()` `send_email.delay(123)`	• Python's distributed task queue standard • supports multiple brokers (RabbitMQ, Redis, SQS), result backends, periodic tasks with Celery Beat, and canvas workflows for complex orchestration.
Hangfire	`BackgroundJob.Enqueue(() =>` `SendEmail(123));` `RecurringJob.AddOrUpdate(` `"cleanup", () => Cleanup(),` `Cron.Daily);`	• .NET background job processor integrated with ASP.NET • uses SQL Server, PostgreSQL, or Redis for persistence, includes built-in dashboard, supports fire-and-forget, delayed, recurring, and continuation jobs.
Temporal	`@workflow.run` `async def order_workflow(order_id):` `await charge_payment(order_id)` `await ship_order(order_id)` `await send_notification(order_id)`	• Durable execution platform for workflows and long-running processes • uses event sourcing to capture state at every step, enabling automatic retries from exact failure point • handles multi-step sagas, human-in-the-loop, and distributed orchestration • supports Go, Java, Python, TypeScript, PHP.
Inngest	`inngest.createFunction(` `{ id: "send-email" },` `{ event: "user.created" },` `async ({ event }) => {` `await sendEmail(event.data);` `}` `);`	• Event-driven workflow platform for TypeScript/JavaScript • serverless-native with automatic retries, step functions, and durable execution • handles long-running workflows without managing infrastructure
Trigger.dev	`task("send-email", async (payload) => {` `await sendEmail(payload.userId);` `});`	• Open-source TypeScript background job platform with no timeouts — runs on long-lived compute, not serverless functions • automatic retries, real-time observability, elastic scaling, and support for AI/agent workflows • self-hostable or managed cloud.
Laravel Queue	`class SendEmailJob` `implements ShouldQueue {` `public function handle() {` `Mail::to($this->user)->send(...);` `}` `}` `SendEmailJob::dispatch($user);`	• PHP/Laravel's built-in async job system supporting Redis, SQS, database, and Beanstalkd drivers • job chaining, batching, rate limiting, and graceful failure handling built in • monitored via Laravel Horizon (Redis dashboard).
Active Job	`class EmailJob < ApplicationJob` `queue_as :emails` `def perform(user_id)` `User.find(user_id).send_email` `end` `end` `EmailJob.perform_later(123)`	• Rails' built-in abstraction layer for background jobs • doesn't process jobs itself but provides unified API for Sidekiq, Solid Queue, GoodJob, Resque, Delayed Job, and others • enables adapter-agnostic job code.
Asynq	`client := asynq.NewClient(...)` `task := asynq.NewTask("email:send",` `payload, asynq.MaxRetry(5))` `client.Enqueue(task)`	• Go task queue backed by Redis • supports weighted priority queues, strict priority queues, unique tasks, periodic tasks, task aggregation, and Prometheus metrics • includes Asynqmon web UI and CLI.
River	`riverClient.InsertTx(ctx, tx,` `SendEmailArgs{UserID: user.ID},` `nil)`	• Go background job library backed by PostgreSQL • transactional enqueueing — job is committed in same DB transaction as your business logic, guaranteeing no lost jobs • no external services required beyond Postgres.
JobRunr	`BackgroundJob.enqueue(` `() -> myService.sendEmail(userId));` `BackgroundJob.scheduleRecurrently(` `Cron.daily(), () -> report());`	• Java distributed background job library; uses any existing database (Postgres, SQL Server, Oracle) as storage • simple lambda-based API, real-time dashboard, automatic retries, and carbon-aware scheduling (v8+).
RQ (Redis Queue)	`from rq import Queue` `q = Queue(connection=redis_conn)` `job = q.enqueue(send_email, 123)`	• Lightweight Python job queue using Redis • simpler than Celery with fewer features; uses fork-based workers • includes dashboard via rq-dashboard; good for small to medium Python apps.
GoodJob	`class NotifyJob < ApplicationJob` `def perform(user_id)` `User.find(user_id).notify` `end` `end` `NotifyJob.perform_later(123)`	• Rails Active Job adapter using PostgreSQL for storage • multithread, multi-process execution; includes web dashboard and cron-style recurring jobs • no Redis required; ACID guarantees for job atomicity.
Solid Queue	`# config/queue.yml` `default:` `workers: 3` `queues: [default, mailers]`	• Rails-official Active Job backend using SQLite or PostgreSQL • ships with Rails 8; designed for simpler deployments without Redis • supports priorities, recurring jobs, and concurrency controls.
Resque	`class EmailJob` `@queue = :emails` `def self.perform(user_id)` `User.find(user_id).send_email` `end` `end` `Resque.enqueue(EmailJob, 123)`	• Redis-backed Ruby library using forked processes instead of threads • simpler than Sidekiq but less efficient (one job per process) • includes web UI; largely superseded by Sidekiq in modern Rails apps.
delayed_job	`class WelcomeMailer` `handle_asynchronously :deliver` `end`	• Rails background job system storing jobs in the database via ActiveRecord • single-threaded polling model; lower throughput than Redis-backed queues • no external dependencies; still used in smaller or legacy Rails apps.
Quartz.NET	`IJobDetail job = JobBuilder` `.Create<ReportJob>().Build();` `ITrigger trigger = TriggerBuilder` `.Create().WithCronSchedule(` `"0 0 3 * * ?").Build();`	• .NET job scheduling library for cron-style recurring jobs • stores schedules in SQL Server, PostgreSQL, or in-memory; thread pool execution with clustering support • often combined with Hangfire for full background processing.
Kue (deprecated)	`const queue = kue.createQueue();` `queue.create('email', {` `userId: 123` `}).save();`	• Node.js priority job queue backed by Redis with web UI • no longer actively maintained; BullMQ is the recommended replacement • still found in legacy Node apps.

Table 2: Message Brokers and Queue Backends

System	Example	Description
Redis	`LPUSH jobs:email '{"user_id":123}'` `BRPOP jobs:email 0`	• In-memory data store most commonly used as job queue backend • extremely fast (microsecond latency) but requires persistence configuration (AOF/RDB) to prevent job loss on crashes • supports lists, sorted sets, and streams.

Table 3: Job Queue Architectural Patterns

Pattern	Example	Description
Fire-and-forget	`BackgroundJob.Enqueue(() =>` `ProcessOrder(orderId))`	• Enqueue job and immediately return; simplest pattern for tasks like sending emails, logging, or cache warming • job executed asynchronously by worker pool with no waiting for result.

Table 4: Retry and Failure Handling Strategies

Strategy	Example	Description
Exponential backoff	Retry delays: 1s, 2s, 4s, 8s, 16s `retry_delay = base_delay ` `(2 * attempt)`	• Progressively increase wait time between retries • reduces load on failing service, giving it time to recover; prevents thundering herd • standard pattern for transient failures (network blips, rate limits, temporary outages).