Background Job Processing Systems Cheat Sheet

Updated 2026-03-18

Next Topic: Bun JavaScript Runtime Cheat Sheet

Background job processing systems enable applications to offload time-consuming tasks (email sending, image processing, data exports, API calls) to separate worker processes that execute asynchronously from the main request-response cycle. These systems prevent user-facing operations from blocking on long-running work by placing jobs into persistent queues backed by Redis, PostgreSQL, RabbitMQ, or cloud services like AWS SQS. Understanding job queue architectures, retry strategies, priority scheduling, and monitoring patterns is essential for building scalable, resilient systems that handle millions of tasks daily while gracefully managing failures. A well-designed background job system transforms a slow, brittle application into one that feels fast and handles spikes in load without degradation—every modern web application processing significant workload relies on one.

What This Cheat Sheet Covers

This topic spans 14 focused tables and 144 indexed concepts. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Job Queue Frameworks by LanguageTable 2: Message Brokers and Queue BackendsTable 3: Job Queue Architectural PatternsTable 4: Retry and Failure Handling StrategiesTable 5: Job Scheduling and TimingTable 6: Concurrency and Worker ManagementTable 7: Message Delivery Guarantees and SemanticsTable 8: Job Queue Monitoring and ObservabilityTable 9: Advanced Queue Features and PatternsTable 10: FIFO and Ordering GuaranteesTable 11: Job Queue Antipatterns and PitfallsTable 12: Cloud-Native Job ProcessingTable 13: Job Queue Performance OptimizationTable 14: Real-World Use Cases

Table 1: Job Queue Frameworks by Language

Framework	Example	Description
Sidekiq	`class EmailJob` `include Sidekiq::Job` `def perform(user_id)` `User.find(user_id).send_welcome` `end` `end` `EmailJob.perform_async(123)`	• Ruby's most popular background job processor • multi-threaded (handles many jobs per process), uses Redis for storage, supports retries and scheduled jobs • powers Shopify, GitHub, and thousands of Rails apps.
BullMQ	`const queue = new Queue('emails');` `await queue.add('send', {` `userId: 123, type: 'welcome'` `});`	• Node.js and Bun job queue built on Redis with advanced features like priorities, rate limiting, delayed jobs, parent-child workflows, and events • successor to Bull with better TypeScript support and performance.
Celery	`@app.task` `def send_email(user_id):` `user = User.objects.get(id=user_id)` `user.send_welcome()` `send_email.delay(123)`	• Python's distributed task queue standard • supports multiple brokers (RabbitMQ, Redis, SQS), result backends (Redis, PostgreSQL, MongoDB), periodic tasks with Celery Beat, and canvas workflows for complex orchestration.
Hangfire	`BackgroundJob.Enqueue(() =>` `SendEmail(123));` `RecurringJob.AddOrUpdate(` `"cleanup", () => Cleanup(),` `Cron.Daily);`	• .NET background job processor integrated with ASP.NET • uses SQL Server, PostgreSQL, or Redis for persistence, includes built-in dashboard, supports fire-and-forget, delayed, recurring, and continuations jobs.
Resque	`class EmailJob` `@queue = :emails` `def self.perform(user_id)` `User.find(user_id).send_email` `end` `end` `Resque.enqueue(EmailJob, 123)`	• Redis-backed Ruby library using forked processes instead of threads • simpler than Sidekiq but less efficient (one job per process) • includes web UI • still used in legacy Rails apps but largely superseded by Sidekiq.
Active Job	`class EmailJob < ApplicationJob` `queue_as :emails` `def perform(user_id)` `User.find(user_id).send_email` `end` `end` `EmailJob.perform_later(123)`	• Rails' built-in abstraction layer for background jobs • doesn't process jobs itself but provides unified API for Sidekiq, Solid Queue, GoodJob, Resque, Delayed Job, and others • enables adapter-agnostic job code.
Temporal	`@workflow.run` `async def order_workflow(order_id):` `await charge_payment(order_id)` `await ship_order(order_id)` `await send_notification(order_id)`	• Durable execution platform for workflows and long-running processes • uses event sourcing to capture state at every step, enabling automatic retries from exact failure point • handles multi-step sagas, human-in-the-loop, and distributed orchestration • available for Go, Java, Python, TypeScript, PHP.

Table 1: Job Queue Frameworks by Language

Framework	Example	Description
Sidekiq	`class EmailJob` `include Sidekiq::Job` `def perform(user_id)` `User.find(user_id).send_welcome` `end` `end` `EmailJob.perform_async(123)`	• Ruby's most popular background job processor • multi-threaded (handles many jobs per process), uses Redis for storage, supports retries and scheduled jobs • powers Shopify, GitHub, and thousands of Rails apps.
BullMQ	`const queue = new Queue('emails');` `await queue.add('send', {` `userId: 123, type: 'welcome'` `});`	• Node.js and Bun job queue built on Redis with advanced features like priorities, rate limiting, delayed jobs, parent-child workflows, and events • successor to Bull with better TypeScript support and performance.
Celery	`@app.task` `def send_email(user_id):` `user = User.objects.get(id=user_id)` `user.send_welcome()` `send_email.delay(123)`	• Python's distributed task queue standard • supports multiple brokers (RabbitMQ, Redis, SQS), result backends (Redis, PostgreSQL, MongoDB), periodic tasks with Celery Beat, and canvas workflows for complex orchestration.
Hangfire	`BackgroundJob.Enqueue(() =>` `SendEmail(123));` `RecurringJob.AddOrUpdate(` `"cleanup", () => Cleanup(),` `Cron.Daily);`	• .NET background job processor integrated with ASP.NET • uses SQL Server, PostgreSQL, or Redis for persistence, includes built-in dashboard, supports fire-and-forget, delayed, recurring, and continuations jobs.
Resque	`class EmailJob` `@queue = :emails` `def self.perform(user_id)` `User.find(user_id).send_email` `end` `end` `Resque.enqueue(EmailJob, 123)`	• Redis-backed Ruby library using forked processes instead of threads • simpler than Sidekiq but less efficient (one job per process) • includes web UI • still used in legacy Rails apps but largely superseded by Sidekiq.
Active Job	`class EmailJob < ApplicationJob` `queue_as :emails` `def perform(user_id)` `User.find(user_id).send_email` `end` `end` `EmailJob.perform_later(123)`	• Rails' built-in abstraction layer for background jobs • doesn't process jobs itself but provides unified API for Sidekiq, Solid Queue, GoodJob, Resque, Delayed Job, and others • enables adapter-agnostic job code.
Temporal	`@workflow.run` `async def order_workflow(order_id):` `await charge_payment(order_id)` `await ship_order(order_id)` `await send_notification(order_id)`	• Durable execution platform for workflows and long-running processes • uses event sourcing to capture state at every step, enabling automatic retries from exact failure point • handles multi-step sagas, human-in-the-loop, and distributed orchestration • available for Go, Java, Python, TypeScript, PHP.