Redis Job Processing with Streams: Workers, Retries, Idempotency, and Recovery

Redis Job Processing with Streams: Workers, Retries, Idempotency, and Recovery

Background jobs move slow or failure-prone work out of the request path: sending email, generating reports, resizing media, calling external APIs, rebuilding search indexes, or processing events.

Redis can support this well, but a production worker system needs more than “push a value and pop it later.” You need to know who owns a job, when it is considered complete, what happens if a worker dies, and whether running the same job twice is safe.

For those requirements, Redis Streams with consumer groups are usually a better mental model than a bare Redis list.

The production job-processing loop

A reliable worker flow looks roughly like this:

The important distinction is that delivery and completion are separate events.

A worker can receive a job, start processing it, and crash. Redis keeps that unacknowledged entry in the consumer group's Pending Entries List (PEL), so another worker can later recover it.

That is the foundation of durable worker ownership in Redis Streams.

Produce jobs with XADD

A producer appends a job to a stream:

XADD jobs:email * \
  job_id 9f2c1 \
  user_id 42 \
  template welcome

Redis creates a stream entry with an ordered ID.

The job payload should usually contain references and execution metadata, not every piece of business data needed forever. For example, storing order_id=8172 and loading the authoritative order during processing is often safer than embedding a large mutable order snapshot in the queue.

Useful fields may include:

job_id
job_type
entity_id
attempt metadata owned by your application
schema/version
created_at if the application needs it

Avoid putting secrets into job payloads unless the worker genuinely needs them and your Redis security model allows it.

Consumer groups distribute work across workers

Create a consumer group for a worker pool:

XGROUP CREATE jobs:email email-workers 0 MKSTREAM

Workers then request new entries through the group:

XREADGROUP GROUP email-workers worker-7 \
  COUNT 10 BLOCK 5000 \
  STREAMS jobs:email >

The official XREADGROUP documentation explains the key behavior: consumers in one group split the arriving messages rather than each consumer receiving every message.

That makes consumer groups a natural fit for a worker pool:

job A -> worker-1
job B -> worker-2
job C -> worker-1
job D -> worker-3

If you instead want every independent subsystem to see every event, create separate consumer groups or use a different delivery model. One worker group is for sharing work, not broadcasting a copy to every worker.

A job is not complete until you acknowledge it

After a worker successfully finishes the side effect, acknowledge the stream entry:

XACK jobs:email email-workers 1726123456789-0

XACK removes the message from the group's Pending Entries List.

The ordering matters:

1. receive the job;
2. perform the intended work;
3. make the result durable;
4. acknowledge the Redis entry.

If you acknowledge before the real work is safely complete, a crash can lose the job from the worker group's perspective.

If you perform the work and crash before acknowledgment, Redis can redeliver the job later. That prevents silent loss, but introduces another requirement: the handler must tolerate duplicate execution.

Redis Streams job processing is at-least-once

Redis' Streams documentation describes consumer-group processing as at-least-once delivery when acknowledgments and recovery are used.

That means a job can be delivered more than once.

Imagine a payment-webhook worker:

1. worker receives event
2. worker writes result to database
3. process crashes
4. XACK never happens
5. another worker later claims the pending entry
6. same event runs again

The correct fix is usually not “try to make Redis never redeliver.” The correct fix is to make side effects idempotent.

Make job handlers idempotent

Idempotency means retrying the same logical job does not create an incorrect duplicate result.

Good techniques include:

• a unique job/event ID stored with the durable result;
• database unique constraints;
• conditional state transitions such as pending -> completed;
• provider idempotency keys when calling external services that support them;
• a durable processed-event table when exactly-once business effects matter.

For example, instead of blindly inserting a second fulfillment record:

INSERT INTO processed_jobs (job_id, processed_at)
VALUES ($1, NOW())
ON CONFLICT (job_id) DO NOTHING;

The application can use that durable uniqueness boundary to determine whether the logical effect has already happened.

A Redis lock alone is generally not a substitute for durable idempotency. The worker can fail after the external/database side effect but before releasing or acknowledging anything.

Recover jobs from crashed workers

When XREADGROUP delivers an entry, Redis tracks it in the group's PEL until it is acknowledged.

You can inspect pending work with XPENDING.

For automated recovery, XAUTOCLAIM can transfer ownership of entries that have been pending longer than a chosen minimum idle time.

Conceptually:

XAUTOCLAIM jobs:email email-workers recovery-worker \
  60000 0-0 COUNT 100

This says: find pending jobs that have been idle long enough and claim eligible entries for a healthy consumer.

Do not choose the idle threshold arbitrarily. It must be longer than a normal healthy execution window—or backed by a lease/heartbeat design for genuinely long-running work—otherwise a recovery worker can steal a job that is still legitimately processing.

Retries need an explicit policy

Redelivery is a mechanism. A retry policy is an application decision.

Classify failures before retrying:

A useful policy defines:

• maximum attempts;
• delay/backoff behavior;
• which errors are retryable;
• how attempts are recorded;
• when a job becomes permanently failed;
• where permanently failed jobs go.

Redis Streams do not magically provide your business retry policy. Your worker framework or application must implement it deliberately.

Delayed retries are different from pending recovery

A failed job that should retry in 30 seconds is not the same as a worker that vanished while holding a job.

For delayed execution, many systems maintain a scheduled structure—often a sorted set keyed by retry time—or use a Redis-backed job library that already implements scheduling safely.

Conceptually:

retry_at timestamp -> sorted-set score
job reference       -> sorted-set member

A scheduler moves due jobs back to the runnable stream.

If you implement this yourself, the transfer from “scheduled” to “ready” must be designed for crashes and duplicate execution. For complex scheduling, a mature queue library may be safer than assembling primitives by hand.

Dead-letter failed work instead of retrying forever

Some jobs will never succeed without human or application intervention.

After the retry budget is exhausted, preserve enough information to investigate the failure. A common application-level approach is a separate failure stream:

XADD jobs:email:failed * \
  job_id 9f2c1 \
  original_stream_id 1726123456789-0 \
  error_class provider_rejected \
  attempts 5

Then acknowledge the original entry only after the failure record has been safely created according to your design.

Do not store unlimited stack traces or sensitive payloads in Redis. Keep failure records bounded and move long-lived operational history to an appropriate durable store if necessary.

Bound stream retention

A stream grows as entries are appended unless you trim or delete data.

Redis supports retention controls such as approximate maximum length:

XADD jobs:email MAXLEN ~ 100000 * job_id 9f2c1 user_id 42

or explicit trimming strategies.

The correct retention policy depends on whether entries are needed for replay, diagnostics, or only active processing.

Be especially careful when trimming streams that have consumer groups with pending work. Retention should be planned alongside your acknowledgment and recovery model, not configured as an unrelated memory optimization.

The Redis Streams documentation covers stream trimming and consumer-group behavior in detail.

Monitor backlog, not just Redis CPU

A worker system can be unhealthy while Redis itself looks fine.

Operational signals should include:

• stream length: how much data is retained;
• consumer-group lag: entries still waiting to be delivered;
• pending count: delivered but unacknowledged entries;
• oldest pending age: whether jobs are stuck;
• worker throughput: completed jobs per interval;
• job latency: enqueue-to-start and enqueue-to-complete time;
• retry rate: how often work fails and re-enters processing;
• permanent failure rate: jobs exhausting the retry policy.

XINFO GROUPS reports fields including consumer count, pending entries, last-delivered ID, and group lag when Redis can determine it.

A growing lag with healthy Redis latency usually means you need to inspect worker capacity or downstream dependencies, not tune Redis commands first.

Use concurrency carefully

Adding workers increases throughput only while downstream systems can absorb the load.

If 100 workers all call the same database or third-party API, the queue may simply move the bottleneck downstream.

Set concurrency from measured constraints such as:

• database connection capacity;
• external API quotas;
• CPU/memory cost per job;
• job duration;
• ordering requirements;
• rate limits;
• contention on shared resources.

A queue is a pressure buffer, not permission to overwhelm every dependency behind it.

When a Redis-backed worker system is a good fit

Redis Streams are attractive when:

• Redis is already operated reliably;
• jobs are relatively short or can use a clear lease/recovery policy;
• at-least-once processing is acceptable;
• handlers can be made idempotent;
• you want consumer groups, acknowledgments, replay, and pending recovery;
• the operational scale fits Redis' memory and persistence model.

Consider a dedicated messaging/job platform when you need features such as very long retention, complex routing, stronger broker-specific durability guarantees, extremely large backlogs, or mature workflow orchestration that would otherwise require substantial custom code.

A production checklist

Before calling a Redis job queue reliable, verify:

• workers read through a consumer group;
• successful jobs are acknowledged only after the real side effect is durable;
• handlers are idempotent;
• stale pending jobs can be reclaimed;
• retryable and permanent errors are separated;
• retry attempts are bounded;
• delayed retries have crash-safe scheduling semantics;
• permanently failed jobs are inspectable;
• stream retention is bounded intentionally;
• group lag and pending age are monitored;
• worker concurrency respects downstream capacity;
• Redis persistence/HA matches the importance of queued work.

The main takeaway

The hard part of background jobs is not enqueueing. It is what happens after a worker receives the job.

Redis Streams give you useful primitives—consumer groups, pending entries, acknowledgments, and claim/recovery operations—but the application must still define idempotency, retry policy, delayed scheduling, failure handling, retention, and observability.

Design those pieces first and Redis becomes a strong worker substrate instead of a deceptively simple list of tasks.