Redis Streams Tutorial: XADD, Consumer Groups, and Recovery

Redis Streams gives you an append-only event log inside Redis. You can add events with XADD, inspect them with XREAD, then use a consumer group when each event should go to one worker and stay pending until that worker acknowledges it.

I ran the Redis CLI flow below against a fresh local Redis instance. It appends two order events, hands one event to a consumer, acknowledges it, then transfers an idle pending event to another consumer.

Each Stream entry keeps its ID after delivery.

Redis stores an entry as fields rather than as one fixed message schema, so the producer decides which fields belong to each event. A checkout event might carry an order ID, item, quantity, and timestamp, while an inventory event in the same Stream can carry a SKU and stock adjustment.

That flexibility is useful during early development, but it does not remove the need for an application contract. Decide which fields are required before you add consumers, validate incoming values at the producer boundary, and keep a schema version when producers and consumers can deploy independently.

A consumer name identifies one worker within its group.

Stream entries remain available until your retention policy removes them. A consumer group’s acknowledgement state is separate from that retention decision, so acknowledging an entry does not delete it from the Stream.

This separation is why a second group can read the same event for analytics after a fulfilment group has completed its work. It also means you must choose a maximum length or an age boundary that gives slow consumers enough time to process their assigned entries.

The pending list belongs to a consumer group, not to the Stream itself.

What Redis Streams stores

A Stream is an ordered sequence of entries. Each entry has an ID and one or more field-value pairs, so a producer can record an event without waiting for a consumer to finish work.

Redis assigns an ID when you pass an asterisk to XADD. The generated ID is ordered by time and sequence, which lets consumers resume from a known point instead of guessing where to continue.

Use generated IDs unless your producer has a strict ordering requirement.

Run a small Redis Streams workflow

Start Redis before running the commands. If you still need the server and the Node.js connection basics, use this Redis installation and commands tutorial first.

Start Redis and append two events

Each XADD call creates one entry in the orders Stream. The first argument after the Stream name is the entry ID, and the asterisk asks Redis to generate it for you.

The dollar sign begins a new group at the latest existing entry.

redis-server
redis-cli XADD orders * item keyboard quantity 1
redis-cli XADD orders * item mouse quantity 2

The command returns IDs similar to 1788278034000-0. Do not hard-code those values in application logic, because Redis creates a different ID for each appended entry.

Read the Stream without a group

XREAD is useful when every reader needs its own view of the same event log. Starting from 0 returns the entries from the beginning, which makes it useful for inspecting a small Stream or replaying a known range.

A direct reader does not change a consumer group’s delivery state.

redis-cli --raw XREAD COUNT 10 STREAMS orders 0

The output contains the Stream name, then each entry ID and its fields. A direct read does not track which reader handled an entry, so it is not the right choice when several workers must divide the work.

Create a consumer group and read new events

A consumer group stores one shared delivery position for a set of workers. Create it once with XGROUP CREATE, then give each process a distinct consumer name.

Acknowledgement is a delivery decision, not a deletion command.

redis-cli XGROUP CREATE orders fulfilment 0 MKSTREAM
redis-cli --raw XREADGROUP GROUP fulfilment worker-a COUNT 1 STREAMS orders >

The 0 position lets the new group consume entries already in the Stream. Use the dollar sign instead when a new group should begin only with entries that arrive after group creation.

The greater-than sign in XREADGROUP requests entries that have not yet been delivered to any consumer in this group. Redis places the delivered entry in the group’s pending entries list, often called the PEL, until the consumer acknowledges it.

An idle threshold should exceed normal processing time.

Acknowledge successful processing

Call XACK only after the side effect succeeds. If the worker sends an email, writes a database record, or calls another service before acknowledgement, keep the entry pending when that operation fails.

redis-cli XACK orders fulfilment 1788278034000-0
redis-cli XPENDING orders fulfilment

Replace the example ID with the ID returned by your own read. XACK removes that entry from the pending list for the fulfilment group, while the entry remains in the Stream for other groups or direct reads.

Recovery should be observable in your worker logs.

Recover an idle pending message

A pending entry protects work from disappearing when a worker stops after receiving it. It also needs a recovery path, otherwise an entry assigned to a dead worker can remain pending forever.

XAUTOCLAIM moves entries that have been idle for at least the specified time to another consumer. Use a threshold that exceeds your normal processing time, then process the returned entries and acknowledge them after the work succeeds.

Independent consumer groups can process the same event for separate purposes.

redis-cli --raw XAUTOCLAIM orders fulfilment worker-b 60000 0-0 COUNT 10

This command asks Redis to scan from 0-0 and assign entries idle for at least 60 seconds to worker-b. Its first returned value is a cursor for the next scan, and 0-0 means the scan reached the end of the pending list.

Claiming can repeat an external side effect if the first worker completed the action but stopped before XACK. Give each event a business identifier and make the downstream action safe to repeat, such as enforcing a unique order ID in the database.

Retention and acknowledgement solve different lifecycle decisions.

Use XPENDING before recovery when you need to inspect the group’s backlog. Its summary tells you how many entries are pending, which IDs bound that pending range, and which consumers hold deliveries.

The detailed XPENDING form can show individual IDs and idle durations, which helps distinguish a worker that is merely busy from one that stopped. Track the age of pending entries with the same care you give failed jobs in any other queue.

An application-level idempotency key protects downstream side effects.

Choose the claim threshold from observed work duration rather than from the request timeout alone. If a normal job completes in a few seconds but a downstream service occasionally takes a minute, a 60-second threshold can move a job while the first worker is still active.

A longer threshold reduces duplicate work but delays recovery after a crash. The safe choice depends on the maximum duration you can tolerate before a customer-facing action needs another attempt.

Measure that duration from successful production-like runs.

Limit each XREADGROUP batch to work your worker can finish before its claim threshold. A batch of one simplifies the first implementation because the worker has one side effect and one acknowledgement decision at a time.

Larger batches reduce Redis round trips, yet they also increase the number of entries that can become pending together if the process exits. Increase the count only after the processing and recovery behavior is observable in your application logs.

Streams support more than one consumer group without duplicating producer writes. A fulfilment group and an analytics group can each advance at their own pace because their pending lists and last-delivered IDs are independent.

Each group stores its own delivery position.

That is useful when the same order event needs a business action and a reporting action. It is not a substitute for separating workloads that need different retention, access control, or failure policies.

Keep those boundaries explicit in the service design.

Choose direct reads or consumer groups

Use XREAD when every reader needs the same event sequence. Use a consumer group when several workers should share one processing workload and Redis should track deliveries that still need acknowledgement.

Choose one delivery model per reader task.

Use Redis Streams from Node.js

Keep the command lifecycle clear before moving it into application code. Redis maintains a current Node.js streaming example that shows a producer, independent consumer groups, and XAUTOCLAIM after a simulated consumer failure.

Use the official example for the client-specific API surface.

Your Node.js worker should use the same boundary as the CLI flow. Read with a unique consumer name, complete the side effect, acknowledge the entry, and schedule a recovery worker for entries that remain idle beyond the expected processing window.

Keep acknowledgement next to the completed side effect.

Redis Streams command reference

These commands cover the smallest operational loop. Redis also provides range reads, trimming, group inspection, and explicit claims when your application needs more control.

FAQ

Start with one Stream, one consumer group, and one worker. Once the pending-entry and recovery loop behaves as expected, add workers and set a retention policy that matches the time you need events to remain available.