Using Redis Streams as a Persistent Message Log

1. Why streams need their own data structure

Redis Streams, introduced in Redis 5.0, solve a problem that neither lists nor Pub/Sub alone could cover satisfactorily: an append only log of events that multiple independent consumers can read at the same time, with each consumer tracking its own read progress, and that does not lose messages even if a consumer is offline at the time of writing. A list can serve as a queue, but the moment an element gets removed with LPOP, it is lost for every other consumer. Pub/Sub delivers messages only to currently connected subscribers and stores nothing, a latecomer misses everything sent during their absence.

A Redis Stream is at its core an append only log in which every entry gets a unique, monotonically increasing ID, typically composed of a timestamp and a sequence number. This ID acts as a cursor: a consumer can specify at any time from which ID onward to keep reading, and thereby misses no message, no matter how long it was offline, as long as the stream has not been cleaned up through trimming. This property makes streams the foundation for event sourcing patterns, activity feeds and message queues with guaranteed delivery.

This post walks through the central stream commands, explains consumer groups for distributed processing in detail, and positions streams against the two obvious alternatives, Pub/Sub and Kafka, to enable a well grounded decision for the use case at hand.

Anyone already familiar with other Redis data types such as lists or sorted sets will find streams to be a new, standalone concept, specifically developed for messaging use cases rather than an extension of an existing data type.

Internally, Redis implements a stream as a radix tree, optimized for fast range queries by ID and efficient appending of new entries at the end. This data structure differs fundamentally from the previously discussed Redis data types such as lists or sets, and was purpose built for the access pattern "many writes, multiple independent readers with different progress".

2. XADD and the stream ID in detail

XADD key * field1 value1 field2 value2 adds a new entry to the stream. The asterisk * instructs Redis to automatically generate a new ID, consisting of the current millisecond timestamp and a sequence number that increments for multiple entries within the same millisecond, for example 1721742001234-0. This automatic ID generation guarantees that IDs within a stream always increase strictly monotonically, which is decisive for later range queries.

Every entry consists of a set of field value pairs, similar to a hash, allowing an entry to carry several structured data fields instead of just a single string value as with a list. That fits event data extremely well, such as event_type, user_id and payload in a single atomic entry. XLEN key queries the current number of entries, XRANGE key start end retrieves a range by ID, where the special values - and + represent the beginning and end of the stream respectively.

Specifying an explicit ID instead of * is also possible and relevant in distributed systems, for example when an ID needs to be adopted from an external source. Redis enforces that every new ID must be strictly greater than the last one assigned, an attempt to insert a smaller or equal ID fails with an error. This guarantee is the foundation that makes range queries and cursor based reading work reliably, without the application having to worry about ordering itself.

# Streams: appending structured events with auto-generated IDs
redis-cli XADD orders:events "*" event_type "created" order_id "5001" amount "129.90"
redis-cli XADD orders:events "*" event_type "paid" order_id "5001" amount "129.90"
redis-cli XLEN orders:events
redis-cli XRANGE orders:events - + COUNT 10
redis-cli XREVRANGE orders:events + - COUNT 1

3. XREAD: simple reading and live tailing

XREAD COUNT n STREAMS key id reads entries starting from a given ID, exclusively, meaning the entry with exactly that ID is not returned again, only everything after it. With the ID 0, the entire stream gets read from the start, with $ only entries added after the moment of the call, useful for live tailing similar to tail -f on a log file.

With the BLOCK milliseconds option, XREAD waits until new entries become available, instead of immediately returning an empty result. This blocking variant replaces polling loops and is the foundation for simple consumer implementations without a consumer group, where a single process reads the entire stream, for example for a monitoring dashboard that should display every event in real time, without multiple consumers needing to share the work.

An important difference from consumer groups: with plain XREAD, the application itself has to keep track of which ID was last read, for example in its own variable or a separate Redis key. Redis stores no server side progress for this simple read variant, unlike consumer groups, which manage their read progress automatically. For a single, permanently running consumer that is usually not a problem, for multiple independent consumers with persistent progress, consumer groups are the more robust choice.

XREAD can also query multiple streams at once in a single call, by listing several key-ID pairs after STREAMS. That is useful when an application has to monitor several thematically separate event logs at once, for example orders and payments as separate streams, but still wants to wait for new events in both with a single blocking call.

# XREAD: live tailing without a consumer group
redis-cli XREAD COUNT 10 STREAMS orders:events 0
redis-cli XREAD BLOCK 5000 STREAMS orders:events "$"
redis-cli XREAD COUNT 5 STREAMS orders:events 1721742001234-0

4. Consumer groups: distributed processing with XREADGROUP

While XREAD suits individual, independent consumers, consumer groups solve the problem of distributed processing: multiple workers are meant to share the entries of a stream, so that every entry gets processed by exactly one worker within the group, similar to a classic message queue. XGROUP CREATE key groupname id creates a consumer group that tracks its own read progress, independent of individual consumers.

XREADGROUP GROUP groupname consumername COUNT n STREAMS key > reads new entries not yet delivered to this group and automatically marks them as pending for the requesting consumer. The special character > means "only new entries that have never been delivered to this group". Multiple workers, all reading with the same group name but a different consumer name, thereby automatically share the entries of a stream, without needing coordination logic in the application, Redis handles the distribution.

A central difference from a plain list as a queue: when an entry gets delivered to a consumer via XREADGROUP, it does not vanish from the stream, it merely gets marked as "in progress" in the group's pending entries list. Only an explicit XACK removes it from that list. If a worker crashes during processing, the entry stays visible in the pending list and can be taken over by another worker, a decisive advantage for reliable processing.

XGROUP CREATECONSUMER and XGROUP DELCONSUMER allow dynamically managing the set of active consumers within a group, for example when workers get added or removed in an auto scaling environment. XINFO CONSUMERS key group shows all known consumers of a group along with the number of unacknowledged entries, which is valuable for monitoring and capacity planning in production systems.

# Consumer groups: distributed processing across multiple workers
redis-cli XGROUP CREATE orders:events order-processors "$"
redis-cli XREADGROUP GROUP order-processors worker-1 COUNT 5 STREAMS orders:events ">"
redis-cli XREADGROUP GROUP order-processors worker-2 COUNT 5 STREAMS orders:events ">"
redis-cli XPENDING orders:events order-processors
redis-cli XINFO GROUPS orders:events

5. XACK and the pending entries list: reliable delivery

XACK key group id confirms the successful processing of an entry and removes it from the group's pending entries list. This explicit acknowledgment step is the core of the "at least once" delivery model of streams: an entry only counts as fully processed once the consumer actively confirms that processing finished successfully. Without XACK, the entry stays in the pending list, even if the consumer has long since moved on to the next entry.

For dealing with crashed workers, XCLAIM, or the more convenient XAUTOCLAIM, offers the ability to take over entries that have been sitting in another consumer's pending list longer than a defined time, and to process them itself. That is the foundation for timeout based retry logic: a watchdog process periodically checks with XPENDING which entries have been unacknowledged for longer than, say, 30 seconds, and takes them over with XAUTOCLAIM for another processing attempt.

The min-idle-time parameter on XCLAIM and XAUTOCLAIM prevents an entry from being taken over prematurely while the original worker might still be actively processing it. Only once an entry has sat in the pending list unacknowledged longer than this threshold does it count as potentially orphaned and become eligible for takeover by another consumer, which prevents slow but functioning processing from being mistakenly treated as crashed.

This interplay of XREADGROUP, XACK and XAUTOCLAIM forms a complete, reliable delivery model directly inside Redis, without needing to operate external message queue infrastructure like RabbitMQ or Kafka, as long as throughput and retention requirements stay within what a single Redis instance or a Redis cluster can handle.

For entries that remain permanently unprocessable, for example because the payload is malformed and every processing attempt fails, a dead letter pattern is recommended: a watchdog process counts delivery attempts via the XPENDING field, and moves entries exceeding a defined limit into a separate error stream instead of retrying them endlessly. That prevents a single broken entry from blocking processing of the entire stream.

# XACK and recovering stuck entries after a worker crash
redis-cli XACK orders:events order-processors 1721742001234-0
redis-cli XPENDING orders:events order-processors - + 10
redis-cli XAUTOCLAIM orders:events order-processors worker-2 30000 0
redis-cli XACK orders:events order-processors 1721742002456-0

6. Streams vs. Pub/Sub: the decisive difference

Redis Pub/Sub, with PUBLISH and SUBSCRIBE, is a pure fire and forget broadcast system: a message gets sent to all currently connected subscribers of a channel and is then no longer stored. If a subscriber is not connected at the moment of PUBLISH, it misses the message irrevocably, there is no history, no cursor and no way to catch up on missed messages. That makes Pub/Sub suitable for use cases where loss is tolerable, such as live notifications in an active user interface.

Streams solve exactly this problem: every message stays stored in the stream until explicitly trimmed, and every consumer, whether via plain XREAD or through a consumer group, can resume reading from any historical position at any time. This persistence makes streams suitable for use cases where no event may be lost, such as order status changes, audit logs, or event sourcing, where the complete history of an aggregate must be reconstructible from its events.

In practice, Pub/Sub and streams can also be combined: Pub/Sub for low latency live notification of active clients, for example to immediately update an open web interface, and in parallel a stream for reliable, catch up capable processing in the background. This combination leverages the respective strengths, instead of forcing a single technology to cover both very different requirements.

7. Streams as Kafka-lite: where the limit lies

The combination of consumer groups, persistent entries and explicit acknowledgment deliberately echoes Kafka, and indeed Redis Streams are frequently used as a lightweight Kafka alternative for medium throughput requirements. For teams already running Redis who do not want to operate dedicated Kafka infrastructure with Zookeeper or KRaft, partitioning and replication configuration, streams offer significantly lower operational overhead at comparable base functionality.

The limits show up at very high throughput and long retention: Kafka is designed for partitioning across many brokers and persistent logs in the terabyte range, while a Redis Stream is bounded by the available memory of the Redis instance, unless trimming gets used. Ordering guarantees differ too: Kafka guarantees order only within a partition, while a single Redis Stream offers a global, strictly monotonic order of all entries, which is simpler to handle for many use cases but harder to scale horizontally at very high throughput.

Replay behavior also differs between the two systems: Kafka consumers can reset an offset arbitrarily and reprocess entire partitions, an established pattern for backfills after a bug fix. Redis Streams support the same principle via XREADGROUP with an explicit ID instead of >, though without the mature tooling and offset management features that have grown over years within the Kafka ecosystem.

For systems with millions of events per second, long term retention over months or years, or complex stream processing topologies with several downstream transformation steps, Kafka remains the more robust choice. For medium sized event logs, typically up to a few thousand events per second with retention in the range of hours to a few days, Redis Streams are a pragmatic, significantly simpler to operate alternative.

An often decisive practical factor in the choice is the operational know-how already present in the team: anyone already running Redis for caching and sessions can introduce streams without new infrastructure, new monitoring tools or new operational knowledge. Switching to Kafka, in contrast, almost always means learning and operating a completely new system with its own operational requirements, backup strategies and scaling patterns.

8. Memory management: XTRIM and MAXLEN

Since a stream grows unbounded by default, active memory management is mandatory for production use. XTRIM key MAXLEN n bounds the stream to the last n entries and removes older ones. Alternatively, MAXLEN can be specified directly on XADD, for example XADD key MAXLEN 10000 * field value, which automatically trims the stream to the configured maximum length on every write, without needing a separate trimming process.

For more precise control, XTRIM key MINID id allows removing all entries before a given ID, for example everything older than 24 hours, which is more practical for time based retention policies than a pure length limit. Important to note: trimming removes entries regardless of whether they have already been processed by all consumer groups. Anyone who needs to guarantee that no entry gets removed before full processing should choose the retention window generously relative to the expected maximum processing delay.

The ~ modifier before the length value, for example XADD key MAXLEN ~ 10000 * field value, tells Redis to perform approximate rather than exact trimming. Redis then removes whole internal nodes of the underlying data structure instead of trimming to the exact target length on every single write, significantly lowering the CPU cost of trimming at very high write frequency, at the cost of a slightly imprecise, but always higher, actual stream length.

# Memory management: bounding stream growth with trimming
redis-cli XADD orders:events MAXLEN "~" 10000 "*" event_type "shipped" order_id "5001"
redis-cli XTRIM orders:events MAXLEN 5000
redis-cli XTRIM orders:events MINID 1721742000000-0
redis-cli XLEN orders:events

9. Streams compared to lists and Pub/Sub

The table below compares streams to the two obvious alternatives, list and Pub/Sub, for queue and messaging use cases.

Streams combine the strengths of list and Pub/Sub: persistence like a list, but with multiple independent readers like Pub/Sub, complemented by consumer groups for distributed processing and explicit delivery guarantees that neither list nor Pub/Sub can offer alone.

For new projects that need a queue or an event log and already run Redis, a stream is almost always the right starting choice over a list used as an improvised queue. Only for very simple fire and forget notifications without any claim to delivery guarantee does Pub/Sub remain the more fitting choice, thanks to its lower latency and simplicity.

10. Summary

Redis Streams solve the problem of a persistent, repeatedly readable message log directly inside Redis. XADD adds entries with automatically generated, monotonically increasing IDs. XREAD suits simple, independent reading, while consumer groups with XREADGROUP, XACK and XAUTOCLAIM enable distributed processing with delivery guarantees, including recovery after a worker crash via the pending entries list.

Streams win over Pub/Sub through persistence and catch up capability, over a plain list through multiple independent consumers with their own cursor. Compared to Kafka, streams remain the lighter weight choice for medium sized event logs, while Kafka stays the more robust infrastructure at very high throughput and long term retention in the terabyte range. XTRIM with MAXLEN or MINID keeps a stream's memory footprint under control in production.

Anyone already running Redis for caching or sessions should evaluate streams as the natural first option for new queue and event log requirements, before introducing additional, separately operated message queue infrastructure. The lower operational overhead at comparable base functionality makes streams one of the most underrated features of Redis.

11. FAQ: Redis Streams as a Message Log