Redis as a Message Queue: Limits and Alternatives

1. Why Redis is attractive for queues

Redis is used as a message queue in a great many systems, even though it was originally designed as an in memory data structure server, not a dedicated queueing solution. The reason for this popularity lies in its simplicity: anyone already running Redis for caching can build a working task queue with the same built in tools, in particular lists and, since version 5, streams, without having to operate an additional infrastructure component like RabbitMQ or Kafka.

This simplicity comes at a price. Redis as a message queue offers neither the sophisticated routing of RabbitMQ with exchanges and bindings nor the full partitioning and replay capabilities of Kafka. For small to medium sized systems with manageable message throughput, Redis is nonetheless a legitimate, often even preferred choice, because operational complexity stays significantly lower than with a dedicated message broker. The following sections show both the practical patterns and the points where Redis as a message queue reaches its limits.

2. The LPUSH/BRPOP pattern in detail

The simplest form of a Redis based queue uses two list commands: LPUSH inserts a new element at the head of a list, BRPOP blockingly removes an element from the tail of the same list. Producers call LPUSH queue:tasks payload to enqueue a task, workers call BRPOP queue:tasks 0 to block while waiting for the next task, where the timeout value 0 means unlimited waiting. This pattern implements a first in first out queue with minimal overhead and practically no configuration.

The decisive advantage of BRPOP over polling with RPOP in a loop: blocking happens server side inside Redis, without the worker process having to actively send requests every second. As soon as an element becomes available, Redis delivers it immediately to the waiting client, enabling a latency of just a few milliseconds between LPUSH and the BRPOP return, entirely without polling overhead.

# Producer: enqueue a task
redis-cli LPUSH queue:emails '{"to":"customer@example.com","template":"order_confirmed"}'

# Worker: block until the next task is available (0 = no timeout)
redis-cli BRPOP queue:emails 0
# 1) "queue:emails"
# 2) "{\"to\":\"customer@example.com\",\"template\":\"order_confirmed\"}"

# Check queue length as a backlog indicator
redis-cli LLEN queue:emails

3. Implementing a simple queue in PHP

In PHP, a worker for the LPUSH/BRPOP pattern is usually implemented as a long lived process that blockingly waits for new tasks in an infinite loop. Clean per task error handling matters: an error while processing a single message must not crash the worker process, otherwise every subsequent task in the queue remains unprocessed.

A critical weakness of this simple pattern is covered in the next section: once BRPOP has removed an element from the list, no copy exists in Redis anymore. If the worker crashes while processing, the task is irretrievably lost, with no other worker able to take it over.

<?php

declare(strict_types=1);

final class EmailQueueWorker
{
    public function __construct(private readonly \Predis\Client $redis)
    {
    }

    /**
     * Blocking worker loop consuming tasks from a Redis list queue.
     */
    public function run(): void
    {
        while (true) {
            // BRPOP blocks server-side until an item is available
            $result = $this->redis->brpop(['queue:emails'], 0);
            [$queueName, $payload] = $result;

            try {
                $task = json_decode($payload, true, 512, JSON_THROW_ON_ERROR);
                $this->sendEmail($task['to'], $task['template']);
            } catch (\Throwable $e) {
                // Do not let a single bad task crash the worker
                error_log("Task processing failed: " . $e->getMessage());
            }
        }
    }

    private function sendEmail(string $to, string $template): void
    {
        // Actual mail dispatch logic goes here
    }
}

4. Limits of the LPUSH/BRPOP pattern

The central structural problem of the LPUSH/BRPOP pattern is a lack of delivery safety. Once BRPOP returns an element, it is irrevocably removed from the list, there is no built in acknowledgment logic. If the worker crashes between receiving and fully processing a task, it is lost, with no trace in Redis and no way to redeliver it. For uncritical tasks like sending a confirmation email, occasional loss may be tolerable, but for payment processing or inventory changes it is not.

A second limitation concerns scaling with multiple consumer types: every message is consumed exactly once by exactly one worker. If both an email dispatch service and an analytics service need to process the same message, a single list offers no sensible fan out distribution without duplicating the message into several separate lists, which requires additional producer logic and coordination effort. Redis Streams solve exactly these two problems structurally.

5. Redis Streams as a more robust queue

Redis Streams, introduced in version 5, are an append only log data structure that fits message queue use cases far better than plain lists. Every message gets a unique, monotonically increasing ID on insertion with XADD, composed of a timestamp and a sequence number. Messages remain in the stream even after being read, enabling replay and later analysis, a fundamental difference from lists, where a read element disappears immediately.

The decisive mechanism for reliable processing is consumer groups. Multiple workers register as a group on a stream, each message is assigned to exactly one group member, but the message stays marked as "pending" in the stream until the worker explicitly acknowledges it with XACK. If a worker crashes without sending XACK, another worker can claim the message with XCLAIM and process it again. This combination of delivery guarantee and fan out capability across multiple consumer groups makes streams the significantly more robust choice for anything beyond simple fire and forget tasks.

# Add a message to a stream, ID is generated automatically
redis-cli XADD orders:stream '*' orderId 8842 status shipped

# Create a consumer group, reading from the current end of the stream
redis-cli XGROUP CREATE orders:stream email-workers '$'

# Consume a message within the group
redis-cli XREADGROUP GROUP email-workers worker-1 COUNT 1 STREAMS orders:stream '>'

# Acknowledge after successful processing
redis-cli XACK orders:stream email-workers 1690000000000-0

# Claim unacknowledged messages from a crashed worker
redis-cli XCLAIM orders:stream email-workers worker-2 30000 1690000000000-0

6. Using consumer groups in PHP

Implementing a stream worker with consumer groups in PHP follows a clear flow: read messages, process, acknowledge. It is important to call XACK only after fully successful processing, so that a crash during processing leaves the message marked as "pending" and lets it be claimed by another worker later.

In addition, a separate, periodically running process should search with XPENDING and XCLAIM for messages that have remained unacknowledged longer than a defined window and actively reassign them to another worker. Without this additional mechanism, messages from crashed workers would not be lost, but they also would not automatically be processed further.

<?php

declare(strict_types=1);

final class StreamOrderWorker
{
    public function __construct(
        private readonly \Predis\Client $redis,
        private readonly string $consumerName = 'worker-1'
    ) {
    }

    /**
     * Consume order events from a Redis Stream via a consumer group.
     */
    public function run(): void
    {
        while (true) {
            $messages = $this->redis->xreadgroup(
                'order-workers',
                $this->consumerName,
                ['orders:stream' => '>'],
                1,
                5000
            );

            if (empty($messages)) {
                continue;
            }

            foreach ($messages['orders:stream'] as $id => $fields) {
                try {
                    $this->processOrder($fields);
                    // Only ACK after successful processing
                    $this->redis->xack('orders:stream', 'order-workers', [$id]);
                } catch (\Throwable $e) {
                    error_log("Order processing failed for {$id}: " . $e->getMessage());
                    // Message stays pending and can be reclaimed later
                }
            }
        }
    }

    private function processOrder(array $fields): void
    {
        // Actual order processing logic goes here
    }
}

7. Where streams hit their limits too

Redis Streams solve the delivery guarantee and fan out problems of plain lists, but remain limited compared to dedicated message broker systems. Complex routing, such as topic based delivery with multiple conditions or priorities between different message types, can only be replicated with streams through additional application logic, while RabbitMQ offers built in support for this via exchange types like topic or headers.

Also at very high throughput of several million messages per second combined with horizontal partitioning across many physical nodes, it becomes clear that Redis, even in cluster mode, was not conceptually designed for this scale. Kafka was built from the ground up for exactly this scenario, with partitioning, replication and a storage architecture built on disk rather than memory, allowing it to hold significantly larger message histories economically.

# Limit stream size to control memory growth
redis-cli XTRIM orders:stream MAXLEN 100000

# List unacknowledged messages older than 30 seconds
redis-cli XPENDING orders:stream order-workers IDLE 30000 - + 10

# Check the memory usage of a stream
redis-cli MEMORY USAGE orders:stream

8. When RabbitMQ or Kafka are the better choice

RabbitMQ is worth considering as soon as complex routing is needed: multiple consumer types that should receive different subsets of the same messages based on routing keys or message properties, prioritized queues, or dead letter exchanges for systematic error handling. When strict AMQP compatibility with existing enterprise systems is required, RabbitMQ is also the more obvious choice over Redis.

Kafka becomes relevant once message throughput and retention duration exceed the limits of Redis: event sourcing architectures where years of history must remain searchable, streaming analytics with several independent consumer groups processing the same data stream in parallel at different speeds, or systems with guaranteed, strict message ordering across partition boundaries. For most web applications with manageable message volume, however, Redis with streams remains the more pragmatic, operationally simpler choice.

9. Redis, RabbitMQ and Kafka compared directly

The following table compares the three systems along the most important decision criteria.

The choice between the three systems should be based on actual need, not popularity. Anyone already running Redis with moderate routing and throughput requirements saves considerable operational complexity with streams compared to an additional message broker.

10. Summary

Redis as a message queue works well for many use cases, with varying robustness depending on the chosen pattern. The simple LPUSH/BRPOP pattern delivers a working FIFO queue without a delivery guarantee, suitable for uncritical, loss tolerant tasks. Redis Streams with consumer groups close this gap through XACK acknowledgment and XCLAIM reassignment, and additionally enable fan out across multiple consumer groups as well as message replay.

For complex routing, very high throughput, or long retention periods, however, even streams reach their limits. RabbitMQ offers more flexible routing through exchanges in these cases, Kafka scales considerably further for event sourcing and streaming analytics. The right decision depends on the concrete requirements for throughput, retention and routing complexity, not on the general question of whether Redis is "good enough" for queues.

11. FAQ: Redis as a Message Queue