Message Queue-Based Offloading for Better Performance

1. The Synchronous Request-Response Cycle as a Bottleneck

In a classic synchronous web application, the user waits until the server has fully completed every operation triggered as part of a request before getting a response. If completing an order also sends a confirmation email, generates a PDF, and updates an external analytics service, all of those steps have to finish sequentially or in parallel before the success page is delivered.

Each of these side operations brings its own latency and its own failure modes: a slow email server or a briefly unreachable external service then delays not just that side task but the entire user experience, even though the actually important part, the order itself, had long since completed successfully. This coupling of core functionality and side effects is the real bottleneck.

2. What Is Message Queue-Based Offloading

With queue-based offloading, instead of executing a slow operation directly, the application simply writes a compact message with the necessary information onto a queue and returns to the caller immediately. One or more separate worker processes continuously read these messages off the queue and perform the actual work independently of the original request.

This decoupling means the web server process handling the original request is only responsible for quickly writing the message onto the queue, an operation that typically takes a few milliseconds regardless of how long the actual background operation later takes. The user gets a fast response, while the real work happens on a delayed timeline, independent of the web server process.

3. Typical Candidates for Offloading

Classic candidates for offloading are operations whose result the user doesn't need to see within the same second: sending confirmation and marketing emails, generating image variants in different sizes after a product upload, generating extensive PDF reports, or exporting large datasets to CSV. All of these tasks share the property that their duration can vary widely and rarely stays under a second.

The PHP example below shows how completing an order in Magento, instead of sending email synchronously, simply writes a message onto a queue that gets processed by a separate consumer. The controller returns as soon as the message is safely on the queue, without having to wait for the actual email delivery.

final class OrderConfirmationOffloader
{
    public function __construct(
        private readonly PublisherInterface $publisher,
    ) {
    }

    /**
     * Writes a compact message onto the queue instead of
     * sending the email synchronously within the request.
     */
    public function scheduleConfirmationEmail(int $orderId): void
    {
        $message = new OrderConfirmationMessage(
            orderId: $orderId,
            requestedAt: new \DateTimeImmutable(),
        );

        $this->publisher->publish('order.confirmation.email', $message);
    }
}

// Consumer runs as a separate worker process
final class OrderConfirmationConsumer
{
    public function __construct(
        private readonly MailSenderInterface $mailSender,
        private readonly OrderRepositoryInterface $orderRepository,
    ) {
    }

    public function process(OrderConfirmationMessage $message): void
    {
        $order = $this->orderRepository->get($message->orderId);
        $this->mailSender->sendOrderConfirmation($order);
    }
}

4. Instant User Feedback Despite Background Processing

For offloading to not feel like a broken experience to the user, the interface needs to give clear feedback that a task has been accepted and is being processed, even though the result isn't available yet. For an image processing upload, that can be an instantly visible preview with a loading indicator that updates automatically once the processed image variants are ready, for example through polling a status endpoint or a WebSocket notification.

For report generation, a proven pattern is showing the user an immediate confirmation with a reference number and then delivering the finished report by email or through a notification center inside the application. What matters is that the interface communicates the asynchronous nature of the operation transparently, rather than pretending the task is already fully complete.

5. Queue Systems Overview

RabbitMQ is a widely used message broker built on the AMQP protocol, offering flexible routing rules through exchanges and queues, and it's a good fit for classic task queues with multiple consumers. Magento itself ships with its own Message Queue Framework as a native abstraction, which by default sits on top of RabbitMQ and is used internally for processes like product reindexing or inventory updates.

Depending on requirements, alternatives include Amazon SQS as a fully managed cloud service, Redis with its list or stream data structures for simpler cases, or Apache Kafka for very high throughput and event streaming. The choice largely depends on whether operational know-how for a given system already exists and how high the expected message rate actually is.

6. Worker Scaling and Idempotency

A key advantage of the queue-based approach is the independent scalability of web servers and worker processes: if the number of incoming image processing jobs rises, additional worker instances can be started without touching the web server at all, since both components only communicate through the queue. Workers can just as easily be scaled back down when load drops, which works especially well in containerized environments with automatic scaling.

Since messages may occasionally be delivered more than once due to connection drops or worker restarts, the processing logic must be idempotent, meaning it produces the same end result no matter how many times the same message is processed, without unwanted side effects like duplicate emails. A proven pattern for this is checking, before actual processing begins, whether a message with a given unique ID has already been successfully processed.

7. Error Handling and Retry Strategies

If processing a message fails, for example because an external email service is briefly unreachable, the message shouldn't be discarded but retried with exponentially growing delay, giving the failing system time to recover. Most queue systems support built-in retry mechanisms for this, or can be combined with a so-called delay queue that holds messages back for a defined period.

After a defined number of failed attempts, a message should be moved to a dead letter queue, where it remains accessible for manual analysis or automated alerting, instead of being retried endlessly or silently lost. This safeguard prevents a single faulty job from blocking the entire queue or important errors from going unnoticed.

8. Limits of the Pattern

Message queue offloading is explicitly not suitable for operations whose result the user needs immediately within the same request cycle, such as calculating a cart total including discounts or checking credit card authorization during checkout. A user waiting for payment confirmation can't simply be handed a reference number, since the result directly determines the next interaction step.

For very short operations that reliably complete in a few milliseconds, the added overhead of a queue usually isn't worth it either, since the latency of message transport itself tends to slow the operation down rather than speed it up. Offloading delivers its value mainly for operations with variable, potentially long duration whose result isn't critical to the immediate user flow.

9. Summary and Practical Recommendation

Message queue-based offloading improves an application's perceived performance by decoupling the actual response time from the total duration of every triggered operation. The user experiences a fast, consistent response, while compute-heavy side tasks are reliably handled in the background, without burdening the application's core functionality.

Three things matter for a successful rollout: a clear distinction between operations that must remain synchronous and those that can be moved, consistently idempotent worker logic, and a well-thought-out retry and dead letter concept for the error case. Anyone who gets these fundamentals right gains noticeable response speed without sacrificing reliability.

10. Zusammenfassung

11. FAQ: Message Queue Offloading