Redis, RabbitMQ, and OpenSearch in Zero-Downtime Deployments

1. Why These Three Services Need Special Attention

Redis, RabbitMQ, and OpenSearch are not simple data stores that can be ignored during a deployment. They hold state that is specific to a particular code version: Redis cache entries are encoded with the data structure of the old release, RabbitMQ messages were produced by consumer code that may have changed, and OpenSearch indices reflect a product catalog structure that may need new fields or changed mapping types.

The problem does not show up with simple deployments, but with deployments that change data structures. A new attribute on a product model, a changed cache key format, a new message format in a queue: these changes are clear in the code base, but their impact on running services is frequently left out of the deployment process. The result is inconsistent state that only manifests as errors after the symlink switch.

The right approach is not to shut these services down during the deployment, since that would defeat the purpose of zero downtime. Instead, it comes down to precise timing: flush Redis only after the symlink switch, briefly pause queue consumers during the critical release transition, and treat the OpenSearch reindex as a scheduled follow-up step using an alias swap. These patterns make it possible to achieve zero downtime while still guaranteeing a clean state transition.

2. Redis in Magento: Cache, Sessions, and Full Page Cache

Magento typically uses Redis for three separate databases: cache (db 0), full page cache (db 1), and sessions (db 2). This separation matters for deployments because the three areas need different flush strategies. The regular cache holds serialized PHP objects and configuration data that are often incompatible after a code change. The full page cache holds fully rendered HTML pages that become stale after a template change. Sessions hold user session data that usually stays compatible after a deployment, but must be cleared when session data structures change.

The most common source of errors: the cache is flushed before the symlink switch, but the new code only starts running after the switch, so the old release refills the cache with entries in the old structure. When the symlink then switches and the new code has to work with entries in the old structure, deserialization errors follow. The correct sequence is to flush the cache after the symlink switch, not before it.

# Deploy job with correct Redis flush timing, after symlink switch
deploy:production:
  stage: deploy
  script:
    - RELEASE_ID="$(date +%Y%m%d-%H%M%S)-${CI_COMMIT_SHORT_SHA}"
    - RELEASE_PATH="${DEPLOY_PATH}/releases/${RELEASE_ID}"

    # Step 1: Transfer artifact to new release directory
    - rsync -az ./ "${DEPLOY_USER}@${DEPLOY_HOST}:${RELEASE_PATH}/"

    # Step 2: Link shared files (env.php, media)
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}" "
        ln -sfn ${DEPLOY_PATH}/shared/app/etc/env.php ${RELEASE_PATH}/app/etc/env.php &&
        ln -sfn ${DEPLOY_PATH}/shared/pub/media ${RELEASE_PATH}/pub/media"

    # Step 3: Run setup:upgrade in new release (before symlink!)
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "cd ${RELEASE_PATH} && php bin/magento setup:upgrade --keep-generated"

    # Step 4: Atomic symlink switch
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "ln -sfn ${RELEASE_PATH} ${DEPLOY_PATH}/current"

    # Step 5: Flush Redis AFTER symlink, new code now handles cache rebuilding
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "cd ${DEPLOY_PATH}/current && php bin/magento cache:flush"

    # Step 6: Verify Redis is still connected
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "redis-cli -h ${REDIS_HOST} ping | grep -q PONG || exit 1"

3. Redis Cache Invalidation in the Deployment Process

The strategy for Redis cache invalidation depends on the type of deployment. For pure code changes without database migrations, a full cache flush after the symlink switch is sufficient and safe. For deployments with database migrations, new configuration options, or changed serialization structures, the cache must be flushed before the symlink switch, and no stale entry may be allowed to land in the cache between the old and new code.

A frequently overlooked detail: bin/magento cache:flush and bin/magento cache:clean have different effects. cache:flush completely deletes all keys in the configured Redis backend. cache:clean only marks Magento specific cache entries as invalid, leaving other backend data untouched. For deployments, cache:flush is the safer choice because it makes no assumptions about which keys might be incompatible.

4. RabbitMQ: Queue Consumers and Message Persistence

RabbitMQ in Magento handles asynchronous operations through queue consumers: sending emails, indexer updates, order processing, and other background tasks. These consumers run as separate processes that continuously pull messages from the queues. The problem during deployment: a consumer process still running the old code processes messages with the old consumer logic while the new code is already active. Depending on the type of message, this can lead to incompatibilities.

The situation is especially critical when a deployment changes the message format of a queue. Messages that were produced by the old code and have not yet been processed still have the old format. Once the consumer is upgraded to the new format, it may no longer be able to correctly deserialize the old messages. In most Magento deployments this is not an issue, because the message format rarely changes, but when it does, the consumer process must be paused during the critical time window.

5. Safely Pausing and Reactivating Queue Consumers

Pausing and reactivating RabbitMQ consumer processes in Magento can be implemented through Supervisor or systemd. The safest point to pause them is immediately before the symlink switch. The consumer processes are stopped, the symlink switch happens, setup:upgrade and the cache flush run, and then the consumers are restarted with the new code. The window is typically under 30 seconds, during which messages accumulate in the queue but are never lost.

Starting with version 2.4, Magento also offers the option to manage consumers through a PID file. bin/magento queue:consumers:start ConsumerName --pid-file-path=/tmp/consumer.pid starts a consumer and stores the PID. This allows a clean stop via kill -SIGTERM $(cat /tmp/consumer.pid), which lets the consumer finish processing the current message before it shuts down. This graceful shutdown prevents a message from being interrupted mid processing.

# Deployment with RabbitMQ consumer management and OpenSearch reindex
deploy:with-services:
  stage: deploy
  script:
    - RELEASE_PATH="${DEPLOY_PATH}/releases/$(date +%Y%m%d-%H%M%S)"

    # Step 1: Stop queue consumers gracefully before symlink switch
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}" "
        supervisorctl stop magento-consumers:* 2>/dev/null || true &&
        echo 'Waiting for consumers to finish current messages...' &&
        sleep 5"

    # Step 2: Transfer artifact and link shared files
    - rsync -az ./ "${DEPLOY_USER}@${DEPLOY_HOST}:${RELEASE_PATH}/"
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}" "
        ln -sfn ${DEPLOY_PATH}/shared/app/etc/env.php ${RELEASE_PATH}/app/etc/env.php &&
        ln -sfn ${DEPLOY_PATH}/shared/pub/media ${RELEASE_PATH}/pub/media"

    # Step 3: Run setup:upgrade in new release
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "cd ${RELEASE_PATH} && php bin/magento setup:upgrade --keep-generated"

    # Step 4: Atomic symlink switch
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "ln -sfn ${RELEASE_PATH} ${DEPLOY_PATH}/current"

    # Step 5: Flush Redis cache with new code active
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "cd ${DEPLOY_PATH}/current && php bin/magento cache:flush"

    # Step 6: Restart consumers with new code
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "supervisorctl start magento-consumers:* 2>/dev/null || true"

    # Step 7: Trigger OpenSearch reindex asynchronously (non-blocking)
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}" "
        cd ${DEPLOY_PATH}/current &&
        nohup php bin/magento indexer:reindex catalogsearch_fulltext \
          > ${DEPLOY_PATH}/shared/var/log/reindex.log 2>&1 &
        echo 'OpenSearch reindex started in background'"

6. OpenSearch: Index Structure and Mapping Compatibility

OpenSearch in Magento holds the product catalog in an index with a defined mapping. When a deployment adds new product attributes, changes existing attributes, or modifies the mapping schema, the index has to be updated. The problem: a full reindex for a large catalog can take anywhere from minutes to hours. During the reindex, search results can be incomplete or incorrect.

The professional solution for zero-downtime index updates is the alias swap pattern: a new index is built in the background (for example magento_catalog_product_v2) while the active alias magento_catalog_product still points at the old index. Only once the new index is fully built does the alias get switched over to the new index. The switch is atomic and invisible to the application. Magento supports this pattern natively starting with version 2.4.6 through the OpenSearch adapter.

7. Index Updates in a Zero-Downtime Context

For deployments without mapping changes, an incremental reindex after the deployment is sufficient. Magento marks products that have changed since the last full index as "require reindex", and an incremental reindex only processes those products, not the entire catalog. This process can start asynchronously after the deployment and blocks neither the symlink switch nor the cache flush phase.

For deployments with mapping changes, meaning cases where new fields need to appear in the index, the alias swap pattern is required. In the GitLab pipeline, that means an additional job after the deploy job that starts the reindex and either waits for it to finish or runs asynchronously and is watched through separate monitoring. In that case, the verify job should include a check confirming that search on the site returns correct results.

8. GitLab Pipeline Integration for All Three Services

Redis, RabbitMQ, and OpenSearch are integrated into the GitLab pipeline through dedicated steps inside the deploy job, not through separate jobs. The reason: the order of the steps is critical and has to be carried out atomically. A separate job would leave a gap between the end of the deploy job and the start of a service management job, during which consumers could keep running with the wrong code version.

The verify stage should explicitly check all three services: Redis reachability via redis-cli ping, queue consumer status via supervisorctl status or bin/magento queue:consumers:list, and OpenSearch reachability via a simple HTTP check against the cluster endpoint. Together, these checks take under 10 seconds and give a team the confidence that all services are working correctly after the deployment.

9. Comparing Services and Their Deployment Risks

The three services have different risk profiles during deployment. Redis is affected most often because Magento keeps all critical cache structures in Redis. RabbitMQ is rarely critical, except when message formats change. OpenSearch reindex delays are the most common cause of "search shows outdated data" reports after deployments.

The table shows that most deployment actions for these services happen after the symlink switch, with the exception of the queue consumer pause, which begins shortly before the switch. This sequence is not arbitrary: it ensures that the new code is the first to work with the new system state, without the old code having filled the data structures with incompatible data.

10. Summary

Redis, RabbitMQ, and OpenSearch are not passive services that simply survive a deployment unscathed. They hold version specific state that must be handled explicitly after a code change. The core principles: flush the Redis cache after the symlink switch, briefly pause queue consumers and restart them with the new code, trigger the OpenSearch reindex asynchronously after the deployment, and check all three services for reachability and correct state in the verify stage.

The good news: these steps can be fully integrated into a GitLab pipeline without creating any downtime. The critical window, from consumer pause to cache flush, lasts under 30 seconds. During that window, messages accumulate in the queue and users may still be served cached pages, but the application never goes down. That is the practical meaning of zero downtime: no outages, just a short window with controlled limitations.

11. FAQ: Redis, RabbitMQ, and OpenSearch in Deployments