GitLab Pipelines for Multi-Server Deployments: Web, Worker, Cron, Queue

1. The multi-server problem in Magento deployments

In a single-server setup, the deployment logic is relatively simple: transfer the artifact, switch the symlink, run the Magento commands, clear the cache. In a multi-server setup with separate web servers, worker hosts, cron servers and queue consumer processes, coordination becomes a challenge in its own right. If one web server finishes its deployment while the worker host is still running the old code, both codebases end up accessing the same database at the same time, possibly with different expectations about the database schema.

GitLab pipelines can take on this coordination because jobs can run sequentially or in parallel, and dependencies between jobs can be defined explicitly. The art lies in defining the right order: which server type gets updated first? When are queue consumers stopped? When is cron released again? These questions need to be answered in the pipeline design before the first deployment ever runs.

Magento makes multi-server deployments particularly demanding because several process types depend on the same code: the web server handles HTTP requests, consumer processes handle RabbitMQ messages, cron jobs run scheduled tasks, and workers such as indexers can run in the background. Each of these process types has different requirements for the deployment moment and needs to be treated accordingly.

2. Server types and their deployment requirements

Web servers (Nginx/PHP-FPM) handle incoming HTTP requests. Their deployment needs to happen as fast as possible and must not hard-interrupt existing connections. Rolling deployments, taking one server out of the load balancer, deploying, and putting it back, minimize the impact on running requests. Worker servers run long-lived background processes that should not be interrupted by a deployment. They need to be stopped in a controlled way before the new code is activated, and restarted afterward.

Cron servers run scheduled tasks. During a deployment, no cron jobs should run that could touch both old and new code artifacts. The simplest approach: disable Magento cron during the deployment and re-enable it afterward. Queue consumers process messages from a message queue such as RabbitMQ. They need to finish all messages currently in progress before the deployment moment (draining), then be stopped, and be restarted with the new code once the deployment is done.

# Multi-server deployment configuration
# Servers: web01, web02 (load balanced), worker01, cron01

stages:
  - build
  - test
  - package
  - pre-deploy    # Stop consumers, disable cron
  - deploy-web    # Deploy to web servers
  - deploy-worker # Deploy to worker servers
  - post-deploy   # Run Magento commands, restart services
  - verify        # Smoke tests on all servers
  - rollback      # Manual rollback across all servers

variables:
  WEB_HOSTS: "web01.mironsoft.de web02.mironsoft.de"
  WORKER_HOST: "worker01.mironsoft.de"
  CRON_HOST: "cron01.mironsoft.de"
  DEPLOY_USER: "deploy"
  DEPLOY_PATH: "/var/www/magento"

3. Deployment sequence: who goes first, who goes last

The deployment sequence in a multi-server setup needs to be clearly defined. The recommended order for Magento: start with the pre-deploy stage, disable cron jobs, stop queue consumers and drain messages still in progress. Then deploy the web servers, ideally in rolling order so at least one server is always available to the load balancer. After that, deploy the worker servers and the cron server. Then run the Magento commands, setup:upgrade, cache:flush, and restart the services. Finally, the verify stage with smoke tests across all server types.

This sequence ensures that two different code versions never access the same database at the same time. The pre-deploy stage is the most critical step: before the first byte of new code lands on any server, every background process that touches the database must already be stopped in a controlled way. Skipping this step risks inconsistent database states caused by race conditions.

4. Web server deployment: rolling or parallel

There are two strategies for web servers: parallel deployment, updating all web servers at once, and rolling deployment, updating servers one after another while briefly pulling each one out of the load balancer. Parallel deployment is simpler to implement, but there is a brief moment with no web server available at all. Rolling deployment is more complex, but the shop stays reachable throughout. For Magento production systems, rolling deployment is the recommendation, because Magento does not boot fast enough to make even a brief total outage acceptable.

GitLab cannot natively orchestrate rolling deployments; that requires a deployment tool such as Deployer, or a custom shell script executed inside a GitLab job. The script pulls a server out of the load balancer, deploys, waits for the PHP-FPM restart, and adds the server back before moving on to the next one. The critical parameter is the wait time after the PHP-FPM reload: too short and running requests get interrupted, too long and the deployment takes unnecessarily long.

deploy:web-servers:
  stage: deploy-web
  image: alpine:3.19
  before_script:
    - apk add --no-cache openssh-client rsync bash
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
  script:
    # Rolling deployment: one web server at a time
    - |
      for HOST in $WEB_HOSTS; do
        echo "==> Deploying to web server: $HOST"
        RELEASE_ID="$(date +%Y%m%d-%H%M%S)"
        RELEASE_PATH="${DEPLOY_PATH}/releases/${RELEASE_ID}"
        ssh "${DEPLOY_USER}@${HOST}" "mkdir -p ${RELEASE_PATH}"
        rsync -az --delete --exclude='.git' --exclude='var/' \
          ./ "${DEPLOY_USER}@${HOST}:${RELEASE_PATH}/"
        ssh "${DEPLOY_USER}@${HOST}" "
          ln -sfn ${DEPLOY_PATH}/shared/app/etc/env.php ${RELEASE_PATH}/app/etc/env.php
          ln -sfn ${DEPLOY_PATH}/shared/pub/media ${RELEASE_PATH}/pub/media
          ln -sfn ${RELEASE_PATH} ${DEPLOY_PATH}/current
          sudo systemctl reload php8.4-fpm
        "
        echo "${HOST}: deployed ${RELEASE_ID}"
        # Brief wait to allow PHP-FPM to reload gracefully
        sleep 5
      done
  rules:
    - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'

5. Worker processes: clean shutdown and restart

Worker processes on dedicated worker servers need to be stopped cleanly before a deployment. The stop must not be hard (SIGKILL), because running operations would otherwise be aborted inconsistently. The correct flow is to send SIGTERM, then wait until the process has exited normally, and only then continue the deployment. For supervisor-managed worker processes, that means supervisorctl stop all before the deployment and supervisorctl start all afterward.

After the new code has been deployed, the workers are restarted. It is important that the new code has been fully transferred and the symlink switched before the workers start, otherwise they end up running a mix of old and new code. The startup sequence in the post-deploy job must reflect this dependency explicitly: workers only start once the deployment has completed on every server.

6. Controlling cron jobs during deployment

Magento cron jobs are driven by two processes: the system cron, which regularly calls bin/magento cron:run, and Magento's internal cron groups. During a deployment, Magento cron should be disabled to prevent cron jobs from running at the exact moment the code is switching over. The simplest approach: comment out the system cron entry during the deployment and re-enable it afterward.

A more robust approach uses Magento locks: bin/magento cron:install --force rewrites the crontab with the correct entry after the deployment. Before the deployment, bin/magento cron:remove removes the cron entry. This approach is more reliable than manually editing the crontab, because it is idempotent and guarantees the cron entry ends up in the correct state after every deployment. Cron jobs that are still running when the deployment starts should be waited out with a timeout mechanism, typically five minutes at most.

pre-deploy:stop-services:
  stage: pre-deploy
  image: alpine:3.19
  before_script:
    - apk add --no-cache openssh-client bash
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
  script:
    # Stop queue consumers on worker server
    - ssh "${DEPLOY_USER}@${WORKER_HOST}" "
        sudo supervisorctl stop all
        echo 'Queue consumers stopped'
      "
    # Disable Magento cron on cron server
    - ssh "${DEPLOY_USER}@${CRON_HOST}" "
        cd ${DEPLOY_PATH}/current
        bin/magento cron:remove
        echo 'Magento cron removed'
      "
    # Wait for running cron jobs to finish (max 5 minutes)
    - |
      TIMEOUT=300
      ELAPSED=0
      while ssh "${DEPLOY_USER}@${CRON_HOST}" "pgrep -f 'cron:run' > /dev/null 2>&1"; do
        if [ $ELAPSED -ge $TIMEOUT ]; then
          echo "Timeout waiting for cron jobs to finish"
          exit 1
        fi
        sleep 10
        ELAPSED=$((ELAPSED + 10))
      done
      echo "All cron jobs finished"
  rules:
    - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'

7. Queue consumers: draining and restart

Queue consumers processing RabbitMQ messages need to be drained before a deployment, meaning every message currently being processed should finish before the consumer is stopped. Magento offers the --max-messages parameter for bin/magento queue:consumers:start for exactly this: if a consumer is started with --max-messages=1000, it terminates itself after 1000 messages. Combined with Supervisor, which automatically restarts the consumer, this produces regular, controlled restarts on its own.

There are two approaches to draining at deployment time: either wait until every running consumer process has finished its current message and then stop it, or send the consumer process a signal telling it to drain the queue and then exit. Supervisor supports supervisorctl stop with SIGTERM, which triggers draining in correctly implemented Magento consumers. Anyone building their own consumers needs to make sure they react correctly to SIGTERM.

8. Sequencing strategies compared

There are several ways to sequence multi-server deployments. The choice depends on the system architecture, the accepted downtime budget and the complexity of the pipeline. Three main strategies have become established for Magento production systems.

In practice, these strategies are combined: web servers get a rolling deployment with no downtime, while worker and cron servers are handled sequentially with a short, controlled stop. For most Magento production systems, this combination is the right approach: it maximizes availability for end users and minimizes the risk of inconsistent database states caused by old and new background processes running in parallel.

9. Summary

Multi-server deployments with GitLab for Magento require a clear sequencing strategy that accounts for every server type: web servers with rolling deployment, worker processes with controlled stop and restart, cron jobs with temporary disabling, and queue consumers with draining. The pre-deploy stage is the most critical step here, it makes sure every background process is stopped before new code gets activated. That is the only way to prevent old and new code from accessing the same database at the same time.

GitLab pipelines are well suited to modeling this coordination: stages define the order, job dependencies ensure no deploy job starts before pre-deploy has finished, and manual rollback jobs hand control back to the team whenever something does not work as expected. The biggest lever is planning: documenting the sequence before the first deployment and rehearsing it in staging saves the team frantic coordination during a production incident.

10. Common mistakes in multi-server deployments

The most common mistake in multi-server deployments is missing the pre-deploy stage. Anyone who starts deploying web servers directly, without stopping workers and consumers first, risks race conditions: the web server processes orders with new code and writes to the database while the worker process is still running old code and interprets the same database schema differently. The result can be inconsistent order data, failing indexer jobs, or locked database rows.

Another common mistake is forgetting to disable cron. If a cron job starts during the deployment, for example the Magento cron running every five minutes, it can hit a state where the symlink is mid-switch or the shared files are not yet linked. The result is a failing cron job that either triggers false alarms or leaves database entries in an inconsistent state. bin/magento cron:remove before the deployment and bin/magento cron:install afterward is the simplest and most reliable safeguard against this.

11. FAQ: GitLab Pipelines for Multi-Server Deployments