Documenting and Securing Manual Emergency Deployments

1. When a Manual Deployment Is Legitimate

A manual emergency deployment is not a defeat for the automation process, it is a deliberate release valve for situations where the regular pipeline run would be too slow. When a critical bug in the live shop is causing lost revenue, a broken checkout, a broken payment integration, a faulty price calculation, every minute counts. A normal pipeline with build, tests, and manual approval can take fifteen minutes. A manual deployment can be finished in three.

The problem is not the manual deployment itself, but the way it is usually carried out: without a log, without a prepared rollback, without documentation in the GitLab issue. Anyone who later wants to know who changed what, when, and on which server often finds nothing but gaps in the audit log. Those are exactly the gaps that need closing, not by banning manual deployments, but through a clearly defined process for the exceptional case.

2. The Risks of Uncontrolled Emergency Deployments

The biggest risk of an improvised emergency deployment is the loss of reproducibility. When a developer under time pressure edits files directly on the server, the result is a state that is not represented anywhere in the repository. The next regular deploy through the pipeline overwrites those changes without anyone knowing. Or worse: the hotfix is in the Git repository, but the regular deploy process runs against a database structure that the hotfix has already changed.

Another risk is the lack of scope on the emergency deployment. Under pressure, developers tend to change more than necessary. What starts as a single-file hotfix ends up with five changed files, a new configuration, and a cache:flush that masks other problems. Documentation forces a clear scope: what exactly is being changed, why, and with what expected outcome? These questions do not slow things down, they keep the emergency deploy itself from becoming the next problem.

3. Preparation: What Needs to Be in Place Before the Emergency

The best preparation for emergency deployments is a complete release structure that makes rollback possible within seconds. That means: the production server holds at least the last five releases as complete directories, a current symlink points to the active state, and a prepared rollback script can be run without any research. If this structure is maintained during normal operations, it is immediately available in an emergency.

Equally important is a prepared emergency Bash script that covers the most common hotfix scenarios: replacing a single file, flushing the cache, applying a single patch. This script should come with full logging built in, every action is written to a log file with a timestamp, the executing user, and the hostname. The script lives in the repository and is rehearsed regularly, not run for the first time during an actual emergency.

# .gitlab-ci.yml: emergency hotfix pipeline (fast path)
hotfix:deploy:
  stage: deploy
  environment:
    name: production
    url: https://shop.example.com
  variables:
    GIT_STRATEGY: fetch
  script:
    # Document who triggered this emergency deploy
    - echo "EMERGENCY DEPLOY by ${GITLAB_USER_LOGIN} at $(date -u)"
    - echo "Triggered from pipeline ${CI_PIPELINE_ID}, commit ${CI_COMMIT_SHA}"
    # Transfer only changed files (faster than full rsync)
    - |
      ssh "${DEPLOY_USER}@${DEPLOY_HOST}" "
        set -euo pipefail
        HOTFIX_LOG=${DEPLOY_PATH}/shared/var/log/hotfix-$(date +%Y%m%d-%H%M%S).log
        echo 'HOTFIX START: $(date -u)' >> \$HOTFIX_LOG
        echo 'User: ${GITLAB_USER_LOGIN}' >> \$HOTFIX_LOG
        echo 'Commit: ${CI_COMMIT_SHA}' >> \$HOTFIX_LOG
        echo 'Pipeline: ${CI_PIPELINE_ID}' >> \$HOTFIX_LOG
        cd ${DEPLOY_PATH}/current &&
        bin/magento cache:flush &&
        echo 'HOTFIX END: $(date -u)' >> \$HOTFIX_LOG"
  when: manual
  allow_failure: false
  rules:
    - if: '$CI_COMMIT_BRANCH == "hotfix/*"'
      when: manual

4. Logging During the Deployment

Every action in an emergency deployment must be logged. The minimum format: a UTC timestamp, username, the command that was run, and the result (success, or failure with an exit code). In Bash this is achieved with a wrapper function that writes each command to the log before running it, then logs the result as well. This function costs about fifteen lines of code and saves hours of debugging when an incident happens.

The log file should land somewhere that regular deployments cannot overwrite, in the shared/var/log/ directory, not inside the release directory. Anyone who wants to establish a root cause analysis process after incidents needs these logs. Without them, every post-mortem is a reconstruction from memory: unreliable, and prone to assigning blame instead of finding solutions.

5. GitLab as an Audit Trail for Manual Actions

GitLab offers several mechanisms for making manual actions in the deployment context visible. The first is the pipeline audit trail: every manually triggered pipeline has an entry showing who triggered it, when, and from which branch. This trail is immutable and cannot be edited after the fact. For compliance requirements in e-commerce projects, that is valuable evidence.

The second mechanism is GitLab environment jobs. When a hotfix is modeled as a manual job in the pipeline, it appears in the environment's deployment tab with a timestamp, pipeline ID, and the executing user. That makes even exceptional deployments transparent for the whole team and prevents the next regular deploy from running into an undocumented state. As a third mechanism, the hotfix commit should carry a meaningful commit message that includes the incident's ticket number and a short description.

# Dedicated verify job after every emergency deploy
verify:hotfix:
  stage: verify
  script:
    # Confirm no stale maintenance flag remains
    - |
      ssh "${DEPLOY_USER}@${DEPLOY_HOST}" "
        test ! -f ${DEPLOY_PATH}/current/var/.maintenance.flag"
    # HTTP check on critical paths
    - curl --fail --silent --max-time 10
        https://shop.example.com/
    - curl --fail --silent --max-time 10
        https://shop.example.com/checkout/cart/
    # Log verification result to shared audit log
    - |
      ssh "${DEPLOY_USER}@${DEPLOY_HOST}" "
        echo 'VERIFY OK: $(date -u) | Pipeline ${CI_PIPELINE_ID}' >>
          ${DEPLOY_PATH}/shared/var/log/hotfix-audit.log"
  needs: ["hotfix:deploy"]
  rules:
    - if: '$CI_COMMIT_BRANCH == "hotfix/*"'

6. Hotfix Pipeline: Fast Yet Controlled

The best solution for emergency deployments is not to switch off the pipeline process entirely, but a streamlined hotfix pipeline that keeps the most important safety steps while skipping the time-consuming ones. This pipeline runs in under five minutes: no full build, no unit tests, but a PHPCS check on the changed files, an rsync of the changed files to the server, a cache flush, and an HTTP verify. That is fast enough for an emergency and safe enough for a production environment.

The critical difference from real improvisation: the hotfix pipeline is defined in the repository in advance, not assembled on the fly during the emergency. It is triggered on a dedicated hotfix/* branch, which deploys automatically to the production environment. In an emergency, the team only needs to create a branch, commit the fix, and confirm the manual approval in the pipeline. That takes three minutes, with a full audit trail.

7. Rollback After an Emergency Deployment

An emergency deployment that becomes a problem in its own right must be reversible immediately. In a clean release structure, the rollback is a single operation: point the current symlink at the previous release directory. For hotfixes that were applied directly on top of the current release without creating a new release directory, rollback is more complicated. That is another argument for the hotfix pipeline instead of direct server intervention.

A rollback after an emergency deployment must also be documented, using the same log format as the original deployment. Who rolled back, why, to which state, and with what result. This information is a mandatory part of the post-mortem analysis. Without it, the team has no basis for deciding whether the rollback was complete or only partial, and whether the original error is still present in the system.

8. Improvised vs. Documented Compared

The difference between an uncontrolled and a documented emergency deployment often only becomes visible days later, at the next regular deployment or the next incident. Teams that define the standards up front pay nothing extra for it.

9. Returning to the Normal Process

An emergency deployment is always only an intermediate state. As soon as the incident is resolved, the hotfix must be folded into the normal development process: as a commit on the main branch, with a ticket reference, a code review, and a regular deployment through the full pipeline. As long as the hotfix only exists on the server and is not represented in the repository, it risks being overwritten by the next regular deploy.

The follow-up also includes a short post-mortem analysis: what caused the incident? Why was the error not caught in staging? Which tests or checks would have caught it? The answers to these questions feed into better tests, clearer deployment checklists, and more robust pipeline steps. That is the real payoff of an emergency deployment: the systematic improvement of the process that is supposed to prevent it in the first place.

10. Summary

Manual emergency deployments are an acceptable exception in Magento operations, but only with complete documentation, a prepared rollback path, and a subsequent return to the normal process. An emergency deployment without a log is a technical debt accumulation event: the shop is running, but the process is damaged. The solution is not additional bureaucracy, but a prepared hotfix pipeline that runs in three minutes and still documents all the relevant information.

The key lies in preparation: a release structure for instant rollback, a prepared Bash logging script, a defined hotfix branch process in GitLab, and a clear expectation of what follow-up is required after an emergency deploy. Teams that build these structures during normal operations pay only a fraction of the price that an uncontrolled emergency deploy would otherwise cost.

11. FAQ: Manual Emergency Deployments

Manual emergency deployments are not a sign of weakness in the process, they are a deliberate safety option. Teams that document and secure them properly stay in control even when the regular pipeline cannot keep up.

The decisive difference between chaos and a controlled intervention is not the technology, it is the playbook: clear steps, clear responsibilities, and clear documentation of the outcome.