Incident Response Fundamentals: The First Real Incident

1. The incident response lifecycle at a glance

An incident response process follows five phases in practice: identify, contain, eradicate, recover and lessons learned. The identify phase confirms that this is actually a security incident and not a harmless false alarm. Contain stops further spread, eradicate removes the actual root cause, such as a backdoor or a compromised credential. Recover brings systems back into normal operation in a controlled way, and lessons learned closes the loop by feeding what happened back into processes, monitoring and training. Most small development teams have never written this sequence down and improvise during the first real incident, which costs valuable time and invites mistakes.

It is important that these phases do not run strictly in sequence. Containment often begins before the identify phase is fully complete, because you cannot wait until every detail is understood. Eradicate and recover can alternate several times, for example when a second persistence mechanism surfaces after the first cleanup. Lessons learned is not an optional closing step, it feeds directly back into preparation for the next incident, for example through new monitoring rules or an updated contact list.

2. Containment first: why containment comes before the full investigation

A common reflex in the first moment of shock is to dive straight into analysis: search logs, understand the root cause, reconstruct every detail before changing anything. That is exactly the wrong instinct, because while the full investigation runs, an attacker remains active in the system. Data keeps being exfiltrated, an attacker moves laterally to further systems, or a ransomware process keeps encrypting additional directories. Containment therefore always takes priority over a full forensic analysis, even if that means not every detail is clarified immediately.

In practice, teams distinguish between short-term and long-term containment. Short-term containment means: isolate the compromised host from the network, immediately lock the affected credentials, pause suspicious processes instead of killing them so forensic traces survive. Long-term containment follows afterward with clean, tested patches and a monitored return to service. The trick is to achieve maximum effect with minimal intervention while changing the system's state as little as possible for later analysis.

#!/usr/bin/env bash
# containment.sh - Short-term containment actions during an active security incident
# Run this BEFORE full investigation starts. Goal: stop the bleeding, preserve state.
set -euo pipefail

INCIDENT_ID="${1:?Usage: containment.sh INCIDENT-2026-07-12-01}"
EVIDENCE_DIR="/var/incident-response/${INCIDENT_ID}"
mkdir -p "$EVIDENCE_DIR"

echo "[containment] Snapshotting volatile state before anything else changes"
ps auxww > "${EVIDENCE_DIR}/processes.txt"
ss -tunap > "${EVIDENCE_DIR}/network-connections.txt"
who -a > "${EVIDENCE_DIR}/active-sessions.txt"
last -n 50 > "${EVIDENCE_DIR}/recent-logins.txt"

echo "[containment] Copying application and auth logs (read-only, not moved)"
cp -a /var/log/auth.log "${EVIDENCE_DIR}/"
cp -a /var/www/html/var/log/*.log "${EVIDENCE_DIR}/" 2>/dev/null || true

echo "[containment] Revoking all active admin sessions"
mysql magento -e "TRUNCATE admin_user_session;" 2>/dev/null || true

echo "[containment] Blocking outbound traffic to known-bad IP if provided"
if [[ -n "${2:-}" ]]; then
  iptables -I OUTPUT -d "$2" -j DROP
  echo "Blocked outbound to $2" >> "${EVIDENCE_DIR}/actions-taken.log"
fi

echo "[containment] Enabling Magento maintenance mode, allow only responder IP"
php /var/www/html/bin/magento maintenance:enable --ip="${RESPONDER_IP:?}"

echo "[containment] Done. Evidence saved to ${EVIDENCE_DIR}"

3. Preserving evidence: logs, snapshots and chain of custody

Under the time pressure of an active incident, evidence is easily destroyed, usually without any bad intent. Rebooting a server irreversibly wipes volatile data such as memory contents and active network connections. An automatic log rotation job overwrites the exact file that documents the moment of attack. A hasty patch changes system state before anyone documented it. The order matters: first preserve volatile data, meaning the process list, network connections and sessions, then persistent data such as log files and database snapshots, and only after that make changes to the system.

Chain of custody refers to the unbroken, documented history of every piece of evidence: who accessed what and when, and how integrity was proven. A checksum comparison with SHA-256 immediately after copying proves that the copy is unchanged and was not tampered with afterward. This documentation matters not only for a possible later criminal complaint, but also for cyber insurance, external forensics providers and your own internal traceability when questions about the incident come up months later.

#!/usr/bin/env bash
# evidence-preserve.sh - Chain-of-custody log preservation with integrity hashes
# Every access to the evidence directory after this point must be logged manually.
set -euo pipefail

INCIDENT_ID="${1:?Usage: evidence-preserve.sh INCIDENT-2026-07-12-01}"
SOURCE_LOGS=("/var/log/nginx/access.log" "/var/log/nginx/error.log" "/var/www/html/var/log/system.log" "/var/www/html/var/log/exception.log")
EVIDENCE_DIR="/var/incident-response/${INCIDENT_ID}/logs"
CUSTODY_LOG="/var/incident-response/${INCIDENT_ID}/chain-of-custody.log"

mkdir -p "$EVIDENCE_DIR"

log_custody() {
  printf '%s | %s | %s | %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$(whoami)" "$1" "$2" >> "$CUSTODY_LOG"
}

for src in "${SOURCE_LOGS[@]}"; do
  [[ -f "$src" ]] || continue
  dest="${EVIDENCE_DIR}/$(basename "$src").$(date -u +%Y%m%d%H%M%S)"
  cp -a "$src" "$dest"
  chmod 440 "$dest"
  sha256sum "$dest" >> "${EVIDENCE_DIR}/checksums.sha256"
  log_custody "COPY" "$src -> $dest"
done

echo "[evidence] Verifying checksums immediately after copy"
sha256sum -c "${EVIDENCE_DIR}/checksums.sha256"
log_custody "VERIFY" "checksums.sha256 validated"

echo "[evidence] Evidence sealed. Any further access must call log_custody manually."

4. Severity classification and escalation paths

Without fixed severity levels, a team's first move in a real incident is to argue about how serious the situation actually is, instead of acting. A simple four-tier classification from SEV1 to SEV4 is enough for most small teams: SEV1 for confirmed data loss or a full production outage, SEV2 for active exploitation attempts without confirmed data loss, SEV3 for automatically blocked but review-worthy anomalies, and SEV4 for findings with no immediate impact. Each tier gets a fixed response time and a fixed escalation list, so nobody has to figure out during the incident who they are even allowed to notify.

A defined escalation path with an on-call rotation matters especially because most serious incidents get noticed outside core business hours, often at night or on weekends. Without a clear path, valuable time is lost because nobody knows who has the authority to shut down production systems or engage external providers. A fixed escalation chain with a timeout, for example automatic forwarding to the next level after 30 minutes without response, prevents an incident from quietly sitting unattended.

{
  "severity_levels": {
    "SEV1": {
      "definition": "Confirmed data breach or full production outage",
      "response_time_minutes": 15,
      "notify": ["on_call_engineer", "cto", "dpo", "legal_counsel"],
      "escalation_after_minutes": 30
    },
    "SEV2": {
      "definition": "Active exploitation attempt, no confirmed data loss yet",
      "response_time_minutes": 30,
      "notify": ["on_call_engineer", "tech_lead"],
      "escalation_after_minutes": 60
    },
    "SEV3": {
      "definition": "Suspicious activity, contained automatically, needs review",
      "response_time_minutes": 240,
      "notify": ["tech_lead"],
      "escalation_after_minutes": 480
    },
    "SEV4": {
      "definition": "Low-risk finding, no immediate customer impact",
      "response_time_minutes": 1440,
      "notify": ["security_backlog"],
      "escalation_after_minutes": null
    }
  },
  "contacts": {
    "on_call_engineer": { "channel": "pagerduty", "rotation": "weekly" },
    "cto": { "channel": "phone", "backup": "deputy_cto" },
    "dpo": { "channel": "email", "sla_minutes": 60 },
    "legal_counsel": { "channel": "email", "sla_minutes": 120 }
  }
}

5. Communication plan: who needs to know and when

Internal communication during an incident needs a single defined channel as the source of truth, typically a dedicated incident channel. Parallel discussions in private chats or email threads lead to important decisions being made in different places, and nobody keeping the full picture anymore. The incident commander decides on actions, the tech lead executes them, and a comms lead collects all facts for later external communication. This separation of roles prevents technical people from having to write status updates and fix systems at the same time.

External communication, especially toward customers, should be honest, timely and handled by a single named person, not by several team members with differing wording. Speculation about root cause or scope has no place in an initial customer notice as long as it is unconfirmed. Legal counsel should ideally review public statements before publication, particularly when personal data might be involved, because imprecise wording can create additional liability risk later.

6. Legal notification duties: GDPR Art. 33/34 and the 72 hour clock

Under GDPR Art. 33, a personal data breach generally has to be reported to the competent supervisory authority within 72 hours of the controller becoming aware of it, unless the breach is unlikely to result in a risk to the rights and freedoms of the affected individuals. What matters is the moment of "becoming aware", not the moment every detail is understood. The clock is already running while containment and initial evidence preservation are still in progress, and an incomplete but timely notification is better than a complete but late one.

GDPR Art. 34 additionally requires notifying the affected individuals themselves when there is a high risk to their rights and freedoms, for example when passwords or payment data were compromised. Notification to the authority is explicitly allowed to happen in phases: an initial report with the information known at that point, followed by supplements as further facts emerge. It is also worth noting the documentation obligation under Art. 33(5): even incidents that are not reported because there is no risk must be documented internally, including the reasoning behind that assessment.

7. Immediate actions in a Magento context: admin lockdown

When admin access in a Magento store is suspected to be compromised, several steps belong in the first minutes: enable maintenance mode with an IP allowlist so only the response team keeps access, invalidate every active admin session in admin_user_session, reset all admin passwords, and rotate every API integration token. In parallel, check whether unknown admin users were created, whether scheduled cron jobs were altered, and whether unexpected observers, plugins or modules are registered in the system that could serve as a persistence mechanism.

After the initial lockdown, compare the codebase against a known-clean reference, for example via git diff against the last trusted deployment commit or a composer lockfile comparison. Only once this comparison is complete and the root cause identified should maintenance mode be lifted again. Lifting a lockdown too early is one of the most common reasons an attacker returns through a second, undiscovered access path.

<?php

declare(strict_types=1);

namespace Mironsoft\IncidentResponse\Console\Command;

use Magento\Framework\App\MaintenanceMode;
use Magento\Security\Model\AdminSessionsManager;
use Magento\User\Model\ResourceModel\User\CollectionFactory;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

/**
 * CLI command to lock down the Magento admin area during an active incident.
 * Enables maintenance mode, kills all active admin sessions and disables
 * every admin user except the responder account passed via --keep-user.
 */
class LockdownCommand extends Command
{
    /**
     * @param MaintenanceMode $maintenanceMode Toggles storefront maintenance mode.
     * @param AdminSessionsManager $sessionsManager Manages active admin sessions.
     * @param CollectionFactory $userCollectionFactory Loads admin user records.
     */
    public function __construct(
        private readonly MaintenanceMode $maintenanceMode,
        private readonly AdminSessionsManager $sessionsManager,
        private readonly CollectionFactory $userCollectionFactory
    ) {
        parent::__construct('incident:lockdown');
    }

    /**
     * Executes the lockdown: maintenance mode, session kill, admin disable.
     *
     * @param InputInterface $input CLI input, expects --keep-user option.
     * @param OutputInterface $output CLI output for status messages.
     * @return int Exit code, 0 on success.
     */
    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $keepUser = (string) $input->getOption('keep-user');

        $this->maintenanceMode->set(true, ['127.0.0.1']);
        $output->writeln('<info>Maintenance mode enabled.</info>');

        $collection = $this->userCollectionFactory->create();
        foreach ($collection as $user) {
            if ($user->getUserName() === $keepUser) {
                continue;
            }
            $user->setIsActive(0);
            $user->save();
            $output->writeln(sprintf('<comment>Disabled admin user: %s</comment>', $user->getUserName()));
        }

        // Invalidate every active admin session, including the ones we just disabled
        $this->sessionsManager->processLogout();
        $output->writeln('<info>All admin sessions invalidated.</info>');

        return Command::SUCCESS;
    }
}

8. A lightweight incident response plan template for small teams

An incident response plan does not need a hundred pages to be effective. A small development team is well served by a one-page document with four elements: clearly named roles (incident commander, tech lead, comms lead, data protection officer), the severity matrix from section 4, a current contact list including backup coverage, and a checklist for the first 30 minutes. What matters most is that this document is actually findable during an incident, even if chat tools or the internal wiki happen to be unreachable, for example because the incident affects exactly those systems.

The plan ideally lives as a markdown file in the repository, for example SECURITY_INCIDENT_RESPONSE.md, versioned like any other code, plus a printed copy kept within reach. A quarterly review keeps contact details current and makes sure new team members know their role. The plan should also link directly to the scripts from the previous sections, so nobody has to hunt for the right command in the middle of an incident.

{
  "plan_name": "Lightweight Incident Response Plan",
  "last_reviewed": "2026-07-12",
  "roles": {
    "incident_commander": "Decides on containment actions, owns the timeline",
    "tech_lead": "Executes technical containment and eradication steps",
    "comms_lead": "Owns internal and external communication, drafts customer notice",
    "dpo": "Assesses GDPR notification duty, owns authority reporting"
  },
  "first_30_minutes": [
    "Confirm the incident is real, assign an incident commander",
    "Open a dedicated incident channel, stop discussing it elsewhere",
    "Classify severity using severity-communication-matrix.json",
    "Start short-term containment, do not begin deep investigation yet",
    "Start the evidence-preserve.sh log preservation script"
  ],
  "escalation_contacts": [
    { "role": "on_call_engineer", "method": "pagerduty" },
    { "role": "cto", "method": "phone" },
    { "role": "dpo", "method": "email" },
    { "role": "hosting_provider", "method": "support_ticket" }
  ],
  "post_incident": [
    "Schedule blameless post-mortem within 5 business days",
    "Document root cause and timeline",
    "File action items with owner and due date",
    "Update this plan if gaps were found"
  ]
}

9. Tabletop exercises and post-mortems: learning from the incident

A tabletop exercise is a simulated scenario played out around a table without touching real systems: "An attacker has stolen valid admin credentials, what do you do in the first 15 minutes." Exercises like this reliably surface gaps, such as outdated phone numbers on the escalation list, unclear ownership, or nobody remembering where the scripts from sections 2 and 3 actually live. Two to four exercises per year are enough for a small team, but they should use realistic scenarios tailored to your own infrastructure rather than generic examples.

After a real incident, a blameless post-mortem belongs in the standard workflow, ideally within five business days while details are still fresh. Blameless explicitly means replacing "who made the mistake" with "what in the system or process allowed the mistake to happen", because otherwise the team is more likely to hide the next incident than report it. A solid post-mortem includes a minute-by-minute timeline, the root cause, and concrete action items with an owner and a due date that actually get tracked afterward.

All nine phases interlock: without a clear lifecycle there is no basis for containment decisions, without preserved evidence there is no reliable root cause analysis, and without a documented post-mortem the same mistake repeats itself at the next incident. The table below contrasts common reflexes during a real incident with the recommended responses.

10. Summary

Incident response fundamentals address one core problem: without a prepared process, the first real security incident turns into an improvised crisis full of avoidable mistakes. The lifecycle of identify, contain, eradicate, recover and lessons learned provides a clear order, with containment deliberately placed before the full investigation to prevent further damage. Chain of custody and cleanly documented log preservation lay the groundwork for later analysis, insurance questions and any potential legal steps.

A fixed severity classification with clear escalation paths prevents valuable time being lost to ownership questions during a real incident. The GDPR 72 hour notification deadline under Art. 33 demands early action over perfectionism, and a lightweight, versioned plan template makes all of this actually achievable for small teams. Regular tabletop exercises and blameless post-mortems close the loop and make sure every incident leaves the team a bit more resilient.

11. FAQ: Incident Response Fundamentals: The First Real Incident