Retry, Backoff and Timeout Strategies in Shell Scripts

1. Why retry logic is necessary in shell scripts

Retry logic in shell scripts is not an optional nicety, it is a basic requirement for any script that interacts with external resources. Network connections fluctuate, APIs enforce rate limits and produce transient errors, services take time to come back up after a restart, and databases become briefly unreachable. A script without a retry mechanism fails at the first such transient error and requires manual intervention, which is exactly what automation is supposed to avoid.

There are three main problems that arise without retry in shell scripts. First, a script aborts immediately on a temporary network error and has to be restarted by hand. Second, a script waits indefinitely on a hung command because there is no timeout logic. Third, multiple instances of the same script overwhelm the target service simultaneously after an outage, because they all retry at the same fixed intervals: the classic thundering herd problem, which occurs when backoff is used without jitter.

The solution to all three problems is a reusable retry function with exponential backoff, optional jitter, and the GNU Coreutils timeout command for time limits. These components can be combined into a compact library that can be included in any shell script. The implementation effort is small, and the payoff in production environments is significant.

2. A simple retry loop: the basic pattern

The simplest retry pattern in shell scripts is a for or while loop that runs a command up to N times and pauses for a fixed wait time after each failed attempt. The structure is: run the command, check the exit code, exit the loop on 0, otherwise wait and increment the attempt counter. After N unsuccessful attempts the function returns the last error exit code. This is the foundation of retry logic on which all more sophisticated strategies are built.

The critical point for a simple retry in shell scripts is this: the script must run with set -euo pipefail, but the retry function must not be aborted by the failure of the command it wraps. The solution is to call the command with || true or in a context where set -e does not apply, for example after || or inside an if expression. That adds a small amount of complexity, but it is unavoidable for correct retry logic in shell scripts.

#!/usr/bin/env bash
# lib/retry.sh: Reusable retry library with exponential backoff and jitter
set -euo pipefail

# retry <max_attempts> <command> [args...]
# Simple retry with fixed wait interval
retry() {
  local max_attempts="$1"; shift
  local attempt=1
  local wait_seconds=1

  while (( attempt <= max_attempts )); do
    printf '[RETRY] Attempt %d/%d: %s\n' "$attempt" "$max_attempts" "$*" >&2
    if "$@"; then
      [[ $attempt -gt 1 ]] && printf '[RETRY] Succeeded after %d attempts\n' "$attempt" >&2
      return 0
    fi
    local exit_code=$?
    printf '[RETRY] Failed (exit %d)\n' "$exit_code" >&2

    if (( attempt < max_attempts )); then
      printf '[RETRY] Waiting %ds before next attempt...\n' "$wait_seconds" >&2
      sleep "$wait_seconds"
    fi
    (( attempt++ ))
  done

  printf '[RETRY] All %d attempts failed\n' "$max_attempts" >&2
  return 1
}

# Usage examples
retry 3 curl -sf "https://api.example.com/health"
retry 5 rsync -avz --partial /data/ user@host:/backup/
retry 3 bin/magento cache:flush

3. Exponential backoff: doubling the wait time

Exponential backoff in shell scripts is the standard algorithm for smart retry behavior: after every failed attempt the wait time is doubled, or multiplied by some other factor. Instead of waiting 1, 1, 1, 1 seconds, the script waits 1, 2, 4, 8 seconds, with a maximum wait time (a cap) to keep the pauses from growing into hours. Typical parameters: a base wait of 1 second, a multiplier of 2, a maximum wait of 60 seconds, and a maximum of 5 attempts.

The implementation of exponential backoff in shell scripts uses Bash arithmetic with $(( base * 2 ** (attempt - 1) )). Since Bash only supports integer arithmetic, the base and multiplier must be whole numbers. For finer control you can use awk or python3 -c for float arithmetic. In practice, integer backoff is sufficient for shell scripts; the usual bounds are 1, 2, 4, 8, 16, 32 seconds, which bridges most transient problems without waiting excessively long.

#!/usr/bin/env bash
set -euo pipefail

# retry_with_backoff <max_attempts> <base_wait> <max_wait> <command> [args...]
retry_with_backoff() {
  local max_attempts="$1"
  local base_wait="$2"      # seconds for first wait
  local max_wait="$3"       # cap for wait time
  shift 3
  local attempt=1
  local wait_seconds="$base_wait"

  while (( attempt <= max_attempts )); do
    printf '[BACKOFF] Attempt %d/%d: %s\n' "$attempt" "$max_attempts" "$*" >&2

    if "$@"; then
      [[ $attempt -gt 1 ]] && \
        printf '[BACKOFF] Succeeded after %d attempts\n' "$attempt" >&2
      return 0
    fi
    local exit_code=$?

    if (( attempt < max_attempts )); then
      printf '[BACKOFF] Failed (exit %d), waiting %ds\n' \
        "$exit_code" "$wait_seconds" >&2
      sleep "$wait_seconds"

      # Double the wait, cap at max_wait
      wait_seconds=$(( wait_seconds * 2 ))
      (( wait_seconds > max_wait )) && wait_seconds="$max_wait"
    fi
    (( attempt++ ))
  done

  printf '[BACKOFF] All %d attempts failed: %s\n' "$max_attempts" "$*" >&2
  return 1
}

# Wait sequence: 1s, 2s, 4s, 8s (capped at 30s)
retry_with_backoff 5 1 30 \
  curl -sf --max-time 10 "https://api.mironsoft.de/v1/health"

# Longer waits for database availability (2s, 4s, 8s, 16s, 30s)
retry_with_backoff 5 2 30 \
  mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASS" -e "SELECT 1" "$DB_NAME"

4. Jitter: preventing the thundering herd

Jitter is the complement to exponential backoff in shell scripts that solves the thundering herd problem. When many instances of the same script start at the same time, for example after a server restart or in a horizontally scaled environment, and all of them retry using the same backoff schedule after a failure, they all hit the target service at once. Jitter adds a random component to the wait time so retry attempts are spread out over time.

Jitter in Bash is implemented using the $RANDOM variable, which returns values from 0 to 32767. The simplest form of jitter: jitter=$(( RANDOM % max_jitter )) and wait=$(( backoff + jitter )). AWS's "full jitter" approach uses sleep $(( RANDOM % wait_seconds )): the entire wait time is random between 0 and the computed backoff value. The "equal jitter" approach halves the wait time and adds a random half: $(( wait_seconds / 2 + RANDOM % (wait_seconds / 2) )). For production shell scripts, full jitter or equal jitter combined with a backoff cap is the recommended strategy.

5. The timeout command: cancelling hung commands

The timeout command from GNU Coreutils is the standard tool for enforcing time limits in shell scripts. The syntax timeout DURATION COMMAND [ARGS] runs the command and sends a SIGTERM (the default) or another signal (--signal) once the duration has elapsed. With --kill-after, a SIGKILL is sent after an additional wait period following the SIGTERM if the process does not respond to it. The exit code of timeout is 0 on success, 124 if the time limit was exceeded, and the command's original exit code otherwise.

For retry logic in shell scripts, combining timeout with the retry function is especially valuable: each attempt gets its own time limit, so a single hung attempt does not block the entire retry sequence. timeout 30 curl -sf ... aborts the curl call after 30 seconds with exit code 124. The retry function treats exit code 124 as a normal failure and continues, including backoff. The per-attempt time limit should be noticeably shorter than the maximum wait window of the entire retry sequence.

#!/usr/bin/env bash
set -euo pipefail

# retry_with_backoff_jitter <max> <base> <max_wait> <cmd> [args...]
retry_with_backoff_jitter() {
  local max_attempts="$1"
  local base_wait="$2"
  local max_wait="$3"
  shift 3
  local attempt=1
  local wait_seconds="$base_wait"

  while (( attempt <= max_attempts )); do
    printf '[RETRY] Attempt %d/%d: %s\n' "$attempt" "$max_attempts" "$*" >&2

    # Each attempt gets its own timeout (30s)
    if timeout 30 "$@"; then
      return 0
    fi
    local exit_code=$?

    # Special handling for timeout exit code
    if [[ $exit_code -eq 124 ]]; then
      printf '[RETRY] Time limit exceeded (30s)\n' >&2
    else
      printf '[RETRY] Failed (exit %d)\n' "$exit_code" >&2
    fi

    if (( attempt < max_attempts )); then
      # Equal jitter: half fixed, half random
      local half=$(( wait_seconds / 2 ))
      local jitter=$(( half > 0 ? RANDOM % half : 0 ))
      local actual_wait=$(( half + jitter ))

      printf '[RETRY] Backoff: %ds (base %ds + jitter %ds)\n' \
        "$actual_wait" "$half" "$jitter" >&2
      sleep "$actual_wait"

      # Exponential increase with cap
      wait_seconds=$(( wait_seconds * 2 ))
      (( wait_seconds > max_wait )) && wait_seconds="$max_wait"
    fi
    (( attempt++ ))
  done

  printf '[RETRY] Gave up after %d attempts\n' "$max_attempts" >&2
  return 1
}

# Wait for service availability on startup
wait_for_service() {
  local host="$1" port="$2"
  retry_with_backoff_jitter 10 1 30 \
    bash -c "echo > /dev/tcp/$host/$port" 2>/dev/null
  printf '[OK] Service %s:%s reachable\n' "$host" "$port" >&2
}

# Use in deployment context
wait_for_service "${DB_HOST:-localhost}" "${DB_PORT:-3306}"
wait_for_service "${REDIS_HOST:-localhost}" "${REDIS_PORT:-6379}"

6. Combining retry with timeout

Combining retry and timeout in shell scripts requires two separate time limits: the timeout per attempt (how long a single command is allowed to run) and the overall timeout for the whole retry sequence (how long to wait for success in total). Without an overall timeout, a retry sequence with 10 attempts and a maximum backoff of 60 seconds could theoretically run for over 10 minutes, which would trigger a job timeout in a CI pipeline before the retry sequence itself gives up.

Implementing an overall timeout for retry in shell scripts uses SECONDS, the built-in Bash variable that counts the seconds elapsed since the current shell session started. At the start of the retry sequence, save start_time=$SECONDS. Before each attempt, check (( SECONDS - start_time >= total_timeout )) and abort if the overall timeout has been exceeded. Alternatively, the entire retry sequence can be wrapped in an outer timeout command.

7. Idempotency: a prerequisite for safe retries

The technical prerequisite for retry logic in shell scripts is idempotency: calling a command repeatedly must produce the same result as calling it once. Not every command is naturally idempotent. An INSERT INTO SQL statement without INSERT OR IGNORE or ON CONFLICT DO NOTHING raises a duplicate error on repetition. An mkdir without -p fails if the directory already exists. An API call that starts a transaction must, on retry, check whether the transaction has already completed.

For retry in shell scripts, the rule is: idempotent operations can be retried without limit. Non-idempotent operations must either be made idempotent (through appropriate flags or transaction IDs) or must not be part of a retry loop at all. Checking for idempotency is not a technical problem of the retry logic itself, but a requirement for the script's architecture that must be answered before the retry loop is implemented.

8. Retrying HTTP APIs: taking status codes into account

For HTTP API calls in shell scripts, retry is not appropriate for every kind of failure. HTTP status codes distinguish between errors where a retry makes sense and errors where it is pointless or even harmful. Status codes 500, 502, 503 and 504 are transient server errors, where retry with backoff is the right strategy. Status code 429 (Too Many Requests) often includes a Retry-After header with the exact wait time, which must be read and honored. Status codes 400, 401, 403 and 404 are permanent errors, where retry is pointless and only wastes resources.

Implementing HTTP retry in shell scripts with curl: curl -w "%{http_code}" appends the HTTP status code to the end of the output. With --silent --output /dev/null the body is suppressed when only the status code matters. A wrapper function checks the status code and decides whether to retry, abort immediately, or treat the call as successful. For production API calls it also helps to use --max-time for curl (an individual timeout per HTTP connection) and --retry-connrefused for automatic retries on connection refusal.

#!/usr/bin/env bash
set -euo pipefail

# http_retry: Retry HTTP request with status-code-aware logic
# Usage: http_retry <max_attempts> <url> [curl args...]
http_retry() {
  local max_attempts="$1" url="$2"; shift 2
  local attempt=1
  local wait_seconds=1
  local max_wait=60

  while (( attempt <= max_attempts )); do
    local response http_code body
    # Capture both body and status code
    response=$(curl -sf --max-time 20 -w "\n%{http_code}" "$@" "$url" 2>&1 || true)
    http_code=$(printf '%s' "$response" | tail -1)
    body=$(printf '%s' "$response" | head -n -1)

    printf '[HTTP] Attempt %d/%d, status: %s\n' "$attempt" "$max_attempts" "$http_code" >&2

    case "$http_code" in
      2??)
        # Success
        printf '%s' "$body"
        return 0
        ;;
      429)
        # Rate limited, respect Retry-After header if present
        local retry_after
        retry_after=$(curl -sI --max-time 5 "$url" 2>/dev/null \
          | grep -i "Retry-After:" | awk '{print $2}' | tr -d '\r' || echo "$wait_seconds")
        printf '[HTTP] Rate limited, waiting %ss (Retry-After)\n' "$retry_after" >&2
        sleep "$retry_after"
        ;;
      5??)
        # Transient server error, exponential backoff with jitter
        local jitter=$(( RANDOM % wait_seconds ))
        local actual_wait=$(( wait_seconds + jitter ))
        printf '[HTTP] Server error %s, waiting %ds (backoff + jitter)\n' \
          "$http_code" "$actual_wait" >&2
        sleep "$actual_wait"
        wait_seconds=$(( wait_seconds * 2 ))
        (( wait_seconds > max_wait )) && wait_seconds="$max_wait"
        ;;
      4??)
        # Permanent client error, no retry
        printf '[HTTP] Permanent error %s, not retrying\n' "$http_code" >&2
        return 1
        ;;
      "")
        # Connection failed / timeout
        printf '[HTTP] Connection error, waiting %ds\n' "$wait_seconds" >&2
        sleep "$wait_seconds"
        wait_seconds=$(( wait_seconds * 2 ))
        (( wait_seconds > max_wait )) && wait_seconds="$max_wait"
        ;;
    esac
    (( attempt++ ))
  done

  printf '[HTTP] All %d attempts failed: %s\n' "$max_attempts" "$url" >&2
  return 1
}

# Usage
response=$(http_retry 5 "https://api.mironsoft.de/v1/deploy/status" \
  -H "Authorization: Bearer ${API_TOKEN:?Token missing}")
printf 'Response: %s\n' "$response"

9. Retry strategies compared

Different retry strategies in shell scripts suit different scenarios. The right choice depends on how often transient errors occur, how latency-sensitive the process is, and whether the thundering herd problem is a concern.

For most production uses of retry logic in shell scripts, exponential backoff with equal jitter is the best choice: it spreads out load well, is simple to implement, and behaves predictably. A fixed interval only makes sense for scenarios where it is known exactly when a service will be available again, such as waiting for a service to start after a restart. Status-code aware retry is mandatory for any HTTP API call, because permanent errors cannot be fixed by retrying.

10. Summary

Retry, backoff and timeout strategies in shell scripts are the foundation of resilient automation. The three core components, a retry loop with a configured maximum number of attempts, exponential backoff with jitter for distributed systems, and the timeout command for hung individual calls, can be combined into a compact, reusable library. Once implemented and included in every script, this library makes the difference between scripts that abort at the first network hiccup and scripts that recover from transient problems on their own.

For HTTP API calls, status-code aware retry logic is also needed: not every HTTP error should trigger a retry. Never retry permanent errors (4xx). Retry transient errors (5xx, connection errors) with exponential backoff. Respect rate limits (429) using the Retry-After header. Idempotency is the prerequisite for any retry logic in shell scripts: non-idempotent operations must be handled explicitly or excluded from the retry loop.

11. FAQ: Retry, Backoff and Timeout in Shell Scripts