Implementing Rate-Limited API Requests with Retry in Bash

1. Why rate-limited API requests fail without retry

Almost every public API limits the number of requests per time window to protect its own infrastructure from overload. A script that performs rate-limited API requests without retry logic simply aborts the first time the limit is exceeded, even if the rest of the dataset would have been retrievable without issue. The result: incomplete data imports, failed synchronizations, and scripts that randomly succeed or fail on every larger batch.

Anyone implementing rate-limited API requests with retry treats a 429 response not as an error but as an expected, temporary state answered with a defined wait and another attempt. This is not a nice-to-have but practically mandatory in environments with payment provider APIs, shipping providers, or third-party systems in Magento integrations, because these systems almost always enforce hard limits.

The difference between a naive script and one that handles rate-limited API requests with retry robustly shows up especially during bulk processing: importing ten thousand products via an external API fails with a fixed pause after every request, either through too-slow processing or through requests that are too aggressive and trigger the limit. An adaptive retry strategy solves both problems at once.

2. Understanding HTTP status codes and headers for rate limits

The central status code for rate limiting is 429 Too Many Requests, defined in RFC 6585. Many APIs also send the Retry-After header, which specifies either a number of seconds or a concrete date for when the next attempt makes sense. Anyone implementing rate-limited API requests with retry should always check this header first before falling back to a self-calculated wait, because the server has the most accurate information about its own capacity.

Additionally, many APIs deliver their own headers such as X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset, which show how many requests remain even before the limit is reached. A script that proactively reads these headers can throttle its own request rate before a 429 error even occurs, instead of reacting only after the first failure.

#!/usr/bin/env bash
# inspect-rate-limit-headers.sh — read rate limit headers from an API response
set -euo pipefail

API_URL="${1:?Usage: inspect-rate-limit-headers.sh <url>}"

headers=$(curl -sI "$API_URL")

echo "=== Rate limit related headers ==="
echo "$headers" | grep -iE '^(retry-after|x-ratelimit-|x-rate-limit-)' || echo "(none found)"

status=$(echo "$headers" | head -n 1 | awk '{print $2}')
echo "=== HTTP status: $status ==="

3. Writing a retry function with exponential backoff

Exponential backoff doubles (or multiplies by another factor) the wait time after each failure, instead of always applying the same short pause. This strategy prevents a script from continuing to hammer an already overloaded server at a constant frequency. For rate-limited API requests with retry, exponential backoff is the industry standard because it adapts to the actual severity of the problem: a single failure leads to a short wait, repeated failures to significantly longer pauses.

A solid Bash implementation encapsulates the entire logic in a reusable function that handles the curl call, the status code check, and the backoff calculation. The base delay, the multiplier, and the maximum number of attempts should be configurable parameters, so the same function can be reused for APIs with different levels of strictness.

#!/usr/bin/env bash
# retry-with-backoff.sh — exponential backoff retry wrapper for curl
set -euo pipefail

retry_request() {
  local url="$1"
  local max_attempts="${2:-5}"
  local base_delay="${3:-1}"
  local attempt=1
  local delay="$base_delay"

  while (( attempt <= max_attempts )); do
    local http_code
    http_code=$(curl -s -o /tmp/response_body.json -w '%{http_code}' "$url")

    if [[ "$http_code" == "200" ]]; then
      echo "[OK] Request succeeded on attempt $attempt"
      cat /tmp/response_body.json
      return 0
    fi

    if [[ "$http_code" == "429" ]]; then
      echo "[WARN] Rate limited (attempt $attempt/$max_attempts), waiting ${delay}s" >&2
      sleep "$delay"
      delay=$(( delay * 2 ))
      (( attempt++ ))
      continue
    fi

    echo "[ERROR] Unexpected status $http_code, aborting" >&2
    return 1
  done

  echo "[ERROR] Exhausted $max_attempts attempts, still rate limited" >&2
  return 1
}

retry_request "https://api.example.com/v1/products" 5 1

4. Adding jitter to avoid thundering herd

Pure exponential backoff has a subtle problem: if several instances of the same script (for example several parallel workers or several servers) hit a rate limit at the same time, they all wait exactly the same calculated time and then send a request again simultaneously, which triggers the limit again. This phenomenon is called thundering herd. The solution: a random component, called jitter, is added to the calculated wait time, so parallel processes retry at slightly different points in time.

For rate-limited API requests with retry originating from several concurrently running cron jobs or worker processes, jitter is not an optional detail but decisive for the stability of the overall system. Bash generates random numbers simply via the built-in variable $RANDOM, which delivers a value between 0 and 32767.

#!/usr/bin/env bash
# retry-with-jitter.sh — exponential backoff with randomized jitter
set -euo pipefail

retry_with_jitter() {
  local url="$1"
  local max_attempts="${2:-5}"
  local base_delay="${3:-1}"
  local attempt=1
  local delay="$base_delay"

  while (( attempt <= max_attempts )); do
    local http_code
    http_code=$(curl -s -o /dev/null -w '%{http_code}' "$url")

    if [[ "$http_code" == "200" ]]; then
      echo "[OK] Succeeded on attempt $attempt"
      return 0
    fi

    if [[ "$http_code" == "429" ]]; then
      # Jitter: random value between 0 and (delay * 1000) milliseconds, added as fractional seconds
      local jitter_ms=$(( RANDOM % (delay * 1000 + 1) ))
      local total_delay
      total_delay=$(awk -v d="$delay" -v j="$jitter_ms" 'BEGIN { printf "%.3f", d + (j / 1000) }')
      echo "[WARN] Rate limited (attempt $attempt/$max_attempts), waiting ${total_delay}s (with jitter)" >&2
      sleep "$total_delay"
      delay=$(( delay * 2 ))
      (( attempt++ ))
      continue
    fi

    echo "[ERROR] Unexpected status $http_code" >&2
    return 1
  done

  return 1
}

retry_with_jitter "https://api.example.com/v1/orders" 6 1

5. Respecting the Retry-After header instead of fixed delays

Self-calculated exponential backoff is a good fallback strategy, but if the server explicitly sends a Retry-After header, it should take priority. The server knows its own capacity and the exact point in time when the limit resets better than any estimated backoff calculation on the client side. For rate-limited API requests with retry that prioritize maximum reliability, combining both strategies is the most robust approach: use Retry-After when available, otherwise fall back to calculated backoff with jitter.

Per the specification, the Retry-After header can contain either an integer number of seconds or an HTTP date. A robust script checks both formats and converts a date into seconds from now if needed.

#!/usr/bin/env bash
# retry-respect-retry-after.sh — prefer the server's Retry-After header
set -euo pipefail

retry_respect_header() {
  local url="$1"
  local max_attempts="${2:-5}"
  local attempt=1
  local fallback_delay=1

  while (( attempt <= max_attempts )); do
    local response_headers http_code retry_after
    response_headers=$(curl -sD - -o /dev/null "$url")
    http_code=$(echo "$response_headers" | head -n 1 | awk '{print $2}')

    if [[ "$http_code" == "200" ]]; then
      echo "[OK] Succeeded on attempt $attempt"
      return 0
    fi

    if [[ "$http_code" == "429" ]]; then
      retry_after=$(echo "$response_headers" | grep -i '^Retry-After:' | awk '{print $2}' | tr -d '\r')

      if [[ "$retry_after" =~ ^[0-9]+$ ]]; then
        echo "[WARN] Server requested wait of ${retry_after}s (Retry-After)" >&2
        sleep "$retry_after"
      else
        echo "[WARN] No numeric Retry-After, falling back to ${fallback_delay}s backoff" >&2
        sleep "$fallback_delay"
        fallback_delay=$(( fallback_delay * 2 ))
      fi

      (( attempt++ ))
      continue
    fi

    echo "[ERROR] Unexpected status $http_code" >&2
    return 1
  done

  return 1
}

retry_respect_header "https://api.example.com/v1/shipments" 5

6. Throttling bulk requests: limiting requests per second

For bulk processing, for example retrieving a thousand product records, it is more efficient to proactively limit the request rate rather than reactively waiting on 429 errors. A simple pattern calculates a fixed pause between requests from the known limit (for example 10 requests per second) and consistently maintains it, regardless of whether individual responses come back faster.

For rate-limited API requests with retry that occasionally still hit a limit despite proactive throttling, for example because other clients share the same API quota, the retry logic from the previous sections remains in place as a second line of defense. Proactive throttling drastically reduces the frequency of 429 responses but does not replace the need for a retry strategy for the remainder.

7. Logging and metrics for retry behavior

A script that performs rate-limited API requests with retry should log every retry attempt, including timestamp, status code, wait time, and attempt number. Without this logging, it remains invisible whether an API limit is chronically too tight or whether it is a matter of rare outliers, which matters for deciding whether to talk to the API provider about a limit increase.

For ongoing integrations, it pays off to capture the number of retries per period as a metric, for example in a simple CSV file or a Prometheus textfile. A sudden increase in the retry rate is often an early indicator that either usage patterns have changed or the API provider has tightened its limits, long before it leads to full outages.

8. Setting limits: maximum attempts and persistent failures

No retry mechanism should run indefinitely. A fixed upper limit on attempts, combined with a maximum total wait time, prevents a script from running endlessly against a persistently broken endpoint (for example after a misconfiguration at the API provider) and tying up resources. Anyone implementing rate-limited API requests with retry should therefore set both a maximum number of attempts and a maximum backoff wait per attempt, so exponential growth does not lead to unrealistically long pauses.

It is also important to distinguish between 429 (temporary rate limit, retry makes sense) and other 4xx status codes such as 401 (authentication failed) or 403 (not authorized), where a retry never solves the problem and only generates unnecessary load. A well-designed retry script explicitly checks the status code and aborts immediately on non-retriable errors instead of blindly treating every failure the same way.

9. Retry strategies at a glance

The table in section six shows the key differences between the strategies. In practice, the combination of respecting Retry-After and falling back to backoff with jitter is almost always the right choice for rate-limited API requests with retry, because it uses server-side information when available and still works robustly when the server sends no explicit headers.

Fixed delays without backoff are only acceptable for very small, infrequent scripts where the added complexity of exponential growth does not justify the benefit. For anything running regularly or at larger scale, the slightly more involved implementation with jitter and header evaluation quickly pays for itself, because it produces significantly less manual follow-up work on failed imports.

10. Summary

Implementing rate-limited API requests with retry in Bash means treating a 429 response as an expected state rather than an error. Exponential backoff prevents hammering an overloaded server, jitter prevents thundering herd effects during parallel processing, and respecting the Retry-After header uses the most accurate information available about the server's capacity.

For bulk processing, proactive throttling reduces the frequency of rate limit errors in the first place, but does not replace the need for a retry strategy as a second line of defense. Clear upper bounds on attempts and wait time, together with clean logging, turn a script that performs rate-limited API requests with retry into a reliable building block of any API integration.

11. FAQ: Rate-Limited API Requests with Retry