wait -n for Parallel Job Harvesting in Bash

1. The problem with classic parallel waiting

Anyone starting several background jobs in Bash and then simply calling wait with no argument blocks until literally every started job is done. That sounds harmless at first, but with jobs of uneven duration it leads to inefficient utilization: a single slow job keeps the whole batch boundary open, while faster jobs have long since finished and the freed-up capacity sits unused until the entire batch has run through.

The classic pattern for limiting parallelism starts jobs in fixed batches: start a fixed number in parallel, wait for the whole batch with wait, start the next batch. This pattern works, but wastes runtime as soon as jobs within a batch take different amounts of time. wait -n solves exactly this problem by returning as soon as any background job finishes, whichever one it is, so a new job can be started immediately.

wait -n thus enables a true worker pool pattern in pure Bash, keeping the number of concurrently running jobs constant at a desired maximum, without reaching for GNU Parallel or xargs -P. The following sections cover the syntax, collecting exit codes, and the full worker pool pattern in detail.

2. wait -n basics: what exactly it waits for

The -n flag was added to the wait builtin in Bash 4.3 and fundamentally changes its behavior: instead of waiting for all specified jobs, wait -n returns as soon as the first of them finishes. Without further arguments, wait -n refers to all currently running background jobs of the shell, making it the ideal building block for loops that want to continuously feed in new work as soon as capacity frees up.

The return value of wait -n is the exit code of the job that finished first, not automatically its PID. To find out which job that was, you either compare jobs -l before and after the call, or, from Bash 5.1 onward, combine wait -n with an explicit PID list and check the shell's return value. This difference between "some job" and "which job exactly" is the central stumbling block the first time you use wait -n.

#!/usr/bin/env bash
set -euo pipefail

# Start three jobs with different durations
sleep 1 &
sleep 3 &
sleep 5 &

echo "Active jobs: $(jobs -r | wc -l)"

# Classic "wait" blocks until ALL jobs are done (up to 5s here)
# wait

# "wait -n" returns as soon as the FIRST job finishes (after ~1s)
wait -n
echo "First job finished after ~1s, remaining: $(jobs -r | wc -l)"

wait -n
echo "Second job finished after ~3s total, remaining: $(jobs -r | wc -l)"

wait -n
echo "Third job finished after ~5s total, remaining: $(jobs -r | wc -l)"

3. Collecting exit codes with wait -n

The exit code that wait -n returns belongs exactly to the job that finished first, regardless of the order in which the jobs were started. For scripts that need to know whether all parallel jobs succeeded, it is enough to check the return value of wait -n in every iteration and increment a counter on a non-zero exit code instead of aborting immediately.

It is important to call wait -n in a loop exactly as many times as jobs were started, otherwise unfinished jobs remain as zombies until the script itself ends. The number of still running jobs can be checked at any time with jobs -r | wc -l or, more robustly, with a maintained counter that is incremented on every start and decremented on every wait -n.

#!/usr/bin/env bash
set -euo pipefail

declare -a exit_codes=()
job_count=0

for url in "${urls[@]}"; do
  curl -fsS -o "/dev/null" "$url" &
  ((job_count++))
done

failed=0
for ((i = 0; i < job_count; i++)); do
  if ! wait -n; then
    ((failed++))
  fi
done

echo "Finished: $job_count jobs, failed: $failed"
if (( failed > 0 )); then
  exit 1
fi

4. Worker pool pattern: keeping parallelism constant

The real power of wait -n shows up in the worker pool pattern: instead of working in fixed batches, a new job is started exactly when the number of running jobs falls below the maximum. This pattern keeps utilization constantly close to the desired degree of parallelism, regardless of how differently long individual jobs take, delivering a significant speed advantage over rigid batch waiting in batches with strongly varying runtimes.

The implementation only needs a counter for active jobs and a simple condition: as long as the counter has reached the maximum, wait with wait -n before starting the next job. This pattern is considerably more compact than the classic batch array pattern with explicit PID management, while delivering better utilization at the same time.

#!/usr/bin/env bash
set -euo pipefail

MAX_JOBS=4
active=0
failed=0

process_file() {
  local file="$1"
  gzip -9 "$file"
}

for file in /var/log/archive/*.log; do
  process_file "$file" &
  ((active++))

  # Throttle: once at max capacity, wait for ANY job to finish first
  if (( active >= MAX_JOBS )); then
    wait -n || ((failed++))
    ((active--))
  fi
done

# Drain remaining jobs after the loop
while (( active > 0 )); do
  wait -n || ((failed++))
  ((active--))
done

echo "Done. Failed jobs: $failed"

5. wait -n with a PID list: waiting for specific jobs

From Bash 5.1 onward, wait -n additionally accepts an explicit list of PIDs or job specs and then only waits for the first one to finish within that subset, instead of all background jobs of the shell. This is especially useful when a script manages several independent groups of background jobs at once, for example one group for downloads and one for database migrations, that should be monitored separately from each other.

With a PID list, wait -n "${pids[@]}" additionally returns the PID of the finished job through the shell variable $! if you search the list before the call, or you combine this with a check for which PID from the list no longer appears in jobs -r. This more targeted use of wait -n pays off once several job groups need independent error handling.

#!/usr/bin/env bash
set -euo pipefail

declare -a download_pids=()
declare -a migration_pids=()

for url in "${download_urls[@]}"; do
  curl -fsS -O "$url" &
  download_pids+=($!)
done

for db in "${migration_dbs[@]}"; do
  ./migrate.sh "$db" &
  migration_pids+=($!)
done

# Bash 5.1+: wait -n on an explicit subset of PIDs
while (( ${#download_pids[@]} > 0 )); do
  wait -n "${download_pids[@]}"
  # Rebuild the list, dropping PIDs that are no longer running
  download_pids=($(jobs -p))
done

6. Combining timeouts: wait -n -t

Bash 5.1 additionally introduced the -t flag for wait, which sets a timeout in seconds. Combined with -n, wait -n -t seconds waits for the first finished job but returns control if no job finishes within the given time. The exit code 128 specifically signals that the timeout was reached, not that a job failed, a distinction scripts must check for explicitly.

This timeout behavior matters for worker pools that should not block indefinitely even in the presence of hanging jobs, for example when a single network request never returns. Without a timeout, wait -n would in that case stall the script's entire progress until the hanging job is terminated manually.

#!/usr/bin/env bash
set -euo pipefail

MAX_JOBS=4
active=0

for host in "${hosts[@]}"; do
  ssh -o ConnectTimeout=5 "$host" "uptime" &
  ((active++))

  if (( active >= MAX_JOBS )); then
    # Bash 5.1+: bail out of waiting after 10s even if nothing finished
    if wait -n -t 10; then
      ((active--))
    else
      status=$?
      if (( status == 128 )); then
        echo "[WARN] Timeout waiting for a job, checking again" >&2
      else
        ((active--))
      fi
    fi
  fi
done

7. Error handling without aborting the rest

In many batch processing scripts, a single failed job should not abort the entire script, just be noted so a summary error appears at the end. wait -n supports this pattern directly, because its return value can be checked per job without triggering set -e immediately, as long as the call sits in an if or || construct.

For a clean summary, it is worth logging failed jobs together with an identifier, such as the file name or URL that was processed. Since wait -n only delivers the exit code, not automatically the context of the job, this mapping must be established manually through a data structure such as an associative array from PID to context, if detailed error reports are needed.

8. Compatibility: Bash versions and fallbacks

wait -n itself requires Bash 4.3, while the PID list variant and the -t flag are only available from Bash 5.1 onward. macOS ships Bash 3.2 by default for licensing reasons, which means wait -n does not work there without a newer Bash version installed via Homebrew. Scripts meant to run on multiple platforms must check the Bash version at runtime before using wait -n.

As a fallback for older Bash versions, only the classic batch pattern remains, with a PID array and explicit waiting on the oldest element, as was common in automation scripts before Bash 4.3. This fallback is less efficient but portable and also works with the Bash 3.2 preinstalled on macOS.

9. Job harvesting strategies compared

The table below compares the different strategies for parallel job harvesting in Bash.

wait -n thus closes exactly the gap between the simple but inefficient batch pattern and external tools like GNU Parallel, which mean an extra dependency. For Bash scripts from version 4.3 onward, wait -n is in most cases the right choice for efficient parallel job harvesting without third-party dependencies.

10. Summary

wait -n solves the core problem of rigid batch processing by waiting for the first finished job instead of all of them. Combined with a simple counter, it forms a worker pool pattern that keeps parallelism constantly at a desired maximum, regardless of how differently long individual jobs take. The return value of wait -n directly delivers the exit code of whichever job finished.

From Bash 5.1 onward, an explicit PID list and the -t flag extend wait -n with targeted waiting on subsets and timeout protection against hanging jobs. For older Bash versions, the classic batch pattern remains as a portable but less efficient fallback. Anyone who regularly uses parallel processing in Bash scripts should establish wait -n as the standard tool instead of rigid batch boundaries.

11. FAQ: wait -n in Bash