Caching Strategies in Bash CI Scripts: Cache Keys, Invalidation and Locking

1. Why caching makes the difference in CI scripts

A typical CI run often spends more time installing dependencies than actually building or testing the code. An npm ci or composer install without a cache pulls the same packages from the network on every single pipeline run, even though the dependencies usually have not changed at all between two commits. Across hundreds of pipeline runs a month, that adds up to significant wasted compute time and, with paid CI minutes, real cost.

A well-built caching setup in Bash CI scripts solves exactly this problem by reusing the result of an expensive operation, such as an installed node_modules directory or a compiled build artifact, across pipeline runs as long as the inputs have not changed. The art is not in caching itself but in reliably detecting when a cache is still valid and when it must be discarded.

2. What is worth caching and what is not

Good candidates for caching are results that arise deterministically from clearly identifiable input files: installed dependencies from a lockfile, compiled build artifacts from source code, or downloaded Docker layers from a Dockerfile. In all these cases it is unambiguous when the result is still valid, namely as long as the input file has not changed.

Poor candidates are intermediate results that depend on external, unversioned state, such as test results carrying timestamps, or data that is deliberately meant to differ on every run, like generated random test data. Caching such results risks stale or incorrect output that is harder to debug than the time saved is worth.

3. Building cache keys from checksums

The central building block of any caching strategy is a cache key derived unambiguously from the relevant inputs. For a dependency installation that is usually the hash of the lockfile, such as package-lock.json or composer.lock, computed with sha256sum. If even a single character in the lockfile changes, the hash changes, and the cache is automatically recognized as invalid without anyone having to maintain invalidation logic by hand.

For more complex cases it pays off to combine several factors into the cache key, such as the lockfile hash together with the language version and the operating system of the CI runner. A cache built under Node 18 on Linux may not be valid under Node 20 on macOS, even with an identical lockfile, and a composite cache key prevents exactly this kind of silent mismatch.

#!/usr/bin/env bash
set -euo pipefail

compute_cache_key() {
  local lockfile="$1"
  local node_version os_name lockfile_hash
  node_version="$(node --version)"
  os_name="$(uname -s)"
  lockfile_hash="$(sha256sum "$lockfile" | awk '{print $1}')"

  # Combine everything that could invalidate the cache into one key
  echo "${os_name}-${node_version}-${lockfile_hash}" | sha256sum | awk '{print $1}'
}

CACHE_KEY="$(compute_cache_key package-lock.json)"
echo "Cache key: $CACHE_KEY"

4. Cache directory structure and storing by cache key

Once the cache key is computed, it should feed directly into the cache's storage path, for example /cache/deps-$CACHE_KEY.tar.gz. This path-based versioning has the advantage that several cache versions can coexist in the same cache directory without overwriting each other, which matters especially with parallel pipeline jobs running slightly different environments.

Many CI systems additionally support so-called restore keys: a prefix that, on a miss against the exact key, looks for the next-best, partially matching cache. A restore key like deps-linux-node20- then finds a somewhat older cache with a slightly different lockfile, which is not a perfect match but still contains most of the already-downloaded packages, enabling an incremental rather than a full rebuild.

#!/usr/bin/env bash
set -euo pipefail

readonly CACHE_DIR="/cache"
readonly CACHE_KEY="$1"
readonly CACHE_FILE="${CACHE_DIR}/deps-${CACHE_KEY}.tar.gz"

save_cache() {
  local source_dir="$1"
  mkdir -p "$CACHE_DIR"
  tar -czf "${CACHE_FILE}.tmp" -C "$source_dir" .
  mv "${CACHE_FILE}.tmp" "$CACHE_FILE"  # atomic: never a half-written cache file
}

save_cache "node_modules"

5. Checking for a cache hit and conditionally skipping installation

The actual time savings only materialize once the script checks, before the expensive operation, whether a matching cache already exists, and skips the operation entirely on success. This check must consider both the existence of the cache file and its basic integrity, because an empty or partially written cache file is worse than no cache at all, since it fakes a failure that is hard to diagnose.

A robust pattern therefore checks not just whether the file exists, but also whether it has a plausible minimum size before extracting it. If extraction still fails, say because the file exists but is corrupt, the script should automatically fall back to a normal, full installation instead of aborting with a cryptic error.

#!/usr/bin/env bash
set -uo pipefail  # no -e: a cache miss or corrupt cache must not abort the script

restore_cache_or_install() {
  local cache_file="$1" target_dir="$2"

  if [[ -s "$cache_file" ]] && tar -tzf "$cache_file" >/dev/null 2>&1; then
    echo "Cache hit: restoring $target_dir from $cache_file"
    mkdir -p "$target_dir"
    tar -xzf "$cache_file" -C "$target_dir"
    return 0
  fi

  echo "Cache miss or corrupt cache, running full installation"
  npm ci
}

6. Controlling invalidation on changes cleanly

Because the cache key is already built from the hash of the relevant files, the actual invalidation happens automatically: if the lockfile changes, the key changes, and the old cache simply never gets referenced again. That alone is not enough, though, because unused old cache files accumulate over time and take up unbounded disk space in the cache directory if nobody cleans them up.

A separate cleanup script that runs regularly, for instance as its own nightly CI job, should delete cache files based on their last-accessed time once they have not been used for a certain number of days. That cleanly separates two independent concerns: the cache key handles correctness, the cleanup script handles bounded storage use, without a single script having to mix both tasks with unnecessary complexity.

#!/usr/bin/env bash
set -euo pipefail

readonly CACHE_DIR="/cache"
readonly MAX_AGE_DAYS=14

# Remove cache files that have not been accessed in MAX_AGE_DAYS
find "$CACHE_DIR" -maxdepth 1 -name '*.tar.gz' -atime "+${MAX_AGE_DAYS}" -print -delete

7. Race conditions with parallel pipelines: locking with flock

When several pipeline jobs run in parallel and potentially write to the same cache directory at once, a race condition emerges: two jobs could simultaneously detect the same cache key as missing, both perform the expensive installation, and then both write to the same cache file at once, which in the worst case produces a corrupt file half-overwritten by each job.

The atomic mv from the previous section already substantially defuses the problem for the write itself, because in the end a complete file from one of the two jobs always wins instead of a mixed one. Anyone wanting extra safety, for instance to prevent two jobs from redundantly running the same expensive installation at once, can guard the critical section with flock against a lock file, so only one job may write at a time while parallel jobs either wait or use the cache read-only.

#!/usr/bin/env bash
set -euo pipefail

readonly LOCK_FILE="/cache/.deps.lock"

(
  # Wait up to 120s for the lock, then proceed -- avoids indefinite hangs
  flock -w 120 200 || { echo "Could not acquire cache lock, proceeding without cache"; exit 0; }

  if [[ ! -f "$CACHE_FILE" ]]; then
    npm ci
    save_cache "node_modules"
  fi
) 200>"$LOCK_FILE"

8. Cache limits and pitfalls

The most common pitfall is a cache key that is too weak and leaves out important influencing factors, for instance hashing only the lockfile but not the language version. The result is a false cache hit, where an actually incompatible cache gets silently reused and causes hard-to-trace failures much later in the build, far removed from the actual cause.

A second, security-relevant pitfall is accidentally writing secrets into the cache, for instance when a .env file ends up in the cached directory because too broad a path got archived. Since caches often outlive individual pipeline runs and are sometimes shared across branches, such a leak can spread credentials across team and time boundaries. Explicitly excluding sensitive files when building the cache is therefore mandatory, not optional.

9. Comparing caching strategies

Depending on the kind of resource being cached, a different combination of cache key composition, storage location, and invalidation logic fits best. The table below places the most common cases.

10. Summary

11. FAQ: Caching in CI Scripts: The Essentials at a Glance