S3 Sync Patterns with awscli in Bash Scripts: Deployment, Backup and Error Handling

1. Why aws s3 sync instead of aws s3 cp for deployment and backup

aws s3 cp copies a single file or, with --recursive, an entire directory, but transfers every file again each time regardless of whether it changed since the last run. For a build folder with thousands of static assets that means unnecessary bandwidth and runtime, even though only a handful of files actually changed between two deployments most of the time.

aws s3 sync instead compares source and destination by file size and modification date, and in some cases also by ETag checksum, transferring only files that are new or different. For deployment scripts that regularly mirror a build folder to S3, and for backup scripts that incrementally back up a directory, that is the decisive difference between a sync run taking seconds versus one taking minutes.

2. Basic syntax and behavior: local to S3 and S3 to local

The syntax of aws s3 sync is symmetric: aws s3 sync source destination works both from local to S3 and the other way around, from S3 to local, and even between two S3 buckets when both arguments start with s3://. That symmetry makes the same basic pattern reusable for deployment (local to S3), restore (S3 to local) and cross-region replication (S3 to S3).

By default, sync does not delete any file in the destination that no longer exists in the source, it only adds or overwrites. That makes the base command safe enough for most backup scenarios where old versions should be preserved, but unsuitable for a real deployment where files removed from the build folder should also disappear from the bucket.

#!/usr/bin/env bash
set -euo pipefail

# Local -> S3 (deployment)
aws s3 sync ./dist/ s3://mironsoft-static-assets/frontend/

# S3 -> local (restore)
aws s3 sync s3://mironsoft-static-assets/frontend/ ./dist/

# S3 -> S3 (replication between buckets)
aws s3 sync s3://mironsoft-static-assets/frontend/ s3://mironsoft-static-assets-eu/frontend/

3. Deployment workflow: syncing the build folder to S3

A typical deployment workflow first builds the frontend, then syncs the build folder to S3, and afterwards invalidates the CloudFront cache so users are not served stale assets for days. To keep old files no longer part of the build from lingering as dead weight in the bucket, this is where the --delete flag actually gets used, making the bucket content match the local build folder exactly.

It matters to use --delete only against a narrowly scoped destination path that contains just the deployment artifacts, never against a bucket root that might also hold other, unrelated data. A wrongly set prefix combined with --delete can otherwise delete data that has nothing to do with the actual deployment.

#!/usr/bin/env bash
set -euo pipefail

readonly BUCKET="s3://mironsoft-static-assets/frontend/"
readonly DISTRIBUTION_ID="E1EXAMPLE12345"

npm run build

aws s3 sync ./dist/ "$BUCKET" \
  --delete \
  --cache-control "public, max-age=31536000, immutable" \
  --exclude "*.html" \
  --exclude "*.map"

# Sync HTML files separately with a short cache header
aws s3 sync ./dist/ "$BUCKET" \
  --exclude "*" --include "*.html" \
  --cache-control "public, max-age=60"

aws cloudfront create-invalidation \
  --distribution-id "$DISTRIBUTION_ID" \
  --paths "/*" > /dev/null

4. Risks of the --delete flag and how to contain them

The --delete flag removes every file in the destination that does not exist in the source, and that is exactly what makes it dangerous once source and destination are not what the script assumes them to be. An empty or wrongly computed source path, for example because a previous build step failed and left the folder empty, combined with --delete, causes sync to wipe the entire destination path without any warning.

A simple safeguard is checking, before every sync with --delete, whether the source directory actually contains files, and aborting the run with a clear error message otherwise instead of silently continuing. S3 bucket versioning additionally protects against deleted objects vanishing irretrievably, because with versioning enabled, sync --delete only sets a delete marker and older versions remain intact.

#!/usr/bin/env bash
set -euo pipefail

readonly SRC="./dist"
readonly DEST="s3://mironsoft-static-assets/frontend/"

# Safeguard: never sync with --delete against an empty source directory
file_count=$(find "$SRC" -type f | wc -l)
if [[ "$file_count" -eq 0 ]]; then
  echo "ERROR: $SRC is empty, aborting sync with --delete" >&2
  exit 1
fi

aws s3 sync "$SRC" "$DEST" --delete

5. Dry run before the real sync: using --dryrun

The --dryrun flag runs the exact same comparison logic as a real sync but only prints which files would be uploaded, downloaded or deleted, without actually changing anything. Before every deployment sync with --delete, a dry run whose output gets a quick glance is worth the effort, to make sure the number of files about to be deleted is in a plausible range.

In automated pipelines the dry run can also be evaluated programmatically: count the number of lines starting with delete: and compare it against a threshold. If the number of planned deletions exceeds a sensible percentage of total files, the script aborts and requires manual confirmation instead of running the destructive sync fully automatically.

#!/usr/bin/env bash
set -euo pipefail

readonly SRC="./dist"
readonly DEST="s3://mironsoft-static-assets/frontend/"

dryrun_output=$(aws s3 sync "$SRC" "$DEST" --delete --dryrun)
delete_count=$(grep -c "^delete:" <<< "$dryrun_output" || true)

echo "$dryrun_output"
echo "Planned deletions: $delete_count"

if [[ "$delete_count" -gt 50 ]]; then
  echo "ERROR: too many planned deletions, aborting" >&2
  exit 1
fi

aws s3 sync "$SRC" "$DEST" --delete

6. Backup workflow: sync with exclude patterns and no --delete

For backup purposes the --delete flag is usually out of place, since the whole point of a backup is to keep long-deleted or overwritten files still available. A plain sync without --delete adds new and changed files but never removes anything from the backup bucket, which is exactly the desired behavior for most backup requirements.

Exclude patterns with --exclude and --include filter out unnecessary files like temporary build artifacts, node_modules or log files to save storage and transfer time. The patterns are evaluated in the order they appear on the command line, with a later --include re-enabling matching files that a previous --exclude had filtered out.

#!/usr/bin/env bash
set -euo pipefail

aws s3 sync /var/www/mironsoft/ s3://mironsoft-backups/www/ \
  --exclude "node_modules/*" \
  --exclude "*.log" \
  --exclude "var/cache/*" \
  --exclude "var/tmp/*"

7. Handling errors from network dropouts and interrupted syncs

A sync across thousands of files can be interrupted midway by an unstable network connection. The AWS CLI internally retries individual failed file transfers by default, but does not guarantee the overall run completes successfully in the end. The exit code of aws s3 sync reliably indicates whether the run finished cleanly overall, and should be checked in every production script instead of silently assuming success.

Because sync only transfers the still-missing or changed files on a repeated run anyway, a simple retry with a limited number of attempts is usually enough to absorb temporary network issues without building complicated resume logic from scratch. A short pause between attempts prevents an ongoing network problem from immediately reproducing the same failure.

#!/usr/bin/env bash
set -uo pipefail

readonly SRC="./dist"
readonly DEST="s3://mironsoft-static-assets/frontend/"
readonly MAX_RETRIES=3

attempt=1
while (( attempt <= MAX_RETRIES )); do
  echo "Sync attempt ${attempt}/${MAX_RETRIES} ..."
  if aws s3 sync "$SRC" "$DEST" --delete; then
    echo "Sync succeeded."
    exit 0
  fi
  echo "Sync failed, waiting 10s before retrying." >&2
  sleep 10
  (( attempt++ ))
done

echo "ERROR: sync still failed after ${MAX_RETRIES} attempts." >&2
exit 1

8. Logging and notifications on sync failures

A deployment or backup script running unattended overnight via cron needs a way to surface failures without anyone manually combing through log files every morning. The output of aws s3 sync can be written to a timestamped log file alongside the exit code, while a failure additionally triggers a Slack webhook or an email notification.

For regularly scheduled sync jobs, a simple heartbeat mechanism also pays off, for instance calling an external monitoring service after every successful run, so a cron job that stops running entirely gets noticed, not just one that fails outright. A sync script that has not run for days causes the same data loss as a failed sync, but without active monitoring it often stays unnoticed considerably longer.

9. Automating S3 sync in CI/CD pipelines

In a CI pipeline, the sync typically runs as the last step after a successful build and test, using credentials provided through short-lived IAM roles scoped to the deployment bucket, instead of long-lived, broadly permissioned access keys. That limits the damage if the pipeline environment itself gets compromised, since the credentials used expire quickly anyway and only apply to that one bucket.

The dry-run threshold from the previous section can be set even stricter in a CI pipeline than locally, for instance by having a sync with more than a handful of planned deletions automatically pause and require manual approval in the pipeline tool instead of running fully automatically. For most projects, that one extra check step is the cheapest insurance against an accidentally emptied production bucket.

10. Summary

11. FAQ: S3 Sync Patterns with awscli: The Essentials at a Glance