AWS and gcloud CLI Scripting Patterns in Bash

1. Why multi cloud scripts need their own patterns

The AWS CLI and gcloud CLI solve similar tasks, but differ so much in authentication, output formats, and pagination that naive scripts for both clouds quickly turn into duplicated, hard to maintain code. A team managing both S3 and Cloud Storage types two completely different command sequences for the same logical task if it does not establish shared AWS and gcloud CLI patterns.

The value of consistent AWS and gcloud CLI scripting patterns shows up especially in mixed infrastructures, where a deployment script has to talk to both clouds, for example because disaster recovery copies live in a second cloud or because different teams chose different providers. A Bash script that offers a unified function like cloud_list_buckets and internally switches between aws s3api list-buckets and gcloud storage buckets list reduces the cognitive load for anyone maintaining the script later.

The following sections build practical AWS and gcloud CLI patterns: from unified authentication through consistent pagination to retry logic that handles the different rate limit behavior of both clouds.

2. Cleanly abstracting authentication for AWS and gcloud

AWS uses profiles and temporary credentials via aws sts assume-role, gcloud works with service account keys and gcloud auth activate-service-account or workload identity federation. A script that needs to address both clouds requires an abstraction that picks the right authentication method depending on the target environment, without the calling code needing to know these details.

What matters for AWS and gcloud CLI authentication is that temporary credentials have an explicit expiration, and a long running script must refresh credentials in time, before an API call fails with an authentication error.

#!/usr/bin/env bash
# cloud-auth.sh — unified authentication for AWS and gcloud
set -euo pipefail

readonly PROVIDER="${1:?Usage: cloud-auth.sh <aws|gcp> <role-or-account>}"
readonly TARGET="${2:?Missing role ARN or service account}"

cloud_authenticate() {
  case "$PROVIDER" in
    aws)
      echo "[INFO] Assuming AWS role: ${TARGET}"
      local creds
      creds="$(aws sts assume-role --role-arn "$TARGET" \
        --role-session-name "bash-automation-$(date +%s)" \
        --duration-seconds 3600 --output json)"
      export AWS_ACCESS_KEY_ID
      export AWS_SECRET_ACCESS_KEY
      export AWS_SESSION_TOKEN
      AWS_ACCESS_KEY_ID="$(jq -r '.Credentials.AccessKeyId' <<< "$creds")"
      AWS_SECRET_ACCESS_KEY="$(jq -r '.Credentials.SecretAccessKey' <<< "$creds")"
      AWS_SESSION_TOKEN="$(jq -r '.Credentials.SessionToken' <<< "$creds")"
      ;;
    gcp)
      echo "[INFO] Activating GCP service account: ${TARGET}"
      gcloud auth activate-service-account --key-file="$TARGET"
      ;;
    *)
      echo "[ERROR] Unknown provider: ${PROVIDER} (expected aws or gcp)" >&2
      exit 1
      ;;
  esac
  echo "[OK] Authenticated against ${PROVIDER}"
}

cloud_authenticate

3. Unified wrapper functions for both CLIs

The central building block of AWS and gcloud CLI scripting patterns is a thin abstraction layer that offers logically equivalent operations under a shared function name. A script calls cloud_list_storage_buckets, the function internally decides based on an environment variable or parameter whether aws s3api list-buckets or gcloud storage buckets list actually runs, and normalizes the output of both CLIs into a common JSON format.

This normalization is the actual added value: AWS returns bucket names under .Buckets[].Name, gcloud under a different JSON path. An AWS and gcloud CLI wrapper translates both structures into a unified schema, so the rest of the application code does not need to know which cloud is currently being addressed.

#!/usr/bin/env bash
# cloud-wrapper.sh — unified interface over aws and gcloud CLI
set -euo pipefail

# cloud_list_storage_buckets <aws|gcp> — normalized bucket listing
cloud_list_storage_buckets() {
  local provider="${1:?Usage: cloud_list_storage_buckets <aws|gcp>}"
  case "$provider" in
    aws)
      aws s3api list-buckets --output json \
        | jq '[.Buckets[] | {name: .Name, created: .CreationDate, provider: "aws"}]'
      ;;
    gcp)
      gcloud storage buckets list --format=json \
        | jq '[.[] | {name: .name, created: .timeCreated, provider: "gcp"}]'
      ;;
    *)
      echo "[ERROR] Unknown provider: ${provider}" >&2
      return 1
      ;;
  esac
}

# cloud_create_instance <aws|gcp> <name> <size> — normalized instance creation
cloud_create_instance() {
  local provider="$1" name="$2" size="$3"
  case "$provider" in
    aws)
      aws ec2 run-instances --image-id ami-0abcdef1234567890 \
        --instance-type "$size" --tag-specifications \
        "ResourceType=instance,Tags=[{Key=Name,Value=${name}}]" \
        --output json | jq '.Instances[0] | {id: .InstanceId, provider: "aws"}'
      ;;
    gcp)
      gcloud compute instances create "$name" --machine-type="$size" \
        --format=json | jq '.[0] | {id: .id, provider: "gcp"}'
      ;;
  esac
}

4. Handling pagination and JSON consistently with jq

AWS APIs return a NextToken for large result sets, gcloud instead uses --page-token and returns the next token in its own field. Anyone writing AWS and gcloud CLI scripts without accounting for pagination silently loses data on larger accounts, because both CLIs return only a limited page size by default.

A robust pagination function collects all pages in a loop until no further token comes back, and returns a complete JSON array at the end, regardless of how many API calls were needed internally.

#!/usr/bin/env bash
# cloud-paginate.sh — consistent pagination handling for both CLIs
set -euo pipefail

# aws_paginate_all <command...> — follow NextToken until exhausted
aws_paginate_all() {
  local next_token="" results="[]"
  while true; do
    local page
    if [[ -n "$next_token" ]]; then
      page="$("$@" --starting-token "$next_token" --output json)"
    else
      page="$("$@" --output json)"
    fi
    results="$(jq -s '.[0] + .[1]' <(echo "$results") <(jq '.Reservations // .Buckets // []' <<< "$page"))"
    next_token="$(jq -r '.NextToken // empty' <<< "$page")"
    [[ -z "$next_token" ]] && break
  done
  echo "$results"
}

# gcp_paginate_all <command...> — follow --page-token until exhausted
gcp_paginate_all() {
  local page_token="" results="[]"
  while true; do
    local page
    if [[ -n "$page_token" ]]; then
      page="$("$@" --page-token="$page_token" --format=json)"
    else
      page="$("$@" --format=json)"
    fi
    results="$(jq -s '.[0] + .[1]' <(echo "$results") <(echo "$page"))"
    page_token="$(jq -r '.nextPageToken // empty' <<< "$page" 2>/dev/null || echo "")"
    [[ -z "$page_token" ]] && break
  done
  echo "$results"
}

5. Resource tagging and multi cloud inventory

A common use case for AWS and gcloud CLI scripting patterns is an inventory report over all running resources across both clouds, for example for cost control or security audits. AWS uses tags in key value pair format, gcloud uses labels with similar but not identical syntax and stricter character restrictions, for example lowercase letters and hyphens only.

An inventory script that queries both clouds should normalize tags and labels into a common format and flag missing or inconsistent tags as a separate category in the report, so teams can fix gaps proactively instead of noticing them only at the next bill.

#!/usr/bin/env bash
# multi-cloud-inventory.sh — combined resource inventory with tag/label check
set -euo pipefail

readonly REQUIRED_TAGS=("owner" "environment" "cost-center")

check_aws_tags() {
  aws resourcegroupstaggingapi get-resources --output json \
    | jq --argjson required "$(printf '%s\n' "${REQUIRED_TAGS[@]}" | jq -R . | jq -s .)" '
      .ResourceTagMappingList[] | {
        arn: .ResourceARN,
        tags: (.Tags | map(.Key) ),
        missing: ($required - (.Tags | map(.Key | ascii_downcase)))
      } | select(.missing | length > 0)
    '
}

check_gcp_labels() {
  gcloud asset search-all-resources --format=json \
    | jq --argjson required "$(printf '%s\n' "${REQUIRED_TAGS[@]}" | jq -R . | jq -s .)" '
      .[] | {
        name: .name,
        labels: (.labels // {} | keys),
        missing: ($required - (.labels // {} | keys))
      } | select(.missing | length > 0)
    '
}

echo "--- AWS resources missing required tags ---"
check_aws_tags
echo "--- GCP resources missing required labels ---"
check_gcp_labels

6. Rate limits, retries and backoff for cloud APIs

Both cloud APIs throttle on too many requests, but with different error codes and wait behaviors. AWS usually responds with Throttling or RequestLimitExceeded, gcloud with HTTP 429 responses and a RESOURCE_EXHAUSTED status. A universal retry wrapper for AWS and gcloud CLI calls recognizes both patterns and waits with exponential backoff instead of aborting immediately on the first rate limit error.

Without this retry logic, bulk operations, for example tagging hundreds of resources in a loop, regularly abort in the middle of the run once API throttling kicks in, leaving inconsistent intermediate states.

#!/usr/bin/env bash
# cloud-retry.sh — exponential backoff for AWS and gcloud rate limits
set -uo pipefail

cloud_retry() {
  local max_attempts=5 attempt=1 delay=2
  local output exit_code

  while (( attempt <= max_attempts )); do
    output="$("$@" 2>&1)"
    exit_code=$?

    if [[ $exit_code -eq 0 ]]; then
      echo "$output"
      return 0
    fi

    if grep -qiE "Throttling|RequestLimitExceeded|RESOURCE_EXHAUSTED|429" <<< "$output"; then
      echo "[WARN] Rate limited (attempt ${attempt}/${max_attempts}), waiting ${delay}s" >&2
      sleep "$delay"
      delay=$(( delay * 2 ))
      ((attempt++))
      continue
    fi

    echo "$output" >&2
    return "$exit_code"
  done

  echo "[ERROR] Command still rate-limited after ${max_attempts} attempts" >&2
  return 1
}

# Usage: cloud_retry aws s3api list-buckets
# Usage: cloud_retry gcloud storage buckets list

7. Automating cost and resource reports

AWS Cost Explorer and the GCP Billing API both provide programmatic access to cost data, but with different grouping options and time ranges. An AWS and gcloud CLI script for regular cost reports queries both APIs with comparable parameters, for example cost of the last 30 days grouped by service, and combines the results into a shared report.

These reports typically run as a scheduled job that sends combined numbers to Slack or an internal dashboard application. The value lies in seeing cost development across both clouds at a glance instead of manually checking two separate consoles.

#!/usr/bin/env bash
# cost-report.sh — combined 30-day cost summary across AWS and GCP
set -euo pipefail

readonly START=$(date -u -d '30 days ago' +%Y-%m-%d)
readonly END=$(date -u +%Y-%m-%d)

aws_cost=$(aws ce get-cost-and-usage \
  --time-period "Start=${START},End=${END}" \
  --granularity MONTHLY --metrics "UnblendedCost" \
  --output json | jq -r '.ResultsByTime[0].Total.UnblendedCost.Amount')

gcp_cost=$(gcloud billing accounts list --format="value(name)" | head -1 | \
  xargs -I{} gcloud alpha billing accounts get-spend-summary {} \
  --format="value(cost)" 2>/dev/null || echo "n/a")

echo "=== Cost Report: ${START} to ${END} ==="
printf "AWS:  \$%s\n" "$aws_cost"
printf "GCP:  %s\n" "$gcp_cost"

8. Security pitfalls: profiles, projects and cross account risk

The biggest security pitfall with AWS and gcloud CLI scripts is accidentally running a command against the wrong account or project. AWS profiles are selected via --profile or the AWS_PROFILE environment variable, gcloud via gcloud config set project or --project. If one of these settings is forgotten or carried over from a previous shell session, a destructive command lands in the wrong context.

A defensive script explicitly checks before critical operations which account or project is currently active, for example with aws sts get-caller-identity or gcloud config get-value project, and compares the result against the expected value before the actual operation runs. This pattern is analogous to the production guards that are also useful for kubectl wrappers, just at the cloud account level instead of the Kubernetes namespace level.

9. AWS CLI and gcloud CLI compared side by side

The following table shows the key differences that AWS and gcloud CLI scripting patterns need to account for.

The comparison shows: both CLIs are well documented and functional on their own, but only well thought out AWS and gcloud CLI scripting patterns make a multi cloud setup maintainable instead of running two parallel codebases without shared structure.

10. Summary

Consistent AWS and gcloud CLI scripting patterns prevent a multi cloud setup from becoming two parallel, disconnected codebases. A unified authentication function abstracts the different login mechanisms, shared wrapper functions normalize output formats into one schema, separate pagination logic per CLI collects complete result sets, and a universal retry wrapper recognizes rate limit errors from both clouds by their respective error patterns.

Inventory and cost reports benefit especially from this unification, because they merge both clouds into a single report instead of manually comparing two separate dashboards. Once these AWS and gcloud CLI patterns are cleanly implemented, adding new cloud operations becomes noticeably faster, because the basic structure for authentication, error handling, and pagination is already in place.

11. FAQ: AWS and gcloud CLI Scripting Patterns in Bash