Combining JSON, YAML and .env Files in Shell Workflows

1. Why configuration formats converge in shell workflows

In modern infrastructure, JSON, YAML and .env files rarely exist in isolation. A typical deployment pipeline reads service configuration from a config.yaml, loads environment variables from an .env file and parses API responses as JSON, all within the same shell script. This convergence is not a coincidence, it is a consequence of the diversity of modern toolchains: Terraform speaks HCL and JSON, Kubernetes speaks YAML, Docker Compose speaks YAML, most APIs speak JSON, and legacy deployments rely on .env files. Anyone who wants to merge these formats in a shell workflow needs reliable tools and clear patterns.

The fundamental problem is not the syntax but the safety of loading: a missing required-field check in a JSON object, an unsafely loaded .env file with special characters in values, a YAML alias that unexpectedly overwrites values, all of these are sources of errors that lead to inconsistent behavior in production environments. The strategies in this article cover the most important patterns: safe loading, validation, combining sources with priority logic, and secure handling of secrets across all three formats.

2. Processing JSON in Bash: jq as a shell tool

jq is the de facto standard for processing JSON in shell scripts. It combines a stream processor with a full filter language and outputs transformed JSON structures or extracted values. The core pattern for use in shell workflows is extracting values into variables: VAR=$(jq -r '.key' config.json) outputs the raw string value without JSON quotes. The -r flag (raw output) is essential here: without it, the returned value contains JSON quotes, which causes errors during further processing.

Particularly valuable for shell workflow use is the ability of jq to extract multiple values in a single call and generate shell variable assignments directly from them. The pattern eval "$(jq -r 'to_entries[] | "export \(.key)=\(.value | @sh)"' config.json)" exports every field of a JSON object as a shell variable, with @sh escaping the values correctly for shell quoting. This pattern avoids code injection through special characters in values and is therefore considerably safer than naive string concatenation.

#!/usr/bin/env bash
# json-loader.sh: Safe JSON parsing and variable export in shell workflows
set -euo pipefail

CONFIG_FILE="${1:-config.json}"

# Guard: check if jq is available
command -v jq >/dev/null 2>&1 || { echo "[ERROR] jq is not installed" >&2; exit 1; }

# Guard: validate JSON syntax before processing
jq empty "$CONFIG_FILE" 2>/dev/null || { echo "[ERROR] Invalid JSON: $CONFIG_FILE" >&2; exit 1; }

# Extract single value (raw string, no JSON quotes)
DB_HOST=$(jq -r '.database.host // empty' "$CONFIG_FILE")
DB_PORT=$(jq -r '.database.port // 3306' "$CONFIG_FILE")

# Guard: required field must not be empty
[[ -n "$DB_HOST" ]] || { echo "[ERROR] database.host is required in $CONFIG_FILE" >&2; exit 1; }

# Export entire object as shell variables safely (values are @sh-escaped)
eval "$(jq -r '
  .environment // {} |
  to_entries[] |
  "export \(.key)=\(.value | @sh)"
' "$CONFIG_FILE")"

# Iterate over JSON array
declare -a services=()
while IFS= read -r svc; do
  services+=("$svc")
done < <(jq -r '.services[]?.name // empty' "$CONFIG_FILE")

echo "Loaded ${#services[@]} services from $CONFIG_FILE"
echo "DB: ${DB_HOST}:${DB_PORT}"

3. Reading, transforming and exporting YAML with yq

yq is the YAML equivalent of jq and, in its modern version (Mike Farah's Go implementation, v4+), supports a largely compatible syntax. Important for shell workflow use: there are two competing yq implementations, the Python-based one (pip install yq, which uses jq internally) and the Go-based one (snap install yq or a binary download). Both have different syntax, which leads to surprises in CI pipelines. The first pattern for any YAML shell workflow should therefore be a version check at the start.

Compared to JSON, YAML has the advantage of human-readable configuration, but it brings its own pitfalls: implicit types (the string yes is interpreted as a boolean by some parsers), multiline strings and anchors with aliases can unexpectedly merge values. When exporting YAML values into shell variables, the same principle applies as with JSON: values must be escaped correctly for the shell. The yq -r flag produces raw output, and for bulk export the same eval approach with explicit shell escaping of values is recommended.

#!/usr/bin/env bash
# yaml-loader.sh: YAML parsing with yq in shell workflows
set -euo pipefail

YAML_FILE="${1:-config.yaml}"

# Check yq version: Go-based v4 vs Python-based
YQ_VERSION=$(yq --version 2>&1 | grep -oP 'version v?\K[0-9]+' | head -1)
if [[ "${YQ_VERSION:-0}" -lt 4 ]]; then
  echo "[WARN] yq v4+ (Go) recommended for this script" >&2
fi

# Validate YAML before processing
yq eval 'true' "$YAML_FILE" >/dev/null 2>&1 || {
  echo "[ERROR] Invalid YAML: $YAML_FILE" >&2; exit 1
}

# Read scalar values
APP_NAME=$(yq eval '.app.name // ""' "$YAML_FILE")
APP_ENV=$(yq eval '.app.environment // "production"' "$YAML_FILE")

# Export all keys from a YAML map as shell variables (Go yq v4 syntax)
eval "$(yq eval '.config | to_entries | .[] | "export " + .key + "=" + (.value | @sh)' "$YAML_FILE" 2>/dev/null || true)"

# Iterate over YAML sequence
declare -a hosts=()
while IFS= read -r host; do
  [[ -n "$host" ]] && hosts+=("$host")
done < <(yq eval '.servers[].host' "$YAML_FILE" 2>/dev/null)

# Convert YAML to JSON for jq post-processing
yq eval -o=json "$YAML_FILE" | jq -r '.deploy.steps[]?.name // empty'

echo "App: $APP_NAME ($APP_ENV), ${#hosts[@]} hosts"

4. Loading and exporting .env files safely

.env files are the oldest of the three formats and also the most error-prone to load in shell scripts. The naive pattern source .env runs the file as shell code, which works as long as all values are simple strings without special characters, but a value like PASSWORD=pa$$w0rd!&echo hacked immediately leads to code execution. The safe .env loading pattern reads the file line by line, filters comments and empty lines, and uses declare or safe assignment patterns instead of source.

Another common problem: .env values may or may not contain quotes, DB_PASS="my secret" and DB_PASS=my secret are both valid notations in certain dotenv dialects, but the shell treats them differently. The robust pattern strips surrounding single and double quotes from the value that was read before it is exported. In addition, variable names must be checked against code injection, only alphanumeric characters and underscores are permitted for variable names.

#!/usr/bin/env bash
# dotenv-loader.sh: Safe .env file loading without source
set -euo pipefail

load_dotenv() {
  local env_file="$1"
  local export_vars="${2:-false}"

  [[ -f "$env_file" ]] || { echo "[ERROR] .env file not found: $env_file" >&2; return 1; }

  local line key raw_val val
  while IFS= read -r line || [[ -n "$line" ]]; do
    # Skip comments and empty lines
    [[ "$line" =~ ^[[:space:]]*# ]] && continue
    [[ "$line" =~ ^[[:space:]]*$ ]] && continue

    # Split at first = only
    key="${line%%=*}"
    raw_val="${line#*=}"

    # Validate key: only [A-Za-z_][A-Za-z0-9_]* allowed
    [[ "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || {
      echo "[WARN] Skipping invalid key: $key" >&2; continue
    }

    # Strip surrounding quotes (single or double)
    val="${raw_val}"
    if [[ "$val" =~ ^\"(.*)\"$ ]]; then
      val="${BASH_REMATCH[1]}"
    elif [[ "$val" =~ ^\'(.*)\'$ ]]; then
      val="${BASH_REMATCH[1]}"
    fi

    if [[ "$export_vars" == "true" ]]; then
      export "$key"="$val"
    else
      declare -g "$key"="$val"
    fi
  done < "$env_file"
}

# Load and export .env file
load_dotenv ".env" true

# Override with environment-specific .env (higher priority)
[[ -f ".env.${APP_ENV:-production}" ]] && load_dotenv ".env.${APP_ENV:-production}" true

echo "DB_HOST=${DB_HOST:-not set}"

5. Combining three formats: priority logic and override rules

When JSON, YAML and .env files come together in the same shell workflow, a clear priority logic is needed: which format overrides which? The common convention in modern deployment systems is: shell environment variables have the highest priority, followed by .env files, then YAML configuration, and finally JSON defaults. This ordering reflects the principle that more specific sources override more general ones, a CI variable should always win over file-based configuration.

The implementation pattern for this priority logic uses Bash parameter expansion: variables are set with the ${VAR:-} pattern, if a variable is already set in the environment it remains unchanged. The trick lies in loading the sources in the right order: first JSON defaults into temporary variables, then YAML overriding them, then .env overriding that, and finally real environment variables, which are never overwritten. This pattern makes the configuration source transparent when debugging: a CONFIG_SOURCE=debug ./deploy.sh can print every loaded value along with its origin.

#!/usr/bin/env bash
# config-merger.sh: Combine JSON defaults, YAML config and .env with priority
set -euo pipefail

# Priority order: env vars > .env > YAML > JSON defaults
# Load in REVERSE order: lowest priority first

# 1. JSON defaults (lowest priority)
if [[ -f "config.defaults.json" ]]; then
  while IFS='=' read -r k v; do
    # Only set if not already in environment
    [[ -z "${!k+x}" ]] && declare -g "$k"="$v"
  done < <(jq -r 'to_entries[] | "\(.key)=\(.value | @sh | gsub("^'"'"'|'"'"'$";""))"' config.defaults.json)
fi

# 2. YAML config (overrides JSON defaults)
if [[ -f "config.yaml" ]] && command -v yq >/dev/null 2>&1; then
  while IFS='=' read -r k v; do
    [[ -z "${!k+x}" ]] && declare -g "$k"="$v"
  done < <(yq eval '.config | to_entries[] | .key + "=" + (.value | tostring)' config.yaml 2>/dev/null)
fi

# 3. .env file (overrides YAML)
load_dotenv_safe() {
  local file="$1"
  while IFS= read -r line || [[ -n "$line" ]]; do
    [[ "$line" =~ ^[[:space:]]*(#|$) ]] && continue
    local k="${line%%=*}" v="${line#*=}"
    [[ "$k" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || continue
    # .env wins over YAML/JSON but NOT over real env vars
    [[ -n "${!k+x}" ]] || export "$k"="${v//\"/}"
  done < "$file"
}
[[ -f ".env" ]] && load_dotenv_safe ".env"

# 4. Real environment variables already set: highest priority, nothing to do

# Debug output
if [[ "${CONFIG_DEBUG:-0}" == "1" ]]; then
  echo "=== Effective configuration ==="
  echo "APP_ENV=${APP_ENV:-production}"
  echo "DB_HOST=${DB_HOST:-localhost}"
  echo "DB_PORT=${DB_PORT:-3306}"
fi

6. Validation: checking required fields, types and value ranges

Loading configuration without subsequent validation is one of the most common patterns that leads to hard-to-debug errors in production systems. An empty, silently loaded DB_HOST does not fail immediately, only once the database command runs does a cryptic error message appear that has nothing obvious to do with the real cause. The validation pattern for shell workflows checks all loaded configuration values right after loading, before the script executes any further steps. This way configuration errors surface early with clear messages.

For JSON-based validation, jq with JSON-Schema-like filters offers an elegant option. The pattern checks whether required fields are present, whether ports fall within a valid range, whether URLs use the correct scheme, and whether enum values come from a defined set. The same principle applies to YAML: yq can verify after loading that structures are present as expected. The combination of early validation, clear error messages and exiting with code 1 on failure makes shell workflows noticeably more robust against configuration errors.

7. Handling secrets from JSON and YAML safely

Secrets in JSON or YAML files are a common anti-pattern, but in practice not always fully avoidable. When passwords, API keys or certificates must be loaded from these files, there are clear security rules for the shell workflow: never export secrets into environment variables that stay visible longer than necessary, never write them to log files, and never pass them as command-line arguments. The safe pattern is: read the secret value, use it immediately, then delete the variable (unset SECRET_VAR).

Another critical rule: never load secrets from JSON or YAML while set -x is active, set -x prints every command including all variable values, which exposes secrets in log files or CI output. The pattern { set +x; load_secrets; set -x; } 2>/dev/null temporarily disables tracing for the secret-loading block. In addition, .env files containing secrets should never be loaded via source, always use the line-by-line reading pattern, which never executes values as shell code.

8. Integration into CI/CD pipelines

In CI/CD environments, the full spectrum of configuration formats comes together: pipeline variables from the CI system (effectively environment variables), .env files from the repository, YAML configuration from Helm charts or Kubernetes manifests, and JSON responses from Vault or cloud metadata APIs. The shell workflow in a pipeline must be able to handle all of these sources without execution order or missing tools causing silent failures.

The most important pattern for CI pipelines: check all tools (jq, yq) at the start of the script and abort with a clear error message if one is missing. Many CI images include jq but not yq. A fallback pattern converts YAML to JSON using Python or Ruby if yq is unavailable: python3 -c "import sys,yaml,json; json.dump(yaml.safe_load(sys.stdin), sys.stdout)" < config.yaml. This conversion pattern is available in most Linux environments without any additional installation and makes the shell workflow more portable.

9. Configuration formats compared side by side

Choosing the right configuration format for a shell workflow depends on several factors: readability, tool availability, type system and safety when loading. The following table compares the three formats against these criteria and gives recommendations for typical use cases.

For complex configuration structures with nesting and comments, YAML is the first choice. For machine-generated output (API responses, Terraform output, state files) JSON is better suited because it is more precise in its type system and has no parser ambiguities. .env files are best suited exclusively for flat key-value configuration without nesting and without secrets in repositories. In shell workflows that process all three formats, it is worth building a shared loading library that abstracts all formats behind a single unified interface.

10. Summary

Safely combining JSON, YAML and .env files in shell workflows requires three core competencies: safe tool-assisted processing with jq and yq, a clear priority system for overriding configuration sources, and consistent validation right after loading. jq -r with @sh escaping is the safe pattern for turning JSON into shell variables. Reading .env files line by line with key validation prevents code injection through manipulated values. The priority order environment variables, then .env, then YAML, then JSON defaults reflects the principle that more specific configuration overrides more general configuration.

For production use in shell workflows, the rule is: check tools at the start, test JSON/YAML for syntactic validity before processing, verify required fields immediately after loading, and never keep secrets in variables longer than necessary. These patterns turn deployment scripts that combine JSON, YAML and .env into a reliable, transparent link in the infrastructure chain, instead of an invisible source of failure right before a production deployment.

11. FAQ: JSON, YAML and .env in Shell Workflows