jq for JSON in Everyday Shell Work: Filter, Transform, Validate

1. Why jq solves the JSON problem in the shell

jq is a command line JSON processor that closes the gap in Bash scripts between raw API responses and usable data. Before jq, you either parsed JSON in the shell with grep and sed, error prone, not Unicode safe, and unworkable for nested structures, or you called Python with an import json one liner, which requires a separate process and might not be available in minimal environments. jq is a single binary that is available through the package manager on most Linux distributions and brings along a complete JSON processing language.

The strength of jq lies in its functional filter pipeline: each filter transforms the input JSON and passes the result to the next filter. .|.items[].name is a pipeline that first accesses the input object, then iterates the items array, and extracts the name value from each element. This notation makes complex JSON traversal readable and easy to follow. Compared with grep based approaches, the advantage is immediately obvious: jq understands the JSON structure, while grep only ever works at the text level.

A practical advantage of jq in CI/CD pipelines is that it is streaming capable. Very large JSON files or continuous streams can be processed with jq --stream without loading the entire file into memory. For typical API responses in the KB range this is not necessary, but for log files in the GB range it is a decisive factor. jq is not a tool for occasional use, once you have learned it thoroughly, you end up reaching for it daily in shell scripts.

2. Basic filters: extracting and navigating fields

The simplest jq filter is .fieldname: it extracts the value of the given field from the input object. Nested fields are navigated with dot notation: .metadata.labels.environment. Array elements are accessed with .items[0] (first element), .items[-1] (last element), or .items[2:5] (slice). The ? operator after a field name (.optional_field?) returns null instead of an error when the field does not exist, an important distinction for defensive jq programming.

The identity filter . returns the input JSON unchanged and is often used as the starting point of a pipeline. jq '.' formats JSON in a readable way (pretty print). jq -c '.' outputs compact JSON without whitespace, ideal for further processing into Bash variables, since no whitespace escaping is required. jq -r (raw output) strips quotation marks from string output, which is essential when the value is used in a Bash variable: name="$(jq -r '.name' <<< "$json")", without -r this would assign "myname" including the quotes.

#!/usr/bin/env bash
# jq_basics.sh: core jq filter patterns for shell automation
set -euo pipefail

# Sample JSON (typical API response)
API_RESPONSE='{
  "id": 42,
  "name": "deploy-prod",
  "status": "running",
  "metadata": { "env": "production", "version": "2.3.1" },
  "tags": ["release", "hotfix"],
  "artifacts": [
    {"name": "app.tar.gz", "size": 1024000, "checksum": "abc123"},
    {"name": "db-migration.sql", "size": 4096, "checksum": "def456"}
  ]
}'

# Basic field extraction
name="$(jq -r '.name' <<< "$API_RESPONSE")"
env="$(jq -r '.metadata.env' <<< "$API_RESPONSE")"
first_tag="$(jq -r '.tags[0]' <<< "$API_RESPONSE")"
last_artifact="$(jq -r '.artifacts[-1].name' <<< "$API_RESPONSE")"

echo "Name: $name | Env: $env | First tag: $first_tag | Last artifact: $last_artifact"

# Multiple fields in one jq call, more efficient than separate calls
read -r id status version < <(jq -r '[.id, .status, .metadata.version] | @tsv' <<< "$API_RESPONSE")
echo "ID: $id | Status: $status | Version: $version"

# Optional field access, no error if field missing
optional="$(jq -r '.missing_field? // "default_value"' <<< "$API_RESPONSE")"
echo "Optional: $optional"

# Array length
artifact_count="$(jq '.artifacts | length' <<< "$API_RESPONSE")"
echo "Artifacts: $artifact_count"

# Extract all checksums as newline-separated values
jq -r '.artifacts[].checksum' <<< "$API_RESPONSE"

3. Iteration with .[] and array operations

The jq iterator .[] applied to an array outputs each element individually, as a separate JSON document, not as an array. That is the key difference for Bash integration: in a Bash loop you can process the output of jq '.items[]' line by line if the elements are single line values. For multi line or nested objects, jq -c '.items[]' (compact JSON, one object per line) combined with while IFS= read -r item is the recommended approach.

The length of an array is returned by length: jq '.items | length'. Empty arrays return 0, a defined behavior you can use directly in Bash checks. To test whether an array has at least one element: jq -e '.items | length > 0', with -e (exit status) returning exit code 1 on false. first and last are shorthand for .[0] and .[-1]. nth(n; expr) returns the nth element of a stream expression. jq ships a complete library for array operations that requires no external tool.

4. Conditional filtering with select()

select() is the central filter operator in jq for conditional selection. select(cond) returns the element unchanged if cond is true, and returns nothing if it is false. Combined with .[], it filters an array by condition: .items[] | select(.status == "active") outputs all active items. Multiple conditions combine with and and or: select(.status == "active" and .env == "production").

For string matching, jq offers test(regex): select(.name | test("^feat-")) filters all elements whose name starts with "feat-". startswith() and endswith() are alias functions for common string tests. contains(val) checks whether a string contains a substring or an array contains an element, in jq contains is defined generically for all JSON types. These expressions can be combined and nested arbitrarily to express complex filter logic concisely.

#!/usr/bin/env bash
# jq_select_map.sh: filtering and transformation with jq
set -euo pipefail

DEPLOYMENTS='[
  {"id": 1, "name": "api-v2", "env": "production", "status": "success", "size": 2048},
  {"id": 2, "name": "frontend", "env": "staging",    "status": "failed",  "size": 1024},
  {"id": 3, "name": "worker",   "env": "production", "status": "success", "size": 512},
  {"id": 4, "name": "cron-job", "env": "staging",    "status": "success", "size": 256}
]'

# Filter: production deployments only
echo "=== Production deployments ==="
jq -r '.[] | select(.env == "production") | "\(.id): \(.name) [\(.status)]"' <<< "$DEPLOYMENTS"

# Filter: failures across all environments
echo "=== Failed deployments ==="
jq -r '[.[] | select(.status == "failed")] | length' <<< "$DEPLOYMENTS"

# Combined filter: successful and larger than 500KB
echo "=== Large successful deployments ==="
jq -r '.[] | select(.status == "success" and .size > 500) | .name' <<< "$DEPLOYMENTS"

# map(): transform array elements
echo "=== Name + status pairs ==="
jq -c 'map({name: .name, ok: (.status == "success")})' <<< "$DEPLOYMENTS"

# map(select()): filter-map in one step
echo "=== Staging names only ==="
jq -r '[.[] | select(.env == "staging") | .name] | join(", ")' <<< "$DEPLOYMENTS"

# Group by environment, requires sort_by first for group_by
echo "=== Grouped by env ==="
jq 'group_by(.env) | map({env: .[0].env, count: length, names: map(.name)})' \
  <<< "$DEPLOYMENTS"

# Count by status
jq 'group_by(.status) | map({status: .[0].status, count: length})' <<< "$DEPLOYMENTS"

5. Transformation with map() and map_values()

map(expr) in jq is equivalent to .[] | expr | [...] with brackets: it applies the expression to every element of an array and returns the result as an array. This is the fundamental transformation operation in jq. map(.name) extracts the name value from every object and returns a string array. map(. * 2) doubles every numeric value. map(if .active then . else empty end) combines filter and map in a single step.

map_values(expr) applies an expression to every value of an object, not to an array, but to the values of a key value map. That is useful for transforming configuration objects: map_values(. + "_v2") appends "_v2" to every value. to_entries and from_entries in jq allow you to transform objects as an array of key value pairs: to_entries | map(select(.value != null)) | from_entries removes all null values from an object. These combinations enable complex data reshaping entirely inside jq, without any external tools.

6. Structure checks with has() and type

Defensive jq programming means checking whether fields are present and have the expected type before accessing them. has("fieldname") returns true if the field is present in the object, even if its value is null. in(object) is the inverse: "key" | in(object). For arrays, has(n) returns whether the array has at least n+1 elements. These checks prevent jq scripts from failing on unexpected API responses or outdated schemas.

Type checks with type return the type as a string: "null", "boolean", "number", "string", "array", "object". Combined with select: select(type == "array") filters only array elements from a mixed input. The functions arrays, objects, strings, numbers, booleans, nulls, values, and scalars are shorthand for select(type == "..."). In jq scripts that process external API responses, every access to a required field should be guarded by a type check or a null check.

#!/usr/bin/env bash
# jq_validation.sh: structural validation and safe field access with jq
set -euo pipefail

# Validate JSON structure before processing
validate_deployment_json() {
  local json="$1"

  # Check required fields exist and have correct types
  local errors
  errors="$(jq -r '
    [
      (if has("id")     and (.id | type) == "number" then empty
       else "Missing or invalid field: id" end),
      (if has("name")   and (.name | type) == "string" then empty
       else "Missing or invalid field: name" end),
      (if has("status") and (.status | strings | test("^(success|failed|running)$")) then empty
       else "Missing or invalid field: status (must be success|failed|running)" end),
      (if has("artifacts") and (.artifacts | type) == "array" then empty
       else "Missing or invalid field: artifacts (must be array)" end)
    ] | join("\n")
  ' <<< "$json")"

  if [[ -n "$errors" ]]; then
    echo "[ERROR] JSON validation failed:" >&2
    echo "$errors" >&2
    return 1
  fi
}

# Safe extraction with type check and default
safe_get() {
  local json="$1" path="$2" expected_type="${3:-string}" default="${4:-}"
  jq -r --arg type "$expected_type" --arg default "$default" \
    'getpath($path | split(".")) |
     if . == null then $default
     elif type == $type then (if $type == "string" then . else tostring end)
     else $default end' <<< "$json" 2>/dev/null || echo "$default"
}

# Type-aware field processing
process_config() {
  local config="$1"

  # Extract only string values from config object
  echo "String config values:"
  jq -r 'to_entries[] | select(.value | type == "string") | "\(.key)=\(.value)"' \
    <<< "$config"

  # Extract numbers and calculate total
  echo "Numeric totals:"
  jq '[to_entries[] | select(.value | type == "number") | .value] | add // 0' \
    <<< "$config"

  # Find nested arrays
  echo "Array fields:"
  jq -r 'to_entries[] | select(.value | type == "array") | .key' <<< "$config"
}

SAMPLE='{"timeout": 30, "env": "prod", "features": ["flag-a", "flag-b"], "debug": false}'
process_config "$SAMPLE"

7. Passing Bash variables safely with --arg

The most common security problem when using jq in Bash is directly interpolating Bash variables into jq filter strings. jq ".name == \"$user_input\"" breaks on quotation marks in the input, on backslashes, and on certain special characters, and it is vulnerable to jq injection. The correct method is always --arg name value: the variable is passed as a typed JSON string, jq takes care of correct escaping, and the filter references it as $name.

Besides --arg (string type), there is --argjson name json_value for JSON values (numbers, booleans, arrays, objects), --rawfile name file for file contents as a string, and --slurpfile name file for JSON files as an array. --args and --jsonargs at the end of the command pass $ARGS.positional as a string or JSON array respectively, for dynamic argument lists. These methods make jq filters in Bash scripts safe and portable, regardless of special characters in the data being processed.

#!/usr/bin/env bash
# jq_variables.sh: safe variable passing and output formatting
set -euo pipefail

DEPLOYMENTS='[
  {"id": 1, "name": "api",      "env": "production", "version": "2.3.0"},
  {"id": 2, "name": "frontend", "env": "staging",    "version": "1.8.4"},
  {"id": 3, "name": "worker",   "env": "production", "version": "2.3.0"}
]'

# WRONG: variable interpolation, breaks with special chars, injection risk
# jq ".[] | select(.env == \"$target_env\")" <<< "$DEPLOYMENTS"

# RIGHT: --arg passes string safely, --argjson for numbers/booleans
target_env="production"
min_id=2

echo "=== Safe variable passing ==="
jq -r --arg env "$target_env" \
  '.[] | select(.env == $env) | .name' <<< "$DEPLOYMENTS"

jq -r --argjson min_id "$min_id" \
  '.[] | select(.id >= $min_id) | "\(.id): \(.name)"' <<< "$DEPLOYMENTS"

# Build JSON payload from Bash variables (never use string concatenation)
deploy_version="2.4.0"
deploy_env="production"
deploy_timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

payload="$(jq -n \
  --arg version "$deploy_version" \
  --arg env "$deploy_env" \
  --arg ts "$deploy_timestamp" \
  --argjson dry_run false \
  '{version: $version, environment: $env, triggered_at: $ts, dry_run: $dry_run}')"

echo "=== Built payload ==="
echo "$payload"

# Format as CSV, TSV or key=value for downstream tools
echo "=== Tab-separated output for further processing ==="
jq -r '.[] | [.id, .name, .env, .version] | @tsv' <<< "$DEPLOYMENTS"

echo "=== CSV output ==="
jq -r '.[] | [.id, .name, .env, .version] | @csv' <<< "$DEPLOYMENTS"

8. Error handling with try/catch and alternatives

jq offers two mechanisms for error handling: the alternative operator // and the try/catch expression. The alternative operator expr // default returns default if expr evaluates to null or false, but not on actual errors. For real error handling (for example a type mismatch or an invalid access), try expr catch "error: \(.)" is the right approach: the catch block traps the error and lets you process it as a string.

An important aspect of jq error handling in Bash scripts: jq returns exit code 0 even if the output is null, as long as the JSON is valid. With -e (exit status), jq returns exit code 1 if the last result is null or false. That is the basis for using jq expressions as Bash conditions. On the other hand, if the input is not valid JSON, jq writes an error message to stderr and returns exit code 5. A pre check with jq . <<< "$input" >/dev/null 2>&1 lets you catch that in Bash.

9. Advanced techniques: streaming and multi document input

In everyday shell work you also encounter jq in scenarios where the input is not a single JSON document but several. jq with -s (slurp) reads all input documents into a single array. Without -s, jq processes each JSON document separately. That turns cat file1.json file2.json | jq -s '.' into a simple merge operator for JSON files.

For NDJSON (newline delimited JSON), a line based format emitted by many logging systems, jq works directly: each line is processed as a separate JSON document. jq 'select(.level == "error")' < app.ndjson filters only error log lines out of an NDJSON log. That enables efficient log analysis right in the shell, with no need to import into a database. Combined with grep for a first pass line filter and jq for structured evaluation, you get a powerful log analysis pipeline.

10. Summary

Using jq professionally in everyday shell work means: extract fields with -r without quotation marks, iterate arrays with .[], filter with select(), transform with map(), check structures with has() and type, pass Bash variables exclusively with --arg, and use -e (exit status) so jq expressions can act as Bash conditions. These six points cover ninety percent of everyday JSON processing tasks in shell scripts.

The single most important rule: never interpolate Bash variables directly into jq filter strings. That is the most common bug and a security risk. --arg is always the correct method. Next comes validating the JSON structure before accessing fields, especially for external API responses that can change over time. With these principles, jq in Bash becomes a dependable part of every API automation, CI/CD pipeline, and log analysis workflow.

11. FAQ: jq for JSON in everyday shell work