Using Claude in CI/CD Pipelines

1. Why Claude Makes Sense in CI/CD Pipelines (and Where It Does Not)

CI/CD pipelines run on every commit, often dozens of times a day. An AI step only pays for itself if the value it adds justifies the extra time and cost budget. Claude is well suited to tasks where an approximate, human-reviewed result is good enough: summaries, drafts, categorization. For tasks that need exact, deterministic answers, such as whether tests pass or a build compiles, classic tooling remains the right choice.

The key difference from traditional CI steps: LLM output is not deterministic. Two identical diffs can easily produce slightly different summaries. That makes it unsuitable as a merge gate, but perfectly fine as a supporting comment on a pull request. Treating Claude as an additional source of information rather than an automated decision-maker sidesteps most of the risk. An AI step should never decide a merge or a deployment on its own.

2. Use Cases: PR Summaries, Changelogs, Risk Flagging

Three use cases have proven robust in practice. First, automated PR summaries: Claude reads the diff and posts a comment covering the key changes, affected components, and open questions before a human reviewer even opens the code. Second, changelog drafts: from the commit history since the last tag, a structured release-notes draft emerges, grouped by feature, fix, and breaking change.

Third, diff risk assessment: changes to authentication, payment logic, migration scripts, or the CI configuration itself get flagged and trigger a mandatory review by a second person. All three cases share a pattern: Claude delivers a draft or an assessment, and a human makes the decision. That reduces review effort without delegating responsibility to a model that can simply be wrong about a specific diff.

3. Running Claude Non-Interactively: Headless Mode in the Pipeline

Claude Code offers a headless mode via the -p flag, which accepts a prompt without asking interactive follow-up questions and writes the result to stdout. For pipelines, --output-format json matters too: instead of free text, Claude returns structured data that can be parsed and processed directly, for example to feed a GitHub API call. The API key is injected as a CI secret, never hardcoded in the workflow file.

For simple single-shot tasks, such as summarizing a diff, a direct call to the Anthropic API without Claude Code is often enough and saves overhead. Claude Code pays off when the step needs to read multiple files, run tests, or interact with the repository. Either way, the process must run without a TTY, return defined exit codes, and must not block the pipeline if the request fails or hits a timeout.

#!/usr/bin/env bash
# ci/summarize-pr.sh - Run Claude Code in headless mode to summarize a pull request diff
set -euo pipefail

# ANTHROPIC_API_KEY is injected as a CI secret, never hardcoded
export ANTHROPIC_API_KEY="${ANTHROPIC_API_KEY:?Missing ANTHROPIC_API_KEY secret}"

BASE_BRANCH="${GITHUB_BASE_REF:-main}"
DIFF=$(git diff "origin/${BASE_BRANCH}"...HEAD)

# Headless mode: -p takes a prompt, no interactive prompts, no TTY required
# --output-format json returns structured data instead of free text
claude -p "Summarize this pull request diff for a reviewer. Return JSON matching the pr_summary schema." \
  --output-format json \
  --allowedTools "" \
  --max-turns 1 \
  <<< "$DIFF" > pr-summary.json

# Fail the step (but not the pipeline) if Claude produced no usable output
if [[ ! -s pr-summary.json ]]; then
  echo "::warning::Claude produced no PR summary, skipping comment"
  exit 0
fi

4. Guardrails: Permissions, Sandboxing, and Output Validation

A non-interactive AI step in the pipeline needs tighter boundaries than a local development session. --allowedTools defines which tools Claude Code is even permitted to call; for a pure text-analysis task, read access is enough, and bash or write permissions are usually unnecessary and an avoidable risk. The runner should execute in an isolated, short-lived environment with no access to production secrets, deploy keys, or merge permissions.

The output must be validated before use: if you expect a JSON schema, check the response against that schema before the value gets substituted into an API call or a comment template. Never pass raw text directly into a shell command or unfiltered Markdown, that opens the door to prompt injection via manipulated commit messages or file contents. A timeout per call and a token budget per run prevent a single step from blocking the entire pipeline or triggering unexpected cost.

5. Automating PR Summaries

The workflow triggers on the pull_request event, fetches the diff via git diff against the target branch, and hands it to Claude with an instruction to return a structured summary in JSON: the core change, affected modules, open questions. That structure renders reliably into a PR comment via the GitHub API, without formatting errors in free-text output making the comment unreadable.

On every subsequent push to the same PR, the existing comment should be updated rather than a new one created, otherwise dozens of AI comments pile up on long-lived PRs. The GitHub API lets you locate a previous bot comment via a fixed marker in the comment body. Important: the summary does not replace a review, it merely shortens the time a reviewer needs to get oriented in a large diff.

{
  "type": "object",
  "properties": {
    "summary": {
      "type": "string",
      "description": "One paragraph describing the core change"
    },
    "affected_modules": {
      "type": "array",
      "items": { "type": "string" }
    },
    "open_questions": {
      "type": "array",
      "items": { "type": "string" }
    },
    "risk_level": {
      "type": "string",
      "enum": ["low", "medium", "high"]
    }
  },
  "required": ["summary", "affected_modules", "risk_level"],
  "additionalProperties": false
}

6. Flagging Suspicious Diffs for Extra Review

Certain diff categories warrant extra scrutiny: changes to authentication and authorization logic, new or changed dependencies, files matching secrets patterns, unusually large diffs, and changes to the CI configuration itself, because a compromised pipeline definition can undermine the entire build process. Claude receives the diff together with this category list and returns a risk assessment with reasoning, not a binary approval.

On high risk, the pipeline applies a label, requests a second reviewer, or blocks auto-merge rules, depending on how the team has configured its merge policy. The error rate here should deliberately be tolerated asymmetrically: a false positive costs one extra review round, a false negative on genuinely risky code potentially costs far more. The classification should therefore be calibrated to err on the cautious side rather than the lenient one.

// .github/scripts/flag-risky-diff.js
// Runs inside actions/github-script - posts a warning comment when Claude
// classifies the diff as high risk and requests an extra reviewer.
module.exports = async ({ github, context, core }) => {
  const fs = require('fs');
  const result = JSON.parse(fs.readFileSync('risk-assessment.json', 'utf8'));

  // Guardrail: never trust the model output blindly, validate the shape first
  const validLevels = ['low', 'medium', 'high'];
  if (!result.risk_level || !validLevels.includes(result.risk_level)) {
    core.warning('Risk assessment output did not match expected schema, skipping');
    return;
  }

  if (result.risk_level !== 'high') {
    core.info(`Risk level: ${result.risk_level}, no extra review required`);
    return;
  }

  await github.rest.issues.addLabels({
    owner: context.repo.owner,
    repo: context.repo.repo,
    issue_number: context.payload.pull_request.number,
    labels: ['needs-security-review'],
  });

  await github.rest.issues.createComment({
    owner: context.repo.owner,
    repo: context.repo.repo,
    issue_number: context.payload.pull_request.number,
    body: `**Claude flagged this diff as high risk.**\n\nReasons: ${result.reasons.join(', ')}\n\nPlease request a second reviewer before merging.`,
  });
};

7. Drafting Changelogs from Commit History

For release notes, a script collects every commit since the last git tag and hands them to Claude with a request to group them into feature, fix, and breaking change sections and summarize them in plain language. The result lands as a draft in a pull request against the CHANGELOG file, not as an automatic commit to the main branch, so a human proofreads it before publication.

The quality of the draft depends heavily on the quality of the commit messages: without a conventional-commits convention, Claude has to guess the category from free-text messages, which occasionally leads to misclassification, for instance a fix showing up as a feature. That risk cannot be fully eliminated, only caught by review before publication. The time saved compared to a manually written changelog is still substantial, especially with many small commits.

#!/usr/bin/env python3
"""ci/draft_changelog.py - Draft release notes from commit history via the Claude API."""
import os
import subprocess
import sys

import anthropic

# Guardrail: hard timeout so a hanging request cannot block the pipeline
REQUEST_TIMEOUT_SECONDS = 30
# Guardrail: cap output tokens to bound cost per run
MAX_OUTPUT_TOKENS = 2000


def get_commits_since_last_tag() -> str:
    last_tag = subprocess.run(
        ["git", "describe", "--tags", "--abbrev=0"],
        capture_output=True, text=True, check=True,
    ).stdout.strip()
    log = subprocess.run(
        ["git", "log", f"{last_tag}..HEAD", "--pretty=format:%s"],
        capture_output=True, text=True, check=True,
    ).stdout
    return log


def main() -> int:
    api_key = os.environ.get("ANTHROPIC_API_KEY")
    if not api_key:
        print("::error::ANTHROPIC_API_KEY is not set", file=sys.stderr)
        return 1

    commits = get_commits_since_last_tag()
    if not commits.strip():
        print("No commits since last tag, skipping changelog draft")
        return 0

    client = anthropic.Anthropic(api_key=api_key)
    response = client.with_options(timeout=REQUEST_TIMEOUT_SECONDS).messages.create(
        model="claude-haiku-4-5",
        max_tokens=MAX_OUTPUT_TOKENS,
        messages=[{
            "role": "user",
            "content": (
                "Group these commit messages into Feature, Fix and Breaking Change "
                "sections for a changelog draft. Commits:\n" + commits
            ),
        }],
    )

    draft = next(b.text for b in response.content if b.type == "text")
    with open("CHANGELOG.draft.md", "w") as f:
        f.write(draft)

    print(f"Draft written, {response.usage.output_tokens} output tokens used")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

8. Cost and Latency: What an AI Step Actually Costs Per Commit

With several dozen commits a day in an active repository, even small model calls add up. For routine tasks such as PR summaries, a cheap, fast model like Claude Haiku is enough, while deeper risk analysis benefits from a stronger model like Claude Sonnet. Reflexively using the most expensive available model for every pipeline run is rarely justified and drives up both cost and latency unnecessarily.

Latency is the second factor: an AI step that blocks the pipeline extends the wait time for every developer on every push. A non-blocking, parallel step that posts the PR comment while tests and build run independently is preferable. Trigger conditions should be limited to PR events rather than every single push, and a cache keyed on the diff hash skips the call when only documentation or already-analyzed code has changed.

Counting tokens before rollout helps with cost estimation: a typical diff of a few hundred lines often falls in the range of a few thousand tokens, which at current pricing for fast models lands in the range of a few cents per run. Multiplying that figure by the expected number of daily PR runs gives a realistic budget baseline before the step is rolled out across every repository's pipeline.

#!/usr/bin/env bash
# ci/should-run-ai-step.sh - Skip the AI step for unchanged or docs-only diffs
set -euo pipefail

DIFF_HASH=$(git diff origin/main...HEAD -- . ':!docs' ':!*.md' | sha256sum | cut -d' ' -f1)
CACHE_FILE=".ci-cache/${DIFF_HASH}.done"

if [[ -f "$CACHE_FILE" ]]; then
  echo "Diff already analyzed (hash ${DIFF_HASH:0:12}), skipping Claude step"
  echo "skip=true" >> "$GITHUB_OUTPUT"
  exit 0
fi

# Model selection: cheap model on feature branches, stronger model on release branches
if [[ "${GITHUB_REF_NAME}" == release/* ]]; then
  echo "model=claude-sonnet-5" >> "$GITHUB_OUTPUT"
else
  echo "model=claude-haiku-4-5" >> "$GITHUB_OUTPUT"
fi

mkdir -p .ci-cache
touch "$CACHE_FILE"
echo "skip=false" >> "$GITHUB_OUTPUT"

9. Comparing Claude Usage Patterns in the Pipeline

The overview below summarizes which usage patterns tend to cause unnecessary cost, false alarms, or security risk in practice, and which alternative has proven itself instead.

None of these patterns is a one-time configuration. Trigger conditions, model choice, and permissions should be reviewed regularly as the repository, team behavior, or cost structure changes. An AI step that was sensibly configured at launch can accumulate stale baggage a year later, for instance because it still points at a more expensive model that could long since have been replaced by a cheaper alternative.

10. Summary

Claude pays off in CI/CD pipelines for supportive, non-decisive tasks: PR summaries, changelog drafts, and risk flags for extra review. Headless mode with structured JSON output makes the integration technically straightforward, but the real work lies in the guardrails: restricted tool permissions, isolated runners, validated output, and timeouts.

Cost and latency are not a side issue in a pipeline that reacts to every commit, they are a core criterion for model choice and trigger conditions. Cheap models for routine tasks, targeted triggers instead of every push, and a cache over already-analyzed diffs keep ongoing costs within a reasonable range. In the end, humans remain the final authority on merge and deployment decisions, Claude simply delivers the information that makes that decision faster.

11. FAQ: Using Claude in CI/CD Pipelines