Designing Agent Pipelines for Complex Tasks

1. What distinguishes an agent pipeline from a single prompt

An agent pipeline breaks a complex task down into an ordered sequence of clearly separated stages, where each stage is handled by its own agent call with a specific mandate and passes its result on as input to the next stage. The fundamental difference from a single, comprehensive prompt is that a pipeline explicitly structures the task into checkable intermediate steps, instead of expecting an agent to handle the entire complexity in one pass.

With a monolithic prompt for a complex task, for example fully implementing a new feature from requirement to tests, there is a risk that errors in early reasoning steps stay unnoticed, because they are processed in the same pass as all subsequent steps. An agent pipeline turns every intermediate step into a standalone, checkable artifact, which allows errors to be detected early, before they propagate into subsequent stages.

Building an agent pipeline pays off especially for recurring, structurally similar tasks where the stage sequence can be reused stably, such as introducing new features, database migrations, or modernizing legacy modules. For one off, unstructured requests, the additional design effort of a pipeline usually does not pay off.

2. Defining pipeline stages: breaking down complex tasks

The first design step for every agent pipeline is breaking the overall task down into individual stages, each with a clearly defined input and output format. For typical feature development, this often results in four to six stages: requirements analysis, technical design, implementation, test generation, documentation, and a final check. Each stage should answer exactly one question and deliver its result in a form the next stage can directly consume.

A good rule of thumb when defining stages of an agent pipeline: a stage is too large if its result contains several independently checkable statements that cannot be clearly assigned to a single responsibility. In this case, it is worth splitting the stage into two smaller stages, each with its own, narrower mandate. Too many, overly fine grained stages, on the other hand, increase coordination overhead without bringing real added value in traceability.

{
  "pipeline": "feature-development",
  "stages": [
    { "id": "requirements", "input": "ticket_description", "output": "structured_requirements" },
    { "id": "design", "input": "structured_requirements", "output": "technical_design" },
    { "id": "implementation", "input": "technical_design", "output": "code_diff" },
    { "id": "tests", "input": "code_diff", "output": "test_suite" },
    { "id": "documentation", "input": ["code_diff", "test_suite"], "output": "updated_docs" },
    { "id": "final_check", "input": ["code_diff", "test_suite", "updated_docs"], "output": "review_ready_bundle" }
  ]
}

3. Shaping data flow between pipeline stages

The data flow within an agent pipeline should be designed so that each stage receives exactly the information it needs for its mandate, without reprocessing the complete history of all previous stages. In practice, this means every stage produces a structured output artifact, typically as a JSON object with clearly named fields, that serves directly as input for the next stage, supplemented only by the context pieces actually needed.

A common design mistake in an agent pipeline is passing the complete history of all previous stages to every stage, out of concern that important information might otherwise be missing. This leads to unnecessarily large context windows and obscures which information was actually relevant to the current stage. A better approach is an explicit contract between stages: each stage documents exactly which fields it expects as input, and the orchestrator ensures only those fields are passed.

# Explicit contract between pipeline stages: each stage declares its inputs
# The orchestrator passes only the declared fields, not the full history

from dataclasses import dataclass
from typing import Any

@dataclass
class StageContract:
    stage_id: str
    required_inputs: list[str]
    output_schema: dict[str, str]

CONTRACTS = {
    "design": StageContract(
        stage_id="design",
        required_inputs=["structured_requirements"],
        output_schema={"technical_design": "object"},
    ),
    "implementation": StageContract(
        stage_id="implementation",
        required_inputs=["technical_design"],
        output_schema={"code_diff": "object"},
    ),
}

def build_stage_input(stage_id: str, pipeline_state: dict[str, Any]) -> dict:
    contract = CONTRACTS[stage_id]
    missing = [f for f in contract.required_inputs if f not in pipeline_state]
    if missing:
        raise ValueError(f"Stage {stage_id} is missing required inputs: {missing}")
    return {field: pipeline_state[field] for field in contract.required_inputs}

4. Conditional branches and retries in the pipeline

Not every agent pipeline runs linearly. In practice, conditional branches frequently arise, for example when the test generation stage fails and an additional correction step needs to be inserted before the documentation stage. A robust pipeline design explicitly anticipates such branches instead of bolting them on afterward as a special case, by giving every stage a defined error output alongside its regular output, pointing to a correction or escalation stage.

Retries within an agent pipeline should generally be limited. A stage that fails three times in a row usually points to a structural problem in the upstream stage, not a random fluctuation that a simple retry could fix. Once the retry limit is reached, the pipeline should stop and escalate the current state, together with all prior failures, to a human, instead of running indefinitely.

{
  "stage": "tests",
  "regular_output": "test_suite",
  "error_output": {
    "target": "implementation_fix_stage",
    "condition": "test_generation_failed"
  },
  "retry_policy": {
    "max_attempts": 3,
    "on_limit_exceeded": "escalate_to_human"
  }
}

5. Building quality gates between stages

A quality gate is a check sitting between two stages of an agent pipeline that decides whether the result of the previous stage is sufficient to continue. Gates can be automated, for example a linter or test run after the implementation stage, or they can be a second, independent agent check that evaluates the previous stage's result against defined criteria without knowing its reasoning process.

Quality gates are the most important building block for preventing errors from silently propagating through a multi stage agent pipeline. Without gates, errors often confirm each other: a faulty requirements analysis leads to a faulty design, which in turn leads to code that fits the design but is wrong. A gate right after the requirements analysis, checking the completeness and consistency of the extracted requirements, prevents this error from continuing into the following, more expensive stages.

6. Practical example: a pipeline for feature development from spec to test

A concrete example illustrates the interplay of all elements: for introducing a new discount mechanism in a Magento shop, the requirement first goes through the analysis stage, which extracts structured requirements from a ticket text. A quality gate checks whether all necessary business rules, for example combinability with other discounts, were explicitly captured, before the design stage begins.

The design stage translates the requirements into a technical solution referencing existing repository interfaces. The implementation stage turns this design into code, the test stage generates matching PHPUnit tests, and a final quality gate checks code and tests together against the conventions in CLAUDE.md. Only once this last gate passes does the result of the agent pipeline count as review ready and get made available as a pull request.

#!/usr/bin/env bash
# Run a multi stage agent pipeline with quality gates between stages
set -euo pipefail

TICKET_ID="${1:?Usage: run-pipeline.sh TICKET-123}"
STATE_DIR="pipeline/${TICKET_ID}"
mkdir -p "$STATE_DIR"

run_stage() {
  local stage="$1"
  local input_file="$2"
  claude --agent-config "pipeline-stages/${stage}.json" \
    --input "$input_file" \
    --output "${STATE_DIR}/${stage}.json"
}

run_gate() {
  local gate_name="$1"
  local artifact="$2"
  claude --agent-config "pipeline-gates/${gate_name}.json" \
    --input "$artifact" \
    --output "${STATE_DIR}/${gate_name}-result.json"
  jq -e '.passed == true' "${STATE_DIR}/${gate_name}-result.json" > /dev/null
}

run_stage "requirements" "tickets/${TICKET_ID}.md"
run_gate "requirements_gate" "${STATE_DIR}/requirements.json" \
  || { echo "[pipeline] requirements gate failed, escalating" >&2; exit 1; }

run_stage "design" "${STATE_DIR}/requirements.json"
run_stage "implementation" "${STATE_DIR}/design.json"
run_stage "tests" "${STATE_DIR}/implementation.json"

run_gate "final_gate" "${STATE_DIR}/tests.json" \
  || { echo "[pipeline] final gate failed, escalating" >&2; exit 1; }

echo "[pipeline] Feature pipeline complete and review ready for ${TICKET_ID}"

7. Error handling and rollback in multi stage pipelines

Errors in an agent pipeline must be handled differently from errors in a single agent call, because a failure in a late stage may mean results from several previous stages need to be discarded. A well thought out pipeline design therefore persists state after every successfully completed stage, so a new run does not necessarily have to start at the first stage but can resume from the last successful one.

For critical agent pipelines that also touch production systems, for example database migration stages, an explicit rollback mechanism should additionally be provided, undoing the effects of earlier stages if a late stage fails, instead of leaving the system in an inconsistent intermediate state. This rollback mechanism is not an optional extra, but a mandatory part of every pipeline that goes beyond pure code generation.

8. Monitoring and cost control for an agent pipeline

Operating an agent pipeline in production requires monitoring on two levels: technical, to capture runtime and error rate per stage, and economic, to keep an eye on token consumption per stage and across the entire pipeline. A stage that fails at a quality gate above average often points to a structural problem in its prompt or in the upstream stage and should be specifically investigated, instead of simply loosening the gate.

For cost control, an upper limit on tokens per pipeline run is advisable, combined with a warning when a single stage consumes an above average number of tokens. This limit prevents a misconfigured stage from silently causing high costs before anyone notices the problem. Regular evaluations also show which stages of an agent pipeline are worth optimizing, for example through smaller, more targeted prompts.

#!/usr/bin/env bash
# Monitor token consumption per stage and warn on above average usage
set -euo pipefail

STATE_DIR="$1"
TOKEN_LIMIT_PER_RUN=150000
total_tokens=0

for stage_file in "${STATE_DIR}"/*.json; do
  stage_tokens=$(jq -r '.token_usage // 0' "$stage_file")
  stage_name=$(basename "$stage_file" .json)
  total_tokens=$(( total_tokens + stage_tokens ))

  if (( stage_tokens > 20000 )); then
    echo "[monitoring] WARNING: stage ${stage_name} used ${stage_tokens} tokens, above average" >&2
  fi
done

echo "[monitoring] Total pipeline token usage: ${total_tokens}"
if (( total_tokens > TOKEN_LIMIT_PER_RUN )); then
  echo "[monitoring] ERROR: pipeline exceeded token budget of ${TOKEN_LIMIT_PER_RUN}" >&2
  exit 1
fi

9. Monolithic prompt versus agent pipeline compared

The following table compares a single, comprehensive prompt with the approach of a structured agent pipeline.

An agent pipeline pays off especially for recurring, structurally similar tasks where the one time design effort is amortized over many runs. For one off cases, a direct, well written prompt often remains the more efficient choice.

10. Summary

A well designed agent pipeline breaks complex development tasks down into clearly separated, checkable stages, instead of leaving them to a single, overloaded prompt. Clearly defined data flow between stages, built in quality gates and thoughtful error handling with rollback capability are the decisive building blocks that turn a loose sequence of agent calls into a reliable, reusable system.

Designing an agent pipeline pays off especially for recurring, structurally similar tasks, where the effort of stage and gate design is amortized over many runs. Whoever consistently invests this effort and considers monitoring and cost control from the start gains an automation that stays traceable and robust even for complex, multi stage tasks.

11. FAQ: Designing Agent Pipelines for Complex Tasks