Iterating and Systematically Improving Prompts

1. Why prompt development works like debugging

A first prompt draft rarely works completely correctly on the first try, just as little as a first code draft rarely runs without errors on the first pass. The difference lies in how developers handle that fact: with code, an iterative debugging process is taken for granted, with prompts it's frequently skipped. A prompt gets written, tried once, accepted as "basically works", and then never touched again, even if it regularly produces unwanted output in practice.

Treating prompt development as a debugging cycle instead restores that structure: formulate a hypothesis for why the output deviates from expectation, test a specific variant, observe the result, and build the next adjustment on top of it. The key difference from classic debugging is that language models are not deterministic. A single deviating output can be random noise; a pattern that recurs across several runs is a real signal. That distinction is the foundation of any systematic prompt improvement.

2. The iteration cycle: hypothesis, test, refinement

The practical cycle consists of four steps that can be repeated as many times as needed. First: observe precisely which part of the output deviates from expectation, not just note "the result is bad" in general, but name exactly what's missing, what's excessive, or which format doesn't fit. Second: formulate a hypothesis for why that happens, for example because a term in the prompt is ambiguous, context is missing, or two instructions contradict each other. Third: test exactly one change that addresses that hypothesis. Fourth: compare the new result with the previous one and decide whether the hypothesis was confirmed.

It's crucial to change only one variable per iteration. Anyone who adjusts the format requirement, the amount of context, and the examples all at once can no longer tell, when an improvement happens, which change actually caused it, and risks making the same mistake again on the next task. A short note per iteration, what changed, why, and what result it produced, makes the process traceable and considerably speeds up later adjustments, because you can build on already verified hypotheses.

3. Writing a baseline prompt and documenting failures

The starting point of any iteration is a deliberately simple baseline prompt: the most direct formulation of the task, without tricks, without an elaborate role description, without few-shot examples. This prompt gets tested against several real inputs, not just a single one that happened to be at hand. Every deviation gets documented concretely, for example "ignores the line number in the diff" or "answers in English even though the code has German comments", instead of a generic "not good enough".

This baseline serves as the reference point for every comparison that follows. Without a documented starting state, you can no longer determine later whether a change was actually an improvement or just feels better subjectively because you no longer remember the previous failure precisely. In practice, a simple folder in the project repository with prompt versions and a short log file that records each iteration with date, change, and observation is enough.

#!/usr/bin/env bash
# run-prompt-test.sh - runs the current prompt version against sample inputs
# and stores every run for later comparison
set -euo pipefail

PROMPT_VERSION="v3"
PROMPT_FILE="prompts/code-review/${PROMPT_VERSION}.txt"
SAMPLES_DIR="prompts/code-review/samples"
RESULTS_DIR="prompts/code-review/results/${PROMPT_VERSION}"
TIMESTAMP="$(date +%Y%m%d-%H%M%S)"

mkdir -p "$RESULTS_DIR"

for sample in "$SAMPLES_DIR"/*.diff; do
  name="$(basename "$sample" .diff)"
  echo "[RUN] ${PROMPT_VERSION} on ${name}"

  claude --print \
    --append-system-prompt "$(cat "$PROMPT_FILE")" \
    < "$sample" > "${RESULTS_DIR}/${name}-${TIMESTAMP}.txt"
done

echo "[DONE] Results stored in ${RESULTS_DIR}"
echo "Compare against previous run with: diff results/v2/*.txt results/v3/*.txt"

4. Recognizing typical causes of misunderstanding

After several iteration rounds across different tasks, certain causes of misunderstanding keep recurring. The most common is an ambiguous term without a definition: "summarize briefly" means two sentences to one person and a whole paragraph to another, and without further guidance Claude picks its own interpretation. Just as common is a missing explicit format requirement, so the answer arrives sometimes as prose, sometimes as a list, sometimes as a table, even though downstream processing needs a fixed format.

Other recurring causes are implicit assumptions about context the model simply doesn't have, such as project conventions that only exist in the developer's head, as well as contradictory instructions like "be precise" and "explain thoroughly" in the same prompt. Few-shot examples can also backfire when they anchor a pattern that doesn't fit the specific case. Once a pattern shows up more than once, it's worth keeping a short checklist that gets checked against every new prompt. That prevents repeated mistakes and makes the first hypothesis in future iterations noticeably more accurate.

5. Building a prompt library for recurring tasks

A prompt that works reliably after several iterations shouldn't get lost in the chat history; it should land as a versioned file in the project repository, supplemented with metadata: what task it's meant for, when it was last verified against the test set, what known limitations it has, and which model version it was tested with. This library is treated like code, including pull request review before a new version replaces the old one.

In practice, a simple structure has proven effective: one directory per task type, containing the current prompt version as a text file and an accompanying metadata file with test cases and expected patterns. New team members can then draw on already verified prompts instead of starting from zero on every recurring task, like commit message generation or code review, and running through the same misunderstanding patterns again.

{
  "task": "php-code-review",
  "version": "v3",
  "model_tested": "claude-sonnet-4-5",
  "last_verified": "2026-07-08",
  "prompt_file": "prompts/code-review/v3.txt",
  "output_format": "json-array-of-findings",
  "known_limitations": [
    "flags missing PHPDoc even on trivial private getters",
    "does not detect N+1 queries across repository boundaries"
  ],
  "test_cases": [
    { "input": "samples/order-repository.diff", "min_findings": 1 },
    { "input": "samples/clean-refactor.diff", "min_findings": 0 },
    { "input": "samples/sql-injection-risk.diff", "min_findings": 1, "must_contain": "sql injection" }
  ],
  "changelog": [
    { "version": "v1", "change": "initial generic review prompt" },
    { "version": "v2", "change": "added explicit review criteria" },
    { "version": "v3", "change": "added output format and excluded pure style issues" }
  ]
}

6. Practical example: iterating a code review prompt step by step

A concrete example makes the cycle tangible. Version 1 simply read "Review this PHP code". The result: mostly style comments on indentation and variable names, but neither of the two real security issues in the test diff got mentioned. Hypothesis: without concrete criteria, the model picks the most obvious, surface-level observations. Version 2 added explicit criteria: security, N+1 queries, Magento coding standards. Result: noticeably more relevant findings, but the output format varied between prose, a numbered list, and occasionally markdown tables, which made automated post-processing harder.

Version 3 added an explicit output format with an example output (a JSON array with severity, line, and description). The format was consistent from then on, but new false positives appeared around pure formatting questions the team deliberately didn't care about. Version 4 fixed that with an explicit exclusion rule: "Ignore pure style issues like indentation or whitespace, the linter already handles that." Every one of these four versions was tested not just against the one original case, but against a fixed set of eight real diffs, to make sure an improvement in one place didn't create new problems somewhere else.

"""
prompt_eval.py - runs multiple prompt versions against a fixed test set
and reports which cases pass the defined assertions.
"""
import json
from pathlib import Path
from anthropic import Anthropic

client = Anthropic()

def run_prompt(prompt_text: str, diff_content: str) -> str:
    """Send one review prompt with a code diff and return the raw response text."""
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        system=prompt_text,
        messages=[{"role": "user", "content": diff_content}],
    )
    return response.content[0].text

def evaluate_version(version_dir: Path) -> None:
    """Run one prompt version against all configured test cases and print results."""
    meta = json.loads((version_dir / "meta.json").read_text())
    prompt_text = (version_dir / "prompt.txt").read_text()

    passed, failed = 0, 0
    for case in meta["test_cases"]:
        diff_content = Path(case["input"]).read_text()
        output = run_prompt(prompt_text, diff_content)

        findings = json.loads(output) if output.strip().startswith("[") else []
        ok = len(findings) >= case["min_findings"]
        if case.get("must_contain"):
            ok = ok and case["must_contain"] in output.lower()

        passed += ok
        failed += not ok
        print(f"  {'PASS' if ok else 'FAIL'} - {case['input']}")

    print(f"{meta['version']}: {passed} passed, {failed} failed")

if __name__ == "__main__":
    evaluate_version(Path("prompts/code-review/v4"))

7. Testing systematically with multiple examples, not single cases

A common mistake in prompt iteration is testing a change only against the one case that prompted it. The risk: the prompt gets optimized for exactly that case and simultaneously gets worse on other, previously correctly handled inputs, without anyone noticing, because nobody re-checks the rest. A fixed test set of at least five to ten diverse cases, including edge cases like empty input, unusually long input, or ambiguous boundary cases, reliably catches that.

In practice this means re-running the complete test set automatically on every prompt change, not just the one case currently in focus. A simple script that runs through all test cases and compares the results against the previous run makes this effort negligible compared to the risk of an unnoticed regression. It's important to maintain the test set itself: as soon as a new kind of misunderstanding shows up, a matching case gets permanently added, so the same mistake doesn't quietly come back.

#!/usr/bin/env bash
# eval-suite.sh - runs the full regression test set against a prompt version
# and prints a pass/fail summary instead of relying on a single sample
set -euo pipefail

VERSION="${1:?Usage: eval-suite.sh <version>}"
PROMPT_DIR="prompts/code-review/${VERSION}"
CASES_DIR="prompts/code-review/testset"

pass_count=0
fail_count=0

for case_file in "$CASES_DIR"/*.json; do
  case_name="$(basename "$case_file" .json)"
  expected_min="$(jq -r '.min_findings' "$case_file")"
  input_file="$(jq -r '.input' "$case_file")"

  output="$(claude --print --append-system-prompt "$(cat "$PROMPT_DIR/prompt.txt")" < "$input_file")"
  actual_count="$(echo "$output" | jq 'length' 2>/dev/null || echo 0)"

  if (( actual_count >= expected_min )); then
    echo "[PASS] $case_name"
    ((pass_count++))
  else
    echo "[FAIL] $case_name (expected >= $expected_min, got $actual_count)"
    ((fail_count++))
  fi
done

echo "Result: ${pass_count} passed, ${fail_count} failed for ${VERSION}"

8. Versioning and team workflow for prompts

Prompts used in production deserve the same workflow as code: versioning in Git, pull request review by a second pair of eyes, and a changelog that explains why a version changed. That matters especially because prompt changes are rarely obvious, a single added or removed word can noticeably shift behavior without the diff standing out at first glance in code review.

A second, often overlooked aspect is the model switch: when a team moves to a new Claude version, all production prompts should run against the maintained test set again, because response behavior can shift slightly between model versions even when the prompt text stays identical. This can be automated in the CI pipeline: a regression test runs against defined reference outputs on every merge and reports deviations as a warning rather than a hard build failure, since a language model's output is naturally not character-for-character deterministic.

// ci-prompt-regression.js - Node script for CI: compares current prompt
// output against stored golden references and flags drift as a warning
import Anthropic from "@anthropic-ai/sdk";
import { readFileSync, readdirSync } from "node:fs";
import path from "node:path";

const client = new Anthropic();
const versionDir = "prompts/commit-message/v2";
const goldenDir = path.join(versionDir, "golden");

async function checkDrift() {
  const promptText = readFileSync(path.join(versionDir, "prompt.txt"), "utf8");
  let driftCount = 0;

  for (const file of readdirSync(goldenDir)) {
    const diffInput = readFileSync(path.join(goldenDir, file, "input.diff"), "utf8");
    const expected = readFileSync(path.join(goldenDir, file, "expected.txt"), "utf8").trim();

    const response = await client.messages.create({
      model: "claude-sonnet-4-5",
      max_tokens: 256,
      system: promptText,
      messages: [{ role: "user", content: diffInput }],
    });

    const actual = response.content[0].text.trim();
    if (!actual.toLowerCase().includes(expected.toLowerCase().slice(0, 20))) {
      console.warn(`[DRIFT] ${file}: expected pattern not found in output`);
      driftCount++;
    }
  }

  console.log(`Checked ${readdirSync(goldenDir).length} cases, ${driftCount} drifted`);
  // Warning only, never a hard exit(1): non-determinism is expected here
}

checkDrift();

9. Prompt iteration patterns compared

The table below summarizes exactly how unsystematic prompt tinkering differs from a structured debugging loop, and what practical advantage the systematic variant brings in each case.

What stands out in this table: none of the systematic variants require exotic tooling. A test directory, a log file, and a short script are enough to move from lucky guesses to a traceable, repeatable process. The effort pays off most for prompts used more than once, such as recurring reviews, automated commit messages, or support replies.

10. Summary

Systematic prompt iteration solves a fundamental problem that inevitably shows up with prompts written just once: quality fluctuates, mistakes repeat, and improvements can no longer be distinguished from luck. The debugging cycle of hypothesis, targeted test, and refinement brings the same structure that has proven itself in classic code debugging for decades, adapted to the non-deterministic nature of language models. A documented baseline, a fixed test set with several diverse cases, and a versioned prompt library turn isolated successes into a repeatable process.

The biggest lever is recognizing recurring patterns behind misunderstandings, ambiguous terms, missing format requirements, contradictory instructions, and capturing those insights as a checklist for future prompts. In teams, it also pays off to treat prompts like code: with versioning, review, and regression tests on every model switch. The effort is manageable, but the effect is noticeable, especially for prompts used daily in reviews, commit messages, or support replies.

11. FAQ: Iterating and Systematically Improving Prompts