Generating Tests with AI: What to Watch For

1. AI-generated tests as a starting point, not a finished result

Claude Code can turn a PHP class into a complete PHPUnit test class in seconds, complete with mocks, data providers, and fixtures. That saves real time on tedious boilerplate: setup methods, constructor calls with every required dependency, naming that follows PSR conventions. Anyone who has done this step manually gains a noticeable speed boost, especially for legacy classes with no existing test coverage at all.

The mistake happens when that speed gain gets confused with quality. A generated test class that passes only proves that its assertions match the current code behavior, not that this behavior is correct. The actual value of tests lies in protecting future changes against a specification, and the AI only knows that specification if it is given one explicitly. Without context, every generated test remains a suggestion that needs review by a developer with domain knowledge of the actual requirement.

2. The core problem: testing current state instead of intended behavior

When a language model generates a test suite for an existing method, it typically observes what the code currently returns and writes an assertion that confirms exactly that. For a bug-free method, this is harmless. For a method with a bug, though, the bug becomes a locked-in expectation: the test will fail in the future the moment someone fixes the bug, because the fix formally looks like a regression.

This pattern is particularly insidious because the affected tests look completely normal at first glance, pass cleanly, and raise the coverage number. Only comparing the test against the actual business requirement, such as a ticket, documentation, or a deliberate question to the domain expert, reveals that a test has cemented the wrong behavior. Developers who use AI-generated tests therefore need a deliberate answer, for every non-trivial method, to the question of what the method is supposed to do according to the specification, before accepting an assertion.

# Claude Code test generation and verification loop
claude "Generate PHPUnit tests for src/app/code/Mironsoft/Pricing/Model/TierPriceCalculator.php.
Cover null price, negative quantity, and currency rounding edge cases."

# Never trust generated tests blindly - run them first
bin/cli vendor/bin/phpunit app/code/Mironsoft/Pricing/Test/Unit/TierPriceCalculatorTest.php --testdox

# Then check what the tests actually assert, not just whether they pass
bin/cli vendor/bin/phpunit --coverage-text app/code/Mironsoft/Pricing/Test/Unit/TierPriceCalculatorTest.php

3. How Claude Code generates tests: context, prompts, examples

The quality of generated tests depends directly on the context provided. A prompt that only names the file path delivers, at best, tests for the happy path. A prompt that explicitly states the business rule, known edge cases, and the expected error behavior delivers much more targeted tests. Claude Code can incorporate existing code, linked tickets, and even a file's commit history when that information is accessible.

In practice, a two-step approach works most reliably: first, write down the business specification as text, for example as a list of rules and edge cases. Then have the AI generate tests against that specification, not against the existing code. This detour through an explicit description prevents the AI from silently inferring the expectation from the observed behavior, and makes the generated tests noticeably more robust against later refactoring.

final class TierPriceCalculatorTest extends TestCase
{
    // WRONG: generated from observing the current (buggy) output.
    // The calculator currently returns a rounded-down price for negative
    // quantities, which is not the intended business rule - it merely
    // reflects a bug that a generator will happily lock in as "expected".
    public function testCalculatePriceWithNegativeQuantityReturnsZero(): void
    {
        $calculator = new TierPriceCalculator();
        $result = $calculator->calculate(19.99, -5);
        self::assertSame(0.0, $result); // asserts the bug, not the spec
    }

    // RIGHT: derived from the actual business rule in the ticket/spec,
    // independent of what the current implementation happens to return.
    public function testCalculateThrowsOnNegativeQuantity(): void
    {
        $calculator = new TierPriceCalculator();
        $this->expectException(InvalidArgumentException::class);
        $calculator->calculate(19.99, -5);
    }
}

4. Reading coverage numbers correctly: genuine edge cases vs. padding

High line coverage says little about whether the tests actually verify anything. AI models tend to generate at least one assertion per code line, even if that assertion is trivial, such as assertInstanceOf for an object that could never fail to be one, or assertTrue(true) as a pure filler to formally close out a test method. Such lines count toward the coverage metric but do nothing to improve the suite's ability to catch real bugs.

Genuine edge cases involve boundary values, invalid input, empty collections, concurrency, and error paths, exactly the spots where production code actually breaks. Anyone evaluating a generated test suite should therefore not just look at the coverage percentage, but ask a targeted question: which of these tests would fail if I deliberately introduced a bug? Tests that stay green under almost any conceivable code change are pure padding and provide no real safety net.

{
  "file": "Model/TierPriceCalculator.php",
  "line_coverage": "96.4%",
  "mutation_score": "38.2%",
  "mutants_total": 47,
  "mutants_killed": 18,
  "mutants_escaped": 29,
  "assessment": "high line coverage but low mutation score indicates assertions that do not verify actual behavior, a typical signature of AI-generated padding tests"
}

5. Magento specifics: unit, integration, and API tests

In Magento projects, Claude Code reliably generates unit tests for view models and service classes with clear, injected dependencies, because these are easy to mock. It gets harder with integration tests, which need a real database connection, fixtures, and the Magento object manager configuration. Here, the generated test class has to be wired up with the correct XML fixtures from dev/tests/integration/testsuite, otherwise the test fails for purely technical reasons, regardless of the actual business logic.

Extra caution is needed for repositories and plugins: a mock the AI generates for a repository interface often defaults to returning only the success case and leaves out error paths such as NoSuchEntityException entirely. Plugins that intercept before or after a core method, in particular, need tests that also check what happens when the wrapped method itself throws an exception. An AI only generates these cases reliably when explicitly asked to.

6. The review process: what developers must check in AI tests

A structured review of an AI-generated test suite differs from a classic code review, because it examines the expectation, not the implementation. The central question for every assertion is: does this expected value come from the specification, or was it simply copied from the current code behavior? Whenever there is uncertainty, it's worth checking the associated ticket or briefly asking the domain expert before the test gets merged.

A second checkpoint concerns missing cases. Because an AI naturally only tests what it has been told about or what the code makes visible, tests are often missing for states that aren't explicitly handled in the code, such as concurrency, timezone issues, or race conditions in parallel orders. A short checklist of the known risk areas for the module in question helps close these gaps systematically, instead of relying solely on the generated suite.

#!/usr/bin/env python3
"""Flag AI-generated test files with high coverage but low mutation score."""
import json
import sys

COVERAGE_THRESHOLD = 80.0
MUTATION_THRESHOLD = 60.0


def flag_padding(report_path):
    with open(report_path, encoding="utf-8") as handle:
        report = json.load(handle)

    flagged = []
    for entry in report["files"]:
        coverage = entry["line_coverage_percent"]
        mutation = entry["mutation_score_percent"]
        # High coverage with a much lower mutation score is the classic
        # signature of tests that assert output without checking behavior.
        if coverage >= COVERAGE_THRESHOLD and mutation < MUTATION_THRESHOLD:
            flagged.append(entry["file"])

    return flagged


if __name__ == "__main__":
    for path in flag_padding(sys.argv[1]):
        print(f"[REVIEW] possible padding test: {path}")

7. Mutation testing as a countercheck on test quality

Mutation testing answers exactly the question that raw coverage numbers leave open: does the test suite actually catch bugs? A mutation testing tool such as Infection systematically alters production code in small steps, for instance by flipping a comparison operator or changing a constant, and checks whether at least one test fails as a result. If no test fails, the altered code section is effectively untested, regardless of what the coverage number claims.

For AI-generated test suites, this countercheck is especially valuable, because mutation testing reliably exposes padding tests. A suite with 95 percent line coverage but only 40 percent mutation score clearly shows that a large share of the assertions don't perform a genuine check. In practice, it's worth running Infection as a fixed part of the CI pipeline for new, AI-generated test files and defining a minimum score as a merge criterion.

8. Regression traps: when tests fail to catch breaking changes

The practical damage from incorrect assertions usually only shows up months later, when another developer fixes the original bug and, in doing so, breaks a test that was supposedly passing. At that moment, confusion typically follows: the test gets interpreted as a regression, and the actually correct bug fix change either gets rolled back or the test simply gets adjusted without clarifying the deeper cause.

To avoid this trap, a clear naming convention and documentation directly inside the test help: every assertion based on an explicit business rule should carry a comment referencing its source, such as the ticket or the specification. Assertions without that reference should be critically questioned during review before they get added to the suite. This discipline makes later refactoring safer, because it's clear which expectation was deliberately chosen and which one merely reflects the status quo.

// WRONG: AI generated this assertion by running the function once
// and capturing whatever it happened to output, not from a spec.
test('formatPrice rounds down odd cents', () => {
  expect(formatPrice(19.995)).toBe('19.99'); // just mirrors a rounding bug
});

// RIGHT: assertion derived from the documented rounding rule
// (round half up to two decimals), independent of current output.
test('formatPrice rounds half up to two decimals', () => {
  expect(formatPrice(19.995)).toBe('20.00');
});

9. AI-generated tests compared side by side: good vs. bad

The following overview uses typical situations from Magento practice to show how problematic tests oriented toward the current state differ from tests that actually verify the intended behavior.

In all five examples, the difference lies not in the test syntax but in the underlying starting point: was the assertion derived from observed behavior or from the specification? This distinction cannot be automated; it requires a deliberate, brief review of every generated test by a human with domain knowledge of the actual requirement.

10. Summary

AI-generated tests with Claude Code are a powerful starting point, not a finished result. They save time on boilerplate, mocks, and fixtures, but they don't replace the business decision about what a test is actually supposed to verify. The biggest risk is that generated assertions often confirm the current, possibly buggy code state instead of protecting the actual specification, a pattern that only becomes visible when a later bug fix change breaks the test.

Anyone who systematically evaluates AI-generated tests checks coverage numbers critically, uses mutation testing as a countercheck, and documents the business source of every non-trivial assertion. That turns a quick first draft into a resilient test suite that actually protects against regressions, instead of merely faking a green CI pipeline.

11. FAQ: Generating Tests with AI