When Not to Trust AI Code Suggestions

1. Why AI code suggestions deserve baseline skepticism

Claude and comparable assistants generate code by producing statistically plausible continuations drawn from enormous amounts of training data, not by formally proving correctness. The result often reads like code written by an experienced developer: cleanly formatted, with sensible variable names, with comments that sound logical. That persuasive quality is exactly the problem. Review habits calibrated for human-written code look for typos, inconsistent style, and obvious omissions. AI-generated code shows these signals less often, yet can still be wrong at a decisive point, for example a flipped condition or a missing edge case.

A useful lens for assessment combines blast radius, reversibility, and detectability of a potential mistake. A wrong CSS class is cheap to spot and fix. A wrong SQL statement that runs against production data can cause damage that cannot be undone before anyone even notices the mistake. This article is not an argument against using AI assistants, but an argument for a review depth calibrated to actual risk rather than to the convenience of simply accepting generated code as is.

2. Security-critical code: authentication, encryption, access control

In authentication, encryption, and access control, even small logic errors carry weight because they typically surface only through deliberate misuse, not through normal usage. AI assistants occasionally suggest outdated cryptographic approaches, such as unsalted password hashes, or implement permission checks with a flipped condition that happens to work in the test case but lets through one role too many. Such errors frequently pass unit tests anyway, because the tests share the same blind spot as the generated code.

That is why a stricter rule applies to auth and crypto code than to the rest of the codebase: a passing test proves that the tested path works, not that the code is secure. Static security analysis with tools like Semgrep, combined with PHPStan rules at a high level, catches many known patterns automatically. More important is a second look from someone with security experience, regardless of whether the code came from a human or from Claude. That review should be a mandatory part of the merge process, not an optional extra.

#!/usr/bin/env bash
# Run before merging any AI-suggested auth/crypto code, never skip this step
set -euo pipefail

# Static security analysis with Semgrep security rulesets
semgrep --config p/security-audit --config p/php src/app/code/Mironsoft/Auth

# Scan for accidentally hardcoded secrets or weak hashing calls
gitleaks detect --source . --no-git -v
grep -rn "md5(\|sha1(\|ECB" src/app/code/ --include="*.php" || true

# "It passed the tests" is not sufficient evidence for auth-critical code
bin/analyse app/code/Mironsoft/Auth --level=8

3. Payment processing and payment integrations: zero tolerance for guesswork

Payment code has two properties that make it especially unforgiving of mistakes: it moves real money, and many failure paths are invisible without domain expertise. An AI assistant generating a Stripe or PayPal webhook handler often produces a technically functional happy path that accepts incoming data without verification. Without signature verification on the webhook, anyone who knows the URL can forge arbitrary payment events. Without an idempotency key, a retried webhook call leads to double crediting or double fulfillment.

Such gaps do not surface in superficial tests, because the test case rarely simulates a manipulated or repeated request. Currency conversion rounding errors and race conditions between inventory reservation and payment capture compound the risk further. Any AI-generated payment code needs to be tested against the payment provider's official sandbox and current API documentation, not just against self-written mock data, since training data ages and API details change faster than a model relearns them.

// WRONG: AI-suggested webhook handler trusts the payload without verification
app.post('/webhook/stripe', express.json(), (req, res) => {
  const event = req.body;
  if (event.type === 'payment_intent.succeeded') {
    markOrderAsPaid(event.data.object.metadata.orderId);
  }
  res.sendStatus(200);
});

// RIGHT: verify the signature before trusting any payload data
app.post('/webhook/stripe', express.raw({ type: 'application/json' }), (req, res) => {
  let event;
  try {
    event = stripe.webhooks.constructEvent(
      req.body,
      req.headers['stripe-signature'],
      process.env.STRIPE_WEBHOOK_SECRET
    );
  } catch (err) {
    // Reject anything that cannot be proven to come from Stripe
    return res.status(400).send(`Webhook signature verification failed: ${err.message}`);
  }

  if (event.type === 'payment_intent.succeeded') {
    // Idempotency check prevents double-fulfillment on retried webhooks
    markOrderAsPaidIdempotent(event.id, event.data.object.metadata.orderId);
  }
  res.sendStatus(200);
});

4. Data migrations and schema changes: the point of no return

Data migrations differ from most other code changes in that a mistake cannot simply be corrected by a new deployment. Once an UPDATE statement without a sufficient WHERE condition has run against the production database, the original values are gone unless a current backup exists. AI-generated migration scripts often look syntactically complete but typically omit transaction boundaries, batching for large tables, and a dry-run mode, because these aspects are demonstrated less often in training material than plain update logic.

On a large InnoDB table with millions of rows, a single, unbatched UPDATE can also lock the table for the duration of execution and disrupt live store operation. Every AI-suggested migration should therefore first be tested against a full, production-like copy of the database, not against a handful of fixture rows. A dry run that only counts and logs what would happen belongs before every write.

# WRONG: AI-suggested one-shot update, no batching, no dry-run, no transaction
def migrate_customer_addresses(connection):
    cursor = connection.cursor()
    cursor.execute("UPDATE customer_address SET country_id = 'DE' WHERE country_id = ''")
    connection.commit()

# RIGHT: batched, reversible, verifiable before it touches production
def migrate_customer_addresses(connection, dry_run=True, batch_size=500):
    cursor = connection.cursor()
    cursor.execute(
        "SELECT entity_id FROM customer_address WHERE country_id = '' LIMIT %s",
        (batch_size,)
    )
    rows = cursor.fetchall()
    print(f"Would update {len(rows)} rows in this batch (dry_run={dry_run})")

    if dry_run:
        return len(rows)

    connection.autocommit = False
    try:
        ids = [row[0] for row in rows]
        cursor.executemany(
            "UPDATE customer_address SET country_id = 'DE' WHERE entity_id = %s",
            [(i,) for i in ids]
        )
        connection.commit()
    except Exception:
        connection.rollback()
        raise
    return len(rows)

5. Code that touches production data: why staging is not enough

Migrations are not the only risk. Every script, every cron job, and every one-off cleanup command that runs with write access to the production database or to customer files carries the same concern. The decisive question before any execution is not whether the code looks correct, but what happens in the worst case if it isn't. An AI-generated cleanup script meant to remove orphaned records can, given a misunderstood relationship between tables, also catch active customer data. Staging environments rarely represent this risk realistically, since they usually hold smaller data volumes, different edge cases, and fewer genuine historical artifacts than production does.

A practical approach uses three stages. First a purely read-only pass that logs exactly which records would be affected. Then an explicit human confirmation that checks the actual hit count against expectations. Only after that a write run, ideally using a database account with only the minimally necessary privileges. This principle of least privilege limits the damage if the earlier checks still missed something.

6. Confident-sounding explanations as a warning sign

A somewhat counterintuitive but important pattern: the linguistic confidence of an AI explanation says nothing about the actual correctness of the code. A language model produces sentences like "this implementation is now fully thread-safe and handles all edge cases" with the same fluent certainty regardless of whether the claim is true. Human experts typically express uncertainty in language, saying things like "we would still need to test this" or "I'm not entirely sure about this part." A language model often lacks that calibrated hesitation, because the training process rewards fluent, complete-sounding answers, not necessarily correct ones.

The practical consequence: a statement that claims absolute completeness or freedom from bugs should prompt closer inspection rather than relief. It helps to explicitly ask the assistant which assumptions it made and which edge cases it deliberately did not handle. That follow-up question often uncovers gaps hidden inside the original, smooth-sounding explanation, because the model does not volunteer incompleteness on its own unless explicitly asked.

7. Spotting hallucinated APIs, packages, and versions

Language models occasionally invent method names, classes, or entire packages that sound plausible but do not exist, a phenomenon now known as package hallucination or slopsquatting. In a Magento context this shows up as a call to an interface method that does not exist in the actually installed version, or as a suggestion to install a Composer package whose name sounds reasonable but is not registered on Packagist. Attackers exploit exactly this gap by preemptively registering frequently hallucinated package names themselves and filling them with malicious code.

A subtler problem is version drift: training data has a cutoff date, and between that cutoff and actual deployment, API methods may have been removed, renamed, or changed in behavior. Every class or method the AI references should therefore be checked against the code actually installed in the vendor directory or against the current official documentation before it is used, especially for rapidly evolving frameworks.

{
  "require": {
    "php": "^8.4",
    "magento/framework": "^103.0",
    "mironsoft/core": "^2.0",
    "magento/module-secure-payment-validator": "^1.0"
  },
  "_note_unverified_dependency": "magento/module-secure-payment-validator was suggested by an AI assistant and does not exist on Packagist as of this writing. Verify every unfamiliar package name against the official registry before running composer require."
}

8. When to fall back to manual implementation entirely

There is a point where the sensible response is no longer "review more carefully" but "write it yourself." That is the case when an operation is irreversible and no tested rollback exists, when the business logic requires deep domain or legal knowledge, for example tax calculations or GDPR-compliant data deletion, or when money, personal data, and production access are all involved at the same time. In such cases, the time a thorough verification of the AI suggestion costs can exceed the time a manual implementation would take, erasing the supposed speed advantage entirely.

A simple test helps with the decision: can you explain, in your own words, why the code is correct to a colleague without relying on the AI's explanation? If not, you lack the understanding needed to take responsibility for the code, regardless of who wrote it. If the assistant also contradicts itself within the same conversation, for instance when a later explanation quietly revises an earlier assurance, that is an additional signal to rethink the whole approach from scratch yourself.

#!/usr/bin/env bash
# Manual verification checklist before running any AI-suggested migration on production
set -euo pipefail

# 1. Take a verified, restorable backup first, always
bin/mysqldump --single-transaction magento > "backup-$(date +%Y%m%d-%H%M%S).sql"

# 2. Run the migration in dry-run mode against a full-size copy, not a fixture
bin/magento migration:run --dry-run --env=staging-full-copy

# 3. Check the actual row count and EXPLAIN plan, not just "no errors"
bin/mysql magento -e "EXPLAIN UPDATE customer_address SET country_id='DE' WHERE country_id='';"

# 4. Roll out in small batches with a kill switch, not one giant statement
bin/magento migration:run --batch-size=500 --max-batches=1 --env=production
echo "Verify results manually before running the next batch"

9. Risk categories compared directly

Not every code change deserves the same review depth. The overview below sorts typical task categories by how much trust an AI suggestion deserves there and which practice actually secures the corresponding risk.

The table shows that trusting AI code is not a binary decision, but one that follows the blast radius and reversibility of the specific task. A team that recognizes these categories and consistently treats them differently does not lose the speed advantage of AI assistants on low-risk work, while still avoiding the costly mistakes in areas where a single overlooked edge case causes real damage.

10. Summary

AI code suggestions do not carry uniform risk; they deserve different levels of trust depending on the blast radius, reversibility, and detectability of possible mistakes. In authentication, encryption, and access control, logic errors often pass tests anyway, which is why dedicated security analysis remains mandatory. Payment processing demands testing against real sandbox environments, because missing signature checks or idempotency gaps stay invisible in everyday testing. Data migrations and any code touching production data need dry runs, backups, and batch processing, because mistakes there are rarely reversible.

Two patterns deserve special attention: a confident-sounding AI explanation is not proof of correctness, often just a sign of linguistic fluency independent of the actual content. And hallucinated methods, classes, or packages can only be reliably caught by checking against the code actually installed and the current official documentation. Where verifying a suggestion takes longer than implementing it yourself, or where money, personal data, and production access all come together, manual implementation by a human remains the more reliable choice.

11. FAQ: When Not to Trust AI Code Suggestions