Making Database Seeding for E2E Tests Reproducible

1. Why reproducible seeding is the foundation of stable E2E runs

Most discussions about flaky tests revolve around selectors, timeouts, and network latency. In practice, the actual root cause is often the database state: one test expects a cart with exactly two items but finds three, because a previous run didn't clean up completely. Another test looks for a product with a specific SKU that doesn't exist because an earlier import failed. This class of failure has nothing to do with the actual test logic and everything to do with not knowing the database's starting state before the run.

Reproducible seeding solves exactly this problem: before every test run, the database is put into an exactly defined, known state, regardless of what previous runs, manual interventions, or parallel processes changed. For Magento stores with a complex data model, catalog, customer accounts, orders, stock, and price rules, this isn't a side detail, it's the precondition for any reliable statement about passing or failing tests. Without this foundation, every E2E suite is a guessing game whose outcome depends on whichever test happened to run last.

2. Seeding approaches compared: API, CLI, and direct DB fixtures

There are three fundamental ways to put a Magento store into a known state before a test run. The first goes through the application's REST or GraphQL API: a setup script creates customers, products, and orders through the same endpoints the application itself uses. This guarantees the generated data always matches the current schema, but costs at least one HTTP roundtrip per entity and becomes noticeably slow with hundreds of fixtures.

The second way uses Magento's own CLI commands, such as bin/magento sampledata:deploy or custom, project-specific console commands that internally run the same business logic as the API but without HTTP overhead. The third way bypasses Magento's application layer entirely and writes test data directly to the database via SQL. This is by far the fastest, since it skips the ORM, the event system, and indexer logic, but requires exact knowledge of the schema and carries the risk of breaking referential integrity or cache invalidation if not done carefully.

-- Direct fixture insert: fastest seeding path, bypasses Magento's ORM and event system
START TRANSACTION;

INSERT INTO customer_entity (entity_id, email, group_id, website_id, is_active)
VALUES (900001, 'e2e.customer@mironsoft.test', 1, 1, 1);

INSERT INTO customer_address_entity (entity_id, parent_id, city, postcode, country_id)
VALUES (900001, 900001, 'Munich', '80331', 'DE');

INSERT INTO cataloginventory_stock_item (product_id, stock_id, qty, is_in_stock)
VALUES (9001, 1, 0, 0); -- out-of-stock fixture for negative-path tests

COMMIT;

In practice, robust setups combine all three approaches by purpose: the API for realistic edge cases with business logic, CLI for repeatable, versioned standard scenarios, and direct SQL for the bulk of fixture data where speed matters more than realism.

3. Magento-specific: sample data vs. custom fixtures for realistic scenarios

bin/magento sampledata:deploy installs Magento's official sample catalog with several hundred products, categories, and attributes. The command is handy for a quick demo environment or to verify a fresh install's basic functionality, but unsuitable as the sole data source for E2E tests: the sample data is generic, contains no project-specific edge cases like out-of-stock products, locked customer accounts, or complex price rules, and changes with every Magento release, breaking tests that rely on specific SKUs or names.

Custom fixtures tailored to the actual test scenarios are therefore the more robust foundation. A dedicated CLI command, invoked with a scenario parameter, can produce exactly the data constellation a given test case needs, such as a product with zero stock to test the out-of-stock state in the cart. Such fixtures live versioned in the same repository as the test code and are never overwritten by a Magento update.

#!/usr/bin/env bash
# seed-e2e.sh: reset and seed a known Magento state for E2E runs
set -euo pipefail

bin/magento maintenance:enable
bin/magento setup:db-schema:upgrade
bin/magento setup:db-data:upgrade

# Deploy Magento's own sample catalog (broad, not tailored to test cases)
bin/magento sampledata:deploy
bin/magento setup:upgrade

# Layer targeted, test-specific fixtures on top via a custom CLI command
bin/magento mironsoft:e2e:seed --scenario=checkout-happy-path
bin/magento mironsoft:e2e:seed --scenario=out-of-stock-product

bin/magento indexer:reindex
bin/magento cache:flush
bin/magento maintenance:disable

4. Seeding via setup:upgrade and Data Patches: repeatable and versioned

Magento's Data Patch mechanism, introduced in Magento 2.3, provides a declarative, versioned way to ship seed data through bin/magento setup:upgrade. Any class implementing DataPatchInterface and placed in a module's Setup/Patch/Data directory is automatically discovered, executed exactly once per environment, and tracked in the patch_list table. For E2E fixtures, this means: a dedicated fixture module, enabled only in test and staging environments, can roll out fixed, known test data through the same mechanism Magento uses for production migrations.

The decisive advantage over an ad hoc script is idempotency and traceability: a Data Patch is guaranteed to run only once, its execution is explicitly controllable via getDependencies and getAliases, and changes to seed data get reviewed in the same pull request as any other code change. It's important to strictly separate fixture patches from real schema migrations, for example through a dedicated module like Mironsoft_E2eFixtures that is never installed in production.

<?php
declare(strict_types=1);

namespace Mironsoft\E2eFixtures\Setup\Patch\Data;

use Magento\Framework\Setup\Patch\DataPatchInterface;
use Magento\Framework\Setup\ModuleDataSetupInterface;
use Magento\Catalog\Api\ProductRepositoryInterface;
use Magento\Catalog\Api\Data\ProductInterfaceFactory;

/**
 * Seeds a deterministic out-of-stock product fixture for E2E checkout tests.
 * Runs exactly once per environment, tracked in the patch_list table.
 */
class SeedOutOfStockProductPatch implements DataPatchInterface
{
    /**
     * @param ModuleDataSetupInterface $moduleDataSetup Setup connection resource.
     * @param ProductRepositoryInterface $productRepository Product persistence service.
     * @param ProductInterfaceFactory $productFactory Factory for new product instances.
     */
    public function __construct(
        private readonly ModuleDataSetupInterface $moduleDataSetup,
        private readonly ProductRepositoryInterface $productRepository,
        private readonly ProductInterfaceFactory $productFactory
    ) {
    }

    /**
     * Applies the data patch.
     *
     * @return void
     */
    public function apply(): void
    {
        $this->moduleDataSetup->getConnection()->startSetup();

        $product = $this->productFactory->create();
        $product->setSku('E2E-OUT-OF-STOCK')
            ->setName('E2E Out Of Stock Fixture')
            ->setPrice(19.90)
            ->setStatus(1)
            ->setTypeId('simple')
            ->setStockData(['qty' => 0, 'is_in_stock' => 0]);

        $this->productRepository->save($product);

        $this->moduleDataSetup->getConnection()->endSetup();
    }

    /**
     * @return array<int, string>
     */
    public static function getDependencies(): array
    {
        return [];
    }

    /**
     * @return array<int, string>
     */
    public function getAliases(): array
    {
        return [];
    }
}

5. Keeping seed scripts fast: parallelization and targeted seeds

A full catalog import with thousands of products easily takes several minutes, which is unacceptable for every single CI run. The most important lever is therefore targeted rather than full seeding: create only the entities a given test run actually needs, instead of reflexively rebuilding the entire reference catalog. A test that only exercises the checkout flow rarely needs more than two or three products, a customer, and a shipping method.

Direct SQL is significantly faster than going through the ORM or API for large volumes, since it skips PHP object instantiation, event observers, and validation entirely. For bulk inserts, it's also worth setting indexers to "update on save" mode or disabling them entirely during seeding and running bin/magento indexer:reindex once afterward, instead of triggering reindexing on every single insert. Parallelizing across multiple workers speeds up seeding further, as long as each worker writes to its own isolated data range and no shared sequences or auto-increment ranges collide.

6. Resetting state without a full re-seed: transactions and savepoints

A full re-seed between every single test is often needlessly expensive. The fastest alternative is running each test inside a database transaction that is always rolled back at the end, regardless of the test outcome. MySQL savepoints even allow nested rollback points within a single test, for example to reset a partial state without losing the entire outer test context. This method eliminates cleanup code entirely and is orders of magnitude faster than any re-seed.

The limit of this approach lies in Magento's own behavior: certain operations, such as indexer runs, cache invalidation, or email dispatch, happen outside the database transaction and aren't undone by a rollback. For pure browser-driven E2E tests against a running application, the transaction approach is therefore only partially suitable, since the application server opens and closes the transaction itself, not the test. A targeted reset is often more practical: truncate and repopulate only the tables a given test actually modified, instead of rebuilding the entire database.

7. Snapshot and restore strategies at the database level

Where transactions run into the limits of Magento's architecture, database snapshots provide the most reliable foundation for fast resets. A mysqldump of the already-seeded baseline database, compressed and stored as an artifact, can be restored in a few seconds, considerably faster than any repeated seeding run through the API or CLI. It's important to create the dump with --single-transaction to guarantee a consistent snapshot without locking InnoDB tables.

For even larger databases, filesystem-level snapshots are more efficient than logical dumps. Docker volume snapshots, for instance via a pre-built image with an already-initialized data directory, or tools like Percona XtraBackup for physical copies of a running MySQL instance, cut restore time from seconds to milliseconds because no SQL needs to be re-parsed and executed. The trade-off: physical snapshots are tied to the MySQL version and configuration, making them especially well suited to homogeneous CI environments with a fixed database version.

#!/usr/bin/env bash
# db-snapshot.sh: create and restore a fast-loading E2E baseline snapshot
set -euo pipefail

readonly DB_NAME="magento_e2e"
readonly SNAPSHOT_DIR="./var/e2e-snapshots"
readonly SNAPSHOT_FILE="${SNAPSHOT_DIR}/baseline.sql.gz"

create_snapshot() {
  mkdir -p "$SNAPSHOT_DIR"
  mysqldump --single-transaction --quick --no-tablespaces \
    "$DB_NAME" | gzip -1 > "$SNAPSHOT_FILE"
  echo "[OK] Snapshot written to $SNAPSHOT_FILE"
}

restore_snapshot() {
  [[ -f "$SNAPSHOT_FILE" ]] || { echo "[ERROR] No snapshot found" >&2; exit 1; }
  mysql -e "DROP DATABASE IF EXISTS $DB_NAME; CREATE DATABASE $DB_NAME;"
  gunzip -c "$SNAPSHOT_FILE" | mysql "$DB_NAME"
  echo "[OK] Restored $DB_NAME from snapshot in a few seconds"
}

case "${1:-restore}" in
  create) create_snapshot ;;
  restore) restore_snapshot ;;
  *) echo "Usage: $0 {create|restore}" >&2; exit 1 ;;
esac

8. CI integration: seeding in pipelines, caching, and parallelization

In a CI pipeline, every second counts because it multiplies with every commit. The most effective lever is reusing an already-seeded baseline database as a cache artifact between pipeline runs, instead of rebuilding it from scratch every time. If the database schema or fixture definitions haven't changed, a previously created snapshot is restored directly. The cache key is typically based on a hash of the migration and fixture files, so a real schema change automatically forces a fresh snapshot.

For parallel test execution across multiple CI runners, each runner needs either its own isolated database instance or at least its own namespace within a shared instance, so parallel workers don't overwrite each other. Container-based CI systems solve this elegantly, with each job starting its own database container from the same pre-seeded image, making parallelization possible without additional isolation logic in the test code.

# .github/workflows/e2e.yml: cache a pre-seeded DB snapshot across runners
jobs:
  e2e:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4

      - name: Restore cached DB snapshot
        uses: actions/cache@v4
        with:
          path: var/e2e-snapshots/baseline.sql.gz
          key: db-snapshot-${{ hashFiles('setup/Patch/Data/**') }}

      - name: Seed database from snapshot or rebuild on cache miss
        run: |
          if [ ! -f var/e2e-snapshots/baseline.sql.gz ]; then
            bin/magento setup:upgrade
            bash bin/db-snapshot.sh create
          fi
          bash bin/db-snapshot.sh restore

      - name: Run E2E shard
        run: npx playwright test --shard=${{ matrix.shard }}/4

9. Seeding strategies compared

Each of the four common seeding approaches has a clear sweet spot, and the choice has a direct impact on runtime, reliability, and CI-friendliness of the entire suite.

In practice, a combination delivers the best results: a versioned baseline state via Data Patches, frozen as a snapshot for fast resets between test runs, complemented by API-driven seeding for the few edge cases that need to exercise real business logic.

10. Summary

Reproducible database seeding is the foundation of every stable E2E suite, not a downstream detail. API, CLI, and direct SQL seeding solve different problems: the API for realism, CLI for repeatable standard scenarios, SQL for maximum speed with large data volumes. Magento's own Data Patch mechanism provides a versioned, idempotent foundation for this, going through the same review process as any other code change instead of rotting away in a separate, unversioned setup script.

The second decisive building block is a fast reset between test runs: transactions with rollback work as long as Magento doesn't trigger side effects outside the database, snapshot restores are the more robust alternative for genuine browser-driven E2E tests. Combined with caching and parallelization in the CI pipeline, runtime per test run drops drastically while reproducibility rises at the same time, because every run starts from exactly the same known state.

11. FAQ: Database Seeding for E2E Tests