Doctrine Batch Processing for Large Datasets

1. Why naive entity iteration fails on millions of rows

Doctrine's UnitOfWork tracks every loaded entity to detect changes and generate the matching SQL statements on the next flush(). This tracking is the core of Doctrine's change tracking functionality, but it comes at a price: every loaded entity stays in memory as long as the EntityManager exists. In a loop over ten million rows using $query->getResult(), Doctrine attempts to hydrate all ten million entities simultaneously and track them in the UnitOfWork, which inevitably blows through the PHP memory limit.

This behavior surprises many developers, because smaller datasets in testing work fine and the problem only becomes visible in production with real data volumes. Doctrine batch processing refers to the entirety of techniques that solve this fundamental problem: processing data in controlled, smaller portions, resetting the UnitOfWork regularly, and where possible, bypassing entity hydration entirely.

The core trade off in Doctrine batch processing is always the same: full ORM functionality with events, lifecycle callbacks and change tracking costs memory per entity. Whoever wants to save that memory must give up some of that functionality. The following sections show the different points along that scale, from full ORM convenience with periodic resets, all the way to plain SQL without any hydration.

2. The base pattern: EntityManager::clear() and batch size

The fundamental pattern for Doctrine batch processing combines a fixed batch size with periodic calls to $entityManager->clear(). Calling clear() completely detaches the UnitOfWork from all tracked entities, allowing PHP's garbage collector to free the previously occupied memory. Without this periodic reset, memory usage grows linearly with the number of processed rows, regardless of whether flush() is called in between or not.

<?php

declare(strict_types=1);

namespace App\Service;

use Doctrine\ORM\EntityManagerInterface;

final readonly class ProductPriceRecalculator
{
    private const int BATCH_SIZE = 500;

    public function __construct(
        private EntityManagerInterface $entityManager,
    ) {
    }

    public function recalculateAll(): int
    {
        $query = $this->entityManager
            ->createQuery('SELECT p FROM App\Entity\Product p ORDER BY p.id ASC');

        $processed = 0;

        foreach ($query->toIterable() as $product) {
            $product->recalculatePrice();
            ++$processed;

            if ($processed % self::BATCH_SIZE === 0) {
                $this->entityManager->flush();
                // Detach all managed entities to free memory held by the UnitOfWork
                $this->entityManager->clear();
            }
        }

        // Flush the remainder that did not reach a full batch
        $this->entityManager->flush();

        return $processed;
    }
}

An important pitfall with this pattern: after clear(), all previously loaded entities are detached and must not be referenced anymore without reloading them via find(). Anyone holding a reference to an already cleared entity outside the loop and trying to persist it will get an ORMInvalidArgumentException. With Doctrine batch processing using clear(), the entire processing state per batch must be self contained.

3. Iterable queries with toIterable()

The second building block for efficient Doctrine batch processing, already used in the previous example, is toIterable() instead of getResult(). While getResult() fetches the entire result set from the database in one go and immediately hydrates all rows, toIterable() uses a server side cursor and delivers entities one at a time as iteration progresses. Memory usage for the pure data transfer stays constant this way, regardless of the total number of rows.

Important to understand: toIterable() alone does not solve the UnitOfWork's memory problem, it only solves the result set buffer problem. Without combining it with periodic clear(), the UnitOfWork still grows linearly, because every entity read through the cursor keeps being tracked until it is explicitly removed. Effective Doctrine batch processing always needs both techniques together, the cursor for data transfer and periodic clearing for the UnitOfWork.

4. Bulk updates with DQL instead of entity hydration

For pure data changes that do not need lifecycle callbacks or entity events, a DQL UPDATE or DELETE statement is significantly more efficient than any form of entity hydration. Such a bulk statement is translated directly into SQL and executed on the database, without loading a single entity into PHP.

<?php

declare(strict_types=1);

namespace App\Service;

use Doctrine\ORM\EntityManagerInterface;

final readonly class InactiveProductArchiver
{
    public function __construct(
        private EntityManagerInterface $entityManager,
    ) {
    }

    public function archiveInactiveProducts(\DateTimeImmutable $cutoff): int
    {
        // Bulk DQL UPDATE — no entities are loaded into memory at all
        $query = $this->entityManager->createQuery(
            'UPDATE App\Entity\Product p
             SET p.status = :archived
             WHERE p.lastOrderedAt < :cutoff AND p.status != :archived'
        );

        $query->setParameter('archived', 'archived');
        $query->setParameter('cutoff', $cutoff);

        return $query->execute();
    }
}

The decisive limitation of bulk DQL in the context of Doctrine batch processing: Doctrine events like preUpdate and postUpdate are not triggered by bulk statements, because no entity objects pass through the regular persistence cycle. If application logic depends on these events, for example audit logging or cache invalidation through lifecycle callbacks, that logic must be executed separately and explicitly after the bulk update.

5. Offloading batch processing with Messenger

For data volumes where even a batched run exceeds the time limit of a single HTTP request or cron job, the right answer is to split the Doctrine batch processing into several independent, asynchronous messages using Symfony's Messenger component. An initial handler determines the total number of records to process and distributes the work into chunks, each chunk as its own message, processed independently by a worker.

<?php

declare(strict_types=1);

namespace App\MessageHandler;

use App\Message\RecalculatePriceChunk;
use Doctrine\ORM\EntityManagerInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;

#[AsMessageHandler]
final readonly class RecalculatePriceChunkHandler
{
    public function __construct(
        private EntityManagerInterface $entityManager,
    ) {
    }

    public function __invoke(RecalculatePriceChunk $message): void
    {
        $query = $this->entityManager->createQuery(
            'SELECT p FROM App\Entity\Product p WHERE p.id BETWEEN :from AND :to'
        );
        $query->setParameter('from', $message->fromId);
        $query->setParameter('to', $message->toId);

        foreach ($query->toIterable() as $product) {
            $product->recalculatePrice();
        }

        // One flush and clear per chunk, each chunk handled by an isolated worker call
        $this->entityManager->flush();
        $this->entityManager->clear();
    }
}

This split brings two benefits for Doctrine batch processing: every chunk runs in its own short lived worker invocation, so even a memory leak in one chunk has no effect on the others. In addition, processing can be parallelized horizontally by having multiple Messenger workers handle different chunks simultaneously, which shortens the total runtime significantly given sufficient CPU cores and database capacity.

6. Native SQL for maximum bulk insert performance

For inserting very large volumes of data, for example during the initial import of a CSV file with millions of rows, even a batched persist()/flush() pattern through the EntityManager is slower than a native multi row INSERT statement. Every Doctrine entity goes through change tracking, validation and event dispatching when being persisted, overhead that is not needed for a pure mass import.

For such cases, efficient Doctrine batch processing uses Doctrine DBAL's native Connection API directly, with prepared multi row INSERT statements in batches of a few hundred rows per statement. This approach entirely bypasses entity hydration and ORM overhead, but requires that all validation logic that would otherwise live in the entity constructor or lifecycle callbacks is manually reproduced in PHP before the insert, since the database itself only performs basic constraint checks.

7. Progress tracking and error handling

A long running batch process without progress reporting is an operational black box risk: nobody knows whether the process is still running, stuck, or already finished. Symfony's console component provides the ProgressBar class as a simple way to make progress visible in CLI driven Doctrine batch processing, including an estimated remaining runtime based on the processing speed so far.

Error handling in batch processes needs a deliberate decision: should a single faulty record abort the entire batch, or should it be skipped and logged while the rest continues? For most production Doctrine batch processing scenarios, the second option is more robust, with a try catch per individual record inside the batch loop and a separate error list evaluated at the end of the run, instead of losing all progress on a single broken record.

8. Memory profiling with memory_get_usage() and Blackfire

Without measurement, any optimization of Doctrine batch processing remains a guess. memory_get_usage(true) returns the memory actually allocated by the PHP process and can be plugged directly into the batch loop to log the memory trend over the course of processing. Steadily rising memory usage despite periodic clear() almost always points to a reference somewhere in the code holding onto cleared entity material, for example in an array outside the loop.

For deeper analysis, Blackfire's memory profiling delivers detailed call graphs that show exactly which function allocates how much memory. With Doctrine batch processing, such profiles frequently reveal that Doctrine's own second level cache, if enabled, also accumulates memory over the entire runtime of the batch process, an effect that should be considered separately when enabling the second level cache for batch commands.

9. Batch approaches compared directly

The following table contrasts the techniques presented for Doctrine batch processing and shows which approach is the right choice for which data volume and which requirements on ORM functionality.

The table makes the central trade off of Doctrine batch processing visible: the more ORM convenience is retained, the more memory and time processing costs. The right choice depends on whether per record business logic is needed or whether a pure, event free data change is sufficient.

10. Summary

Doctrine batch processing solves a structural problem of the UnitOfWork: without periodic resetting, Doctrine tracks every loaded entity in memory, which inevitably blows the memory limit on large datasets. The combination of toIterable() for constant result set memory and periodic flush() plus clear() for the UnitOfWork is the base pattern for any business logic that must run per entity.

Where no entity events are needed, bulk DQL statements or native SQL inserts are significantly more efficient, because they bypass hydration entirely. For data volumes that take too long even when batched for a single time window, Messenger chunking distributes the work across several independent, parallelizable worker calls. Memory profiling with memory_get_usage() and Blackfire makes visible whether the chosen strategy actually produces constant memory usage, before the process runs against real data volumes in production.

11. FAQ: Doctrine Batch Processing for Large Datasets