Generators and yield: Processing Large Datasets Memory-Efficiently

1. What Generators Are and Why Lazy Evaluation Matters

A generator in PHP is a function that, instead of returning a single value, produces a sequence of values, one at a time, only when the caller actually asks for the next one. As soon as a function contains the keyword yield anywhere in its body, PHP automatically turns it into a generator. The return value is not an array and not a single value, but an instance of the built-in Generator class, which implements the Iterator interface internally and therefore behaves like an ordinary collection in any foreach loop.

The decisive difference from an array lies in the evaluation strategy. A function that returns a complete array must compute and store every single element in memory before the caller sees even the first one. That is called eager evaluation. A generator, on the other hand, only computes the next element when it is actually requested, and holds no growing buffer in memory between two values. This lazy evaluation is the core reason why generators and yield matter so much for large datasets. Memory usage stays nearly constant throughout the whole processing run, regardless of whether ten or ten million elements are being processed.

The difference becomes obvious in a direct comparison: a function that reads an entire file into an array of lines with file(), versus a generator function that delivers one line at a time using fgets() and yield. The code below measures both variants with memory_get_peak_usage() against the same file.

<?php

declare(strict_types=1);

namespace App\Import;

use Generator;
use RuntimeException;

/**
 * Eager approach: loads the entire file into memory as an array of lines.
 */
function readLinesEager(string $path): array
{
    $lines = file($path, FILE_IGNORE_NEW_LINES);
    if ($lines === false) {
        throw new RuntimeException("Cannot read file: {$path}");
    }

    return $lines;
}

/**
 * Lazy approach: yields one line at a time, memory footprint stays constant.
 */
function readLinesLazy(string $path): Generator
{
    $handle = fopen($path, 'rb');
    if ($handle === false) {
        throw new RuntimeException("Cannot open file: {$path}");
    }

    try {
        while (($line = fgets($handle)) !== false) {
            yield rtrim($line, "\r\n");
        }
    } finally {
        fclose($handle);
    }
}

$file = __DIR__ . '/large-export.csv'; // 500,000 lines

$before = memory_get_peak_usage(true);
$eagerLines = readLinesEager($file);
$eagerPeak = memory_get_peak_usage(true) - $before;
unset($eagerLines);

$before = memory_get_peak_usage(true);
foreach (readLinesLazy($file) as $line) {
    // process one line at a time, nothing accumulates
}
$lazyPeak = memory_get_peak_usage(true) - $before;

printf("Eager: %d MB, Lazy: %d MB\n", (int) ($eagerPeak / 1_048_576), (int) ($lazyPeak / 1_048_576));
// Typical result on a 500,000-line CSV file: Eager approx 210 MB, Lazy approx 2 MB

On a test file with 500,000 lines, the peak memory usage of the eager variant sits well over 200 MB, while the generator-based variant stays in the low single-digit MB range throughout, because only one line is held in memory at any given time. That is the practical reason why generators matter so much for imports, exports, and ETL jobs in PHP.

2. The Mechanics of yield: Pausing and Resuming

When a generator function is called, its code does not run at all at first. PHP merely creates a Generator object and remembers the function's execution context. Only when the caller requests a value for the first time, whether through foreach, current(), or next(), does the function actually start running, up to the first yield statement. There, execution pauses, the current value is handed back to the caller, and the function's entire state, all local variables, the position in the code, and the call structure, is preserved until execution is resumed.

yield is not a plain statement but an expression with its own value. $received = yield $value; is valid code. $value is delivered to the caller, execution pauses, and once the caller calls send(), the value passed in becomes the result of the yield expression and gets stored in $received. This detail is the foundation for generators also acting as lightweight coroutines that not only produce values but also accept values.

Besides plain values, yield also supports key-value pairs through the syntax yield $key => $value. This lets you build a generator that behaves exactly like an associative array when iterated with foreach ($generator as $key => $value), without a full associative array ever existing in memory. If no explicit keys are given, PHP automatically assigns sequential integer keys, just like a regular indexed array.

3. yield from: Chaining and Nesting Generators

yield from delegates value production to another generator, an array, or any Traversable object. Instead of manually forwarding every value of a nested generator in a loop, yield from handles that work directly and even forwards keys, return values, and values sent in through send() transparently. This makes it possible to compose small generator functions into larger processing chains without losing the lazy evaluation property.

A classic use case is flattening nested structures, such as a category tree. A recursive function that calls yield from on itself for every nested array ends up producing a flat stream of all leaf values, without the structure ever having to be built as a complete flat array in memory. Each level of the recursion remains its own, independent generator.

<?php

declare(strict_types=1);

namespace App\Tree;

use Generator;

/**
 * Recursively flattens a nested array structure using yield from.
 *
 * @param array<int|string, mixed> $node
 * @return Generator<int|string, mixed>
 */
function flatten(array $node): Generator
{
    foreach ($node as $key => $value) {
        if (is_array($value)) {
            // Delegate to a nested generator call, composing lazily
            yield from flatten($value);
            continue;
        }

        yield $key => $value;
    }
}

$categoryTree = [
    'electronics' => [
        'phones' => ['iphone-15', 'pixel-9'],
        'laptops' => [
            'gaming' => ['rog-strix', 'legion-7'],
            'business' => ['thinkpad-x1'],
        ],
    ],
    'books' => ['fiction', 'non-fiction'],
];

foreach (flatten($categoryTree) as $key => $slug) {
    echo "{$key} => {$slug}\n";
}

It is worth noting that yield from does not produce a new value by itself, it simply forwards the values of the delegated iterable. The number of actual yield points in the call chain stays invisible to the caller, who is just iterating over the outer Generator object.

4. Sending Values In: send() and getReturn()

A generator is not a one-way street. Through Generator::send($value), the caller can push a value into the paused function, and that value becomes the result of the current yield expression. The function then keeps running until the next yield or until it ends, and send() itself returns the next produced value. This pattern turns generators into lightweight coroutines: state, processing, and two-way communication, without threads or external queues.

If a generator function ends with a return statement instead of a final yield, that value does not enter the normal iteration flow, it never shows up in a foreach loop. Instead, it can be retrieved after full execution through Generator::getReturn(). This is useful for delivering, say, a total sum, an error count, or a summary object after every individual value has already been consumed.

<?php

declare(strict_types=1);

namespace App\Coroutine;

use Generator;

/**
 * A generator that consumes values sent into it and accumulates a total,
 * acting as a lightweight coroutine for streaming aggregation.
 *
 * @return Generator<int, float, float|null, float>
 */
function runningTotal(): Generator
{
    $total = 0.0;

    while (true) {
        // yield as an expression: receives the value passed to send()
        $amount = yield $total;

        if ($amount === null) {
            return $total;
        }

        $total += $amount;
    }
}

$totals = runningTotal();
$totals->current(); // primes the generator: runs to the first yield, current value is 0.0

echo $totals->send(100.0) . "\n"; // resumes with $amount = 100.0, prints 100
echo $totals->send(250.0) . "\n"; // prints 350
echo $totals->send(50.0) . "\n";  // prints 400
$totals->send(null);               // resumes with $amount = null, executes return, generator finishes

echo $totals->getReturn() . "\n"; // 400, the final accumulated value

5. Large Datasets: CSV and DB Streaming with Generators

The most practically important use case for generators is processing large datasets row by row without keeping the entire source in memory. Instead of fully reading a CSV file with millions of rows or fully materializing a database result set, a generator function delivers one row, or one record, at a time, while the underlying resource, a file handle or a database cursor, stays open in the background.

To make sure that resource is reliably closed even on early termination, a try/finally block belongs directly inside the generator function. The finally block runs in three cases: when the generator has been fully consumed, when the caller's loop exits early with break, and when the generator reaches garbage collection without being fully iterated. This behavior makes generators safe enough for production resource handling, with no manual cleanup required in the calling code.

<?php

declare(strict_types=1);

namespace App\Import;

use Generator;
use RuntimeException;

/**
 * Streams rows from a large CSV file without loading it into memory.
 */
final class CsvRowStreamer
{
    public function __construct(
        private readonly string $path,
        private readonly string $delimiter = ',',
    ) {
    }

    /**
     * @return Generator<int, array<string, string>>
     */
    public function rows(): Generator
    {
        $handle = fopen($this->path, 'rb');
        if ($handle === false) {
            throw new RuntimeException("Cannot open CSV file: {$this->path}");
        }

        try {
            $header = fgetcsv($handle, 0, $this->delimiter);
            if ($header === false) {
                return;
            }

            while (($row = fgetcsv($handle, 0, $this->delimiter)) !== false) {
                yield array_combine($header, $row);
            }
        } finally {
            // Runs on full consumption, on early break, and on generator destruction
            fclose($handle);
        }
    }
}

$streamer = new CsvRowStreamer(__DIR__ . '/orders-export.csv');

$imported = 0;
foreach ($streamer->rows() as $order) {
    if ((float) $order['total'] > 10_000.0) {
        break; // early exit still triggers the finally block above
    }

    $imported++;
}

echo "Imported {$imported} orders before hitting the threshold\n";

The same pattern works identically for database result sets. A PDO statement with an unbuffered cursor delivers rows one at a time to a generator function, which forwards them, while the connection is only released after the last row or after the finally block runs. This lets you process millions of database rows with constant memory usage as well.

6. Limits of Generators: Forward-Only and Arrays

A generator is fundamentally forward-only. It can only be iterated once, from start to finish. Calling rewind() a second time on a generator that has already started iterating throws an Exception stating that the generator is already running or already finished. A second foreach loop over the same generator also yields no more values, because the internal execution has already reached its end.

The function iterator_to_array() converts a generator into a real array, but doing so cancels out exactly the memory advantage you used the generator for in the first place. There is an additional pitfall: without yield $key => $value using unique keys, values sharing the same automatically assigned integer key overwrite each other, unless the second parameter preserve_keys is explicitly set to false.

There are real situations where an array is still the right choice: when you need the total element count upfront with count(), when the same data must be iterated multiple times, or when array_map(), usort(), or random access by index are needed directly on the structure. In these cases, a generator is not the better solution, but an extra detour through iterator_to_array() that undoes its own benefit.

7. Performance and Memory Usage in Numbers

Generators are not free. Every step of an iteration requires an internal suspend and resume operation inside the Zend Engine, which creates a measurable, if small, overhead per element, typically in the low microsecond range. Unlike an array, a generator also offers no constant-time random access. To reach the thousandth element, all 999 previous values must actually have been iterated through, there is no index access like $array[999].

What a generator saves in return is substantial. Memory usage scales with the size of a single element, not with the number of all elements. An array with a million integer values occupies several hundred megabytes depending on the PHP version and per-cell overhead, because every element is managed as its own zval structure in memory. A generator producing the same million values one at a time holds only a single element and its internal execution state in memory at any given point, usually a few kilobytes.

The practical rule of thumb: where the element count is below a few thousand and the dataset needs to be accessed multiple times or randomly, the per-step overhead barely registers. Beyond a few tens of thousands of elements, especially for file or database streaming, the balance tips clearly in favor of the generator, because the memory saved outweighs the small CPU overhead per step by orders of magnitude.

8. Practical Use Cases: Pagination, Sequences, Pipelines

Beyond file imports, generators are excellent for fetching paginated API responses. A generator function can automatically query a paginated REST API page by page, transparently applying yield from to the individual records of each page, so the caller simply iterates over single records instead of pages, never seeing the pagination logic at all.

A second strong use case is infinite or on-demand sequences. The Fibonacci sequence, prime numbers, or running IDs can be modeled as a generator with an infinite loop and yield in its body. Because only as many values are computed as the caller actually requests, such an infinite loop costs nothing as long as nobody iterates over it without a break. This allows formulations that would never be possible with a plain array, an infinite array would simply never finish being built.

Chaining several small generator functions together produces a pipeline in which each stage stays independent of the others. A source produces raw values, a filter lets only certain values through, a transformation changes them. Because each stage is itself a generator, the whole chain remains lazy, and the consumer at the end decides how many values actually get processed.

<?php

declare(strict_types=1);

namespace App\Pipeline;

use Generator;

/**
 * Produces an infinite sequence of natural numbers, lazily.
 *
 * @return Generator<int, int>
 */
function naturalNumbers(): Generator
{
    $n = 1;
    while (true) {
        yield $n++;
    }
}

/**
 * Filters an iterable lazily using a predicate.
 *
 * @param iterable<int> $numbers
 * @return Generator<int, int>
 */
function filterBy(iterable $numbers, callable $predicate): Generator
{
    foreach ($numbers as $number) {
        if ($predicate($number)) {
            yield $number;
        }
    }
}

/**
 * Maps an iterable lazily using a transformer.
 *
 * @param iterable<int> $numbers
 * @return Generator<int, int>
 */
function mapWith(iterable $numbers, callable $transformer): Generator
{
    foreach ($numbers as $number) {
        yield $transformer($number);
    }
}

$isPrime = static function (int $n): bool {
    if ($n < 2) {
        return false;
    }
    for ($i = 2; $i * $i <= $n; $i++) {
        if ($n % $i === 0) {
            return false;
        }
    }
    return true;
};

// Compose a pipeline: nothing is computed until the consumer iterates
$pipeline = mapWith(
    filterBy(naturalNumbers(), $isPrime),
    static fn (int $prime): int => $prime * $prime,
);

$firstFiveSquaredPrimes = [];
foreach ($pipeline as $value) {
    $firstFiveSquaredPrimes[] = $value;
    if (count($firstFiveSquaredPrimes) === 5) {
        break;
    }
}

echo implode(', ', $firstFiveSquaredPrimes) . "\n"; // 4, 9, 25, 49, 121

9. Array vs. Generator Compared

The choice between an array and a generator is not a matter of taste, it depends on data volume, access pattern, and the number of passes required. The table below summarizes the key differences and shows when each structure is the better choice.

The table makes clear that this is not an either-or decision. In many real applications, both approaches are combined, for example by having a generator stream data from a file and only converting the currently needed slice into a small array with iterator_to_array() at the end, in order to work with array_map() or usort(). The generator handles memory-efficient sourcing, the array handles the targeted, reusable processing.

10. Summary

Generators and yield solve a very concrete problem: processing large or potentially unbounded datasets without holding them entirely in memory. Instead of writing a function that computes everything first and returns it afterward, a generator function produces values one at a time, at the exact moment the caller asks for them. yield from allows composing multiple generators into larger processing chains, while send() and getReturn() extend the model with two-way communication and a final return value after iteration completes.

What remains important is an honest view of the limits. A generator is forward-only, offers no random access, and is no replacement for an array when multiple passes, count(), or direct array_* functions are needed. Anyone who understands these limits and uses generators deliberately for streaming, large datasets, and pipelines, rather than reaching for them everywhere without thought, gains noticeable memory efficiency without sacrificing readability.

11. FAQ: Generators and yield in PHP