The Iterator Pattern is one of the most commonly used GoF patterns in Magento, you use it every day whenever you iterate over a collection. foreach ($productCollection as $product) is the Iterator Pattern in action. But what happens internally? When does the DB query actually run? And how do you avoid memory overflows with 100,000 products? This deep dive walks through the full chain from Varien_Data_Collection to PHP generators.

1. GoF Iterator Pattern: the interface

The GoF Iterator Pattern defines an interface that provides sequential access to the elements of a collection without exposing its internal representation.

+------------------+          uses          +------------------+
|     Client       | ---------------------->|    Iterator      |
|                  |                        |  + current()     |
|  foreach(coll)   |                        |  + key()         |
+------------------+                        |  + next()        |
                                            |  + rewind()      |
                                            |  + valid()       |
                                            +------------------+
                                                    ▲
                                                    |
                                    +----------------------------------+
                                    | Magento AbstractCollection       |
                                    | implements Iterator + Countable  |
                                    +----------------------------------+
    

The four advantages of the Iterator Pattern:

• Decoupling: the client doesn't need to know whether it's an array, a DB query or a remote stream
• Lazy evaluation: data is only loaded when actually needed
• Uniform API: foreach works the same way for every collection
• Composability: iterators can be decorated and combined

2. PHP Iterator interface and IteratorAggregate

PHP provides two ways to make objects traversable with foreach:

<?php

// Option 1: Iterator, the class implements Iterator directly
// 5 methods must be implemented
class DirectIterator implements \Iterator
{
    private int $position = 0;
    private array $data = [];

    public function current(): mixed    { return $this->data[$this->position]; }
    public function key(): int          { return $this->position; }
    public function next(): void        { $this->position++; }
    public function rewind(): void      { $this->position = 0; }
    public function valid(): bool       { return isset($this->data[$this->position]); }
}

// Option 2: IteratorAggregate, delegates to an iterator
// Cleaner when the class itself doesn't represent the iteration
class Collection implements \IteratorAggregate
{
    private array $items = [];

    public function getIterator(): \ArrayIterator
    {
        return new \ArrayIterator($this->items);
    }
}

// Option 3: Generator-based (PHP 5.5+)
// Laziest approach, perfect for large datasets
function yieldProducts(array $ids): \Generator
{
    foreach ($ids as $id) {
        yield $id => loadProduct($id); // Lazy: loaded only when needed
    }
}
    

3. Varien_Data_Collection: the foundation

Magento\Framework\Data\Collection (formerly Varien_Data_Collection) is the base class of all Magento collections. It implements Iterator, Countable and ArrayAccess:

<?php

// vendor/magento/framework/Data/Collection.php (simplified)
namespace Magento\Framework\Data;

use Magento\Framework\DataObject;

class Collection implements \IteratorAggregate, \Countable, \ArrayAccess
{
    /** @var DataObject[] Internal storage (the actual pool of items) */
    protected array $_items = [];

    private int $_curPos = 0;
    private bool $_isCollectionLoaded = false;

    /**
     * PHP IteratorAggregate, used by foreach
     * Triggers load() if not yet loaded
     */
    public function getIterator(): \ArrayIterator
    {
        $this->load();
        return new \ArrayIterator($this->_items);
    }

    /**
     * Add an item to the collection.
     */
    public function addItem(DataObject $item): static
    {
        $itemId = $this->_getItemId($item);
        if ($itemId !== null) {
            $this->_items[$itemId] = $item;
        } else {
            $this->_items[] = $item;
        }
        return $this;
    }

    /**
     * Load collection, base implementation is a no-op
     * Overridden in ResourceModel\Collection to run SQL
     */
    public function load(bool $printQuery = false, bool $logQuery = false): static
    {
        $this->_isCollectionLoaded = true;
        return $this;
    }

    /**
     * Countable interface
     */
    public function count(): int
    {
        $this->load();
        return count($this->_items);
    }

    /**
     * Get all items as array
     */
    public function getItems(): array
    {
        $this->load();
        return $this->_items;
    }
}
    

4. Magento AbstractCollection and lazy loading

The DB-bound collection extends Magento\Framework\Model\ResourceModel\Db\Collection\AbstractCollection and overrides load() to execute an SQL query, but only on first access:

<?php

// vendor/magento/framework/Model/ResourceModel/Db/Collection/AbstractCollection.php

namespace Magento\Framework\Model\ResourceModel\Db\Collection;

use Magento\Framework\Data\Collection\AbstractDb;
use Magento\Framework\DB\Select;

abstract class AbstractCollection extends AbstractDb
{
    protected bool $_isCollectionLoaded = false;

    /**
     * LAZY LOADING: SQL is only executed here
     * Triggered by: foreach, count(), getItems(), getFirstItem()
     */
    public function load(bool $printQuery = false, bool $logQuery = false): static
    {
        if ($this->isLoaded()) {
            return $this; // Already loaded, return immediately
        }

        $this->_beforeLoad();
        $this->_renderFilters();
        $this->_renderOrders();
        $this->_renderLimit();

        // THIS is where the SQL query executes
        $this->printLogQuery($printQuery, $logQuery);
        $data = $this->getData(); // ← SELECT * FROM ... WHERE ... LIMIT ...

        $this->resetData();
        if (is_array($data)) {
            foreach ($data as $row) {
                $item = $this->getNewEmptyItem();
                $item->setData($row); // Hydrate model with row data
                $this->addItem($item);
                // Dispatch event for each item
            }
        }

        $this->_setIsLoaded();
        $this->_afterLoad();
        return $this;
    }
}
    

The critical thing to understand: the collection is a query builder, not an array. Only on the first foreach, count() or getItems() does the SQL query actually hit the database.

<?php

declare(strict_types=1);

use Magento\Catalog\Model\ResourceModel\Product\CollectionFactory;

// This line does NOT run any DB query!
$collection = $collectionFactory->create();

// These lines only configure the query builder:
$collection->addAttributeToSelect(['name', 'price', 'sku'])
    ->addAttributeToFilter('status', 1)
    ->addAttributeToFilter('visibility', ['in' => [3, 4]])
    ->setOrder('created_at', 'DESC')
    ->setPageSize(20)
    ->setCurPage(1);

// ONLY HERE does the SQL query actually execute:
foreach ($collection as $product) { // ← load() triggered here
    echo $product->getName();
}

// Second foreach: no second DB query, data is cached
foreach ($collection as $product) { // ← uses cached $_items
    echo $product->getSku();
}
    

5. Collection filters, joins and pagination

The AbstractCollection is a powerful query builder. The most important methods:

<?php

declare(strict_types=1);

namespace Mironsoft\Catalog\Model;

use Magento\Catalog\Model\ResourceModel\Product\CollectionFactory;
use Magento\Catalog\Model\ResourceModel\Product\Collection as ProductCollection;

/**
 * Demonstrates collection filtering, joining, and pagination.
 */
final class ProductCollectionBuilder
{
    public function __construct(
        private readonly CollectionFactory $collectionFactory,
    ) {}

    /**
     * Builds a filtered, joined, paginated product collection.
     */
    public function getFilteredProducts(int $categoryId, int $page = 1): ProductCollection
    {
        $collection = $this->collectionFactory->create();

        // Select only needed attributes (avoids loading all EAV attributes)
        $collection->addAttributeToSelect(['name', 'price', 'sku', 'thumbnail']);

        // EAV attribute filter
        $collection->addAttributeToFilter('status', ['eq' => 1]);
        $collection->addAttributeToFilter('visibility', ['in' => [3, 4]]);

        // Price range filter
        $collection->addAttributeToFilter('price', ['gteq' => 10.00, 'lteq' => 500.00]);

        // Category filter via join
        $collection->addCategoriesFilter(['in' => [$categoryId]]);

        // Stock filter
        $collection->joinField(
            'is_in_stock',
            'cataloginventory_stock_item',
            'is_in_stock',
            'product_id=entity_id',
            '{{table}}.stock_id=1',
            'left'
        );
        $collection->addFieldToFilter('is_in_stock', 1);

        // Sorting
        $collection->addAttributeToSort('position', 'ASC');
        $collection->addAttributeToSort('created_at', 'DESC');

        // Pagination, crucial for large catalogs
        $collection->setPageSize(24);
        $collection->setCurPage($page);

        return $collection; // Still not loaded, lazy!
    }
}
    

6. Chunked iterator: large catalogs without memory overflow

The most common performance problem: loading 100,000 products all at once. The solution is a chunked iterator that splits the collection into pages:

<?php

declare(strict_types=1);

namespace Mironsoft\Import\Iterator;

use Magento\Catalog\Model\ResourceModel\Product\Collection;
use Magento\Catalog\Model\ResourceModel\Product\CollectionFactory;

/**
 * Memory-efficient chunked iterator for large product collections.
 * Loads only one page at a time, prevents PHP OOM errors.
 */
final class ChunkedProductIterator implements \IteratorAggregate
{
    private const CHUNK_SIZE = 1000;

    public function __construct(
        private readonly CollectionFactory $collectionFactory,
    ) {}

    /**
     * Yields products in chunks, never loads all products at once.
     *
     * @return \Generator<int, \Magento\Catalog\Model\Product>
     */
    public function getIterator(): \Generator
    {
        $page = 1;
        $totalYielded = 0;

        do {
            $collection = $this->createPage($page);
            $count = $collection->count();

            if ($count === 0) {
                break;
            }

            foreach ($collection as $product) {
                yield $totalYielded => $product;
                $totalYielded++;
            }

            // Critical: free memory after each chunk
            $collection->clear();
            unset($collection);

            $page++;

        } while ($count === self::CHUNK_SIZE);
    }

    private function createPage(int $page): Collection
    {
        $collection = $this->collectionFactory->create();
        $collection->addAttributeToSelect(['name', 'sku', 'price']);
        $collection->setPageSize(self::CHUNK_SIZE);
        $collection->setCurPage($page);
        return $collection;
    }
}
    

Usage inside an import service:

<?php

declare(strict_types=1);

namespace Mironsoft\Import\Service;

use Mironsoft\Import\Iterator\ChunkedProductIterator;

/**
 * Processes all products in chunks, memory-safe.
 */
final class ProductExportService
{
    public function __construct(
        private readonly ChunkedProductIterator $iterator,
    ) {}

    /**
     * Exports all products to CSV without loading all at once.
     */
    public function exportToCsv(string $outputPath): void
    {
        $file = fopen($outputPath, 'w');
        fputcsv($file, ['SKU', 'Name', 'Price']);

        $count = 0;
        foreach ($this->iterator as $position => $product) {
            fputcsv($file, [
                $product->getSku(),
                $product->getName(),
                $product->getPrice(),
            ]);
            $count++;

            // Memory check every 10,000 products
            if ($count % 10000 === 0) {
                echo sprintf(
                    "Processed %d products, memory: %s MB\n",
                    $count,
                    round(memory_get_usage(true) / 1024 / 1024, 2)
                );
            }
        }

        fclose($file);
        echo "Exported {$count} products total.\n";
    }
}
    

7. PHP generators as lazy iterators

PHP generators (since 5.5) are the most elegant way to implement lazy iterators:

<?php

declare(strict_types=1);

namespace Mironsoft\Catalog\Iterator;

use Magento\Catalog\Api\ProductRepositoryInterface;
use Magento\Framework\Api\SearchCriteriaBuilder;
use Magento\Framework\Api\SortOrderBuilder;

/**
 * Generator-based lazy product iterator using Service Contracts.
 * Preferred over direct Collection access for API-layer code.
 */
final class ProductIteratorGenerator
{
    private const PAGE_SIZE = 500;

    public function __construct(
        private readonly ProductRepositoryInterface $productRepository,
        private readonly SearchCriteriaBuilder $searchCriteriaBuilder,
        private readonly SortOrderBuilder $sortOrderBuilder,
    ) {}

    /**
     * Lazily yields all active products using SearchCriteria pagination.
     *
     * @return \Generator<string, \Magento\Catalog\Api\Data\ProductInterface>
     */
    public function iterateActive(): \Generator
    {
        $sortOrder = $this->sortOrderBuilder
            ->setField('entity_id')
            ->setAscendingDirection()
            ->create();

        $page = 1;

        do {
            $criteria = $this->searchCriteriaBuilder
                ->addFilter('status', 1)
                ->addSortOrder($sortOrder)
                ->setPageSize(self::PAGE_SIZE)
                ->setCurrentPage($page)
                ->create();

            $result = $this->productRepository->getList($criteria);
            $items = $result->getItems();

            foreach ($items as $product) {
                yield $product->getSku() => $product;
            }

            $page++;

        } while (count($items) === self::PAGE_SIZE);
    }
}
    

What makes generators especially powerful is that they can communicate bidirectionally:

<?php

// Generator with send(), allows feedback from the consumer
function* batchProcessor(): \Generator
{
    $processed = 0;
    while (true) {
        $item = yield $processed; // Yield current count, receive next item
        if ($item === null) break;
        process($item);
        $processed++;
    }
}

$gen = batchProcessor();
$gen->current(); // Start generator
$gen->send($product1); // Send item, generator processes it
$gen->send($product2);
$gen->send(null); // Signal done
    

8. Building a custom iterator for Magento

Sometimes you need an iterator that isn't tied to a DB collection, for example for CSV files or external APIs:

<?php

declare(strict_types=1);

namespace Mironsoft\Import\Iterator;

use Magento\Framework\DataObject;

/**
 * Lazy CSV iterator, reads one line at a time without loading entire file.
 * Memory usage: O(1) regardless of file size.
 */
final class CsvProductIterator implements \Iterator
{
    private mixed $fileHandle = null;
    private array $currentRow = [];
    private int $position = 0;
    private array $headers = [];

    public function __construct(
        private readonly string $filePath,
    ) {}

    public function rewind(): void
    {
        if ($this->fileHandle !== null) {
            fclose($this->fileHandle);
        }

        $this->fileHandle = fopen($this->filePath, 'r');
        $this->position = 0;

        // Read header row
        $this->headers = fgetcsv($this->fileHandle) ?: [];

        // Read first data row
        $this->next();
    }

    public function current(): DataObject
    {
        return new DataObject(array_combine($this->headers, $this->currentRow));
    }

    public function key(): int
    {
        return $this->position;
    }

    public function next(): void
    {
        $row = fgetcsv($this->fileHandle);
        $this->currentRow = $row !== false ? $row : [];
        $this->position++;
    }

    public function valid(): bool
    {
        return !empty($this->currentRow)
            && count($this->currentRow) === count($this->headers);
    }

    public function __destruct()
    {
        if ($this->fileHandle !== null) {
            fclose($this->fileHandle);
        }
    }
}
    

Usage:

<?php

$iterator = new CsvProductIterator('/var/import/products.csv');

foreach ($iterator as $lineNumber => $row) {
    echo $row->getData('sku') . ': ' . $row->getData('name') . PHP_EOL;
    // Memory: ~4KB regardless of whether file has 100 or 1,000,000 rows
}
    

9. Iterator vs. getItems(): performance comparison

A typical mistake when iterating over large collections:

<?php

// WRONG: loads ALL 500,000 products into RAM at once
$collection = $collectionFactory->create();
// No setPageSize() → PHP memory_limit exceeded

foreach ($collection as $product) { // Fatal: Allowed memory exhausted
    processProduct($product);
}

// CORRECT: chunk-based with setPageSize()
$collection = $collectionFactory->create();
$collection->setPageSize(1000);

// Or even better: ChunkedProductIterator (see above)
foreach ($chunkedIterator as $product) {
    processProduct($product);
}
    

10. Conclusion: using collections correctly

The Iterator Pattern is omnipresent in Magento, every collection is an iterator. The most important rules:

Frequently asked questions about the Iterator Pattern in Magento