Asynchronous and Bulk Web APIs in Magento 2

1. Why synchronous REST calls hit their limits on bulk data

A synchronous REST call in Magento 2 occupies a PHP-FPM worker process for the entire duration of the request. For a single product update this is not a problem, since the request completes within milliseconds. But as soon as an ERP or PIM system updates 5,000 or 10,000 SKUs in a loop over individual PUT /rest/V1/products/:sku calls, the accumulated blocking time adds up to several minutes per import run, and each of these calls holds an entire worker hostage the whole time. If the FPM pool is limited via pm.max_children, which is the case in every production installation, fewer workers remain available for other concurrent requests, such as checkout requests from real customers.

Nginx and PHP-FPM ship with default timeout values designed for single requests: fastcgi_read_timeout is often set to 60 seconds, and max_execution_time in php.ini is frequently in the low double digits as well. A synchronous bulk import that internally triggers hundreds of database writes, indexer triggers and event observers reliably exceeds these limits. The result is an aborted HTTP request whose server-side process may keep running regardless, leaving data inconsistent because the client cannot cleanly distinguish the abort from the actual processing state.

Even more critical is the effect on ongoing operations: while a mass import blocks synchronous workers, noticeable delays can occur for concurrent checkout flows, because the same FPM pool serves both loads. This is exactly where the asynchronous bulk API comes in: it decouples accepting a request from actually processing it, immediately frees the HTTP worker, and moves the real work into a consumer process outside the web request cycle.

2. Architecture of the asynchronous web API in Magento

Magento 2 automatically provides an asynchronous variant for practically every existing REST endpoint, without requiring any additional code in the respective module. The key to this is the HTTP header X-Asynchronous: true. When a regular request such as POST /rest/V1/products is sent with this header, the framework intercepts it through the AsynchronousSchemaBuilder and the associated webapi rewrite logic before the actual service contract is invoked. Instead of returning the operation's result synchronously, the request is converted into a message queue entry, referenced in the magento_bulk and magento_operation database tables, and the client immediately receives a response with HTTP status 202 (Accepted) along with a bulk_uuid that can later be used to poll progress.

Technically, this mechanism relies on the generic consumer pipeline async.operation.add, which registers a matching asynchronous route for every service contract based on webapi.xml declarations. Importantly, you do not need to define a separate async route in webapi.xml: the framework generates the asynchronous variant automatically from the synchronous declaration, as long as the underlying service contract follows a *Interface::save() or comparable repository pattern. The snippet below shows a regular synchronous route, as found in nearly every Magento module, from which the framework derives the asynchronous variant.

<?xml version="1.0"?>
<!-- app/code/Mironsoft/PriceSync/etc/webapi.xml -->
<routes xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-webapi:etc/webapi.xsd">

    <!-- Standard synchronous route -->
    <route url="/V1/mironsoft-pricesync/prices" method="POST">
        <service class="Mironsoft\PriceSync\Api\PriceUpdateRepositoryInterface" method="save"/>
        <resources>
            <resource ref="Magento_Catalog::catalog"/>
        </resources>
    </route>

    <!-- Called with header "X-Asynchronous: true" against the SAME url;
         Magento generates the bulk-capable variant automatically -->
</routes>

The decisive difference between a plain synchronous call and its asynchronous counterpart therefore lies not in the URL or in additional XML, but exclusively in the request header and the response behavior. This is what makes the asynchronous web API particularly attractive for existing integrations: a client already talking to /rest/V1/products can switch to asynchronous processing simply by adding a header, without changing the endpoint or adjusting request payloads.

3. The bulk API in detail

While the asynchronous web API turns a single request into a single queue message, the bulk API goes a step further: it accepts an array of items in one HTTP request and splits it server-side into multiple OperationInterface instances, all sharing the same bulk_uuid. This split is handled by BulkManagementInterface::scheduleBulk() from \Magento\AsynchronousOperations\Api, which is centrally responsible for all bulk-capable endpoints. Each individual operation additionally carries its own operation_key, which allows the result of a specific item within the bulk run to be uniquely identified later on.

The practical difference is most obvious with mass operations: if a client sends 500 product changes to an asynchronous single endpoint, 500 separate HTTP requests are created, each with its own bulk_uuid and its own network overhead from TLS handshake, HTTP headers and token validation. If the same client sends the same 500 changes to a bulk-capable endpoint such as POST /rest/V1/products/bySku with an array in the body, a single HTTP request is created with one shared bulk_uuid and a list of operation IDs in the response. Processing itself still runs through individual consumer invocations per operation, but the network and authentication overhead is almost entirely eliminated.

Internally, OperationInterface implements fields such as bulk_uuid, topic_name, serialized_data and status, which are persisted in the magento_operation table. Each operation is routed via its topic name to exactly one consumer, which executes the actual business logic and then updates the status through OperationManagementInterface::changeOperationStatus(). This separation between bulk metadata (one bulk_uuid per request) and operation details (many operations per bulk) is the core of what distinguishes the bulk API from a simple loop over the asynchronous web API.

4. Configuring queue.xml and communication.xml

For a custom message to travel through the message queue at all, three declarations must work together: communication.xml defines the topic together with its request schema, queue.xml binds that topic to a concrete exchange and queue, and consumers.xml registers the PHP consumer that consumes messages from that queue. All three files live in the etc directory of the respective module and are re-read by Magento on cache flush.

A common mistake in custom modules is declaring the topic in communication.xml but forgetting the binding in queue.xml. In that case the message never lands in a queue and the consumer runs empty, without any error message, which needlessly complicates troubleshooting. Equally important is the max_messages value in consumers.xml: it determines after how many processed messages the consumer process cleanly terminates, which is essential when running under Supervisor or systemd so that memory leaks in long-running PHP processes never become a factor.

<!-- app/code/Mironsoft/PriceSync/etc/communication.xml -->
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/communication.xsd">
    <topic name="mironsoft.pricesync.bulk.update"
           request="Mironsoft\PriceSync\Api\Data\PriceUpdateRequestInterface">
    </topic>
</config>

<!-- app/code/Mironsoft/PriceSync/etc/queue.xml -->
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/queue.xsd">
    <broker topic="mironsoft.pricesync.bulk.update" exchange="magento" type="amqp">
        <queue name="mironsoft.pricesync.bulk.update" consumer="mironsoftPriceSyncConsumer" consumerInstance="Mironsoft\PriceSync\Model\Queue\PriceUpdateConsumer"/>
    </broker>
</config>

<!-- app/code/Mironsoft/PriceSync/etc/consumers.xml -->
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/consumer.xsd">
    <consumer name="mironsoftPriceSyncConsumer"
              queue="mironsoft.pricesync.bulk.update"
              connection="amqp"
              max_messages="10000"
              handler="Mironsoft\PriceSync\Model\Queue\PriceUpdateConsumer::process"/>
</config>

After creating or changing these three files, a setup:upgrade and a restart of the consumers via bin/magento queue:consumers:start mironsoftPriceSyncConsumer is required for the new binding to take effect. In production environments, consumers typically run as dedicated Supervisor or systemd services rather than ad-hoc command-line processes, so that a crash automatically triggers a restart and asynchronous processing does not silently grind to a halt.

5. Building custom asynchronous endpoints and consumers

To make a custom service contract method usable asynchronously, a regular webapi.xml declaration with a repository implementing a save() or comparable method is sufficient. The actual processing logic, however, does not belong in the service contract itself, but in a dedicated consumer class that implements ConsumerInterface from \Magento\Framework\MessageQueue. This class is registered via consumers.xml and receives the deserialized message as a typed object, as declared in communication.xml.

With PHP 8.4 and constructor property promotion, such a consumer class can be written considerably more compactly than was common in older Magento versions. It is important that the consumer does not simply swallow errors, but updates the status of the associated operation via OperationManagementInterface, so the caller can determine through the status endpoint whether processing was actually successful.

<?php

declare(strict_types=1);

namespace Mironsoft\PriceSync\Model\Queue;

use Magento\AsynchronousOperations\Api\Data\OperationInterface;
use Magento\Framework\EntityManager\EntityManager;
use Magento\Framework\MessageQueue\ConsumerInterface;
use Mironsoft\PriceSync\Api\Data\PriceUpdateRequestInterface;
use Mironsoft\PriceSync\Api\PriceUpdateRepositoryInterface;
use Psr\Log\LoggerInterface;

/**
 * Consumer that applies a single price update message from the bulk queue.
 */
final class PriceUpdateConsumer implements ConsumerInterface
{
    /**
     * @param PriceUpdateRepositoryInterface $priceUpdateRepository Repository applying the actual price change
     * @param LoggerInterface $logger Logger used for retriable and non-retriable failures
     */
    public function __construct(
        private readonly PriceUpdateRepositoryInterface $priceUpdateRepository,
        private readonly LoggerInterface $logger,
    ) {
    }

    /**
     * Processes a single price update message consumed from the bulk queue.
     *
     * @param PriceUpdateRequestInterface $message Deserialized message matching communication.xml schema
     * @return void
     */
    public function process(PriceUpdateRequestInterface $message): void
    {
        try {
            // Idempotency guard: skip if this external order/reference id was already applied
            $this->priceUpdateRepository->save($message);
        } catch (\Magento\Framework\Exception\CouldNotSaveException $exception) {
            // Transient error, e.g. lock wait timeout: mark retriable so the consumer retries later
            $this->logger->warning('Retriable price update failure', ['sku' => $message->getSku()]);
            throw $exception;
        } catch (\Throwable $exception) {
            // Permanent error, e.g. invalid SKU: mark non-retriable, do not retry indefinitely
            $this->logger->error('Non-retriable price update failure', ['sku' => $message->getSku()]);
            throw $exception;
        }
    }
}

In practice, the actual business logic is not embedded directly in the consumer, but delegated to a repository that can also be reused by synchronous endpoints. This keeps business logic independent of whether it is triggered by a synchronous REST call, an asynchronous bulk API, or an internal cron job, a principle Magento's service contract architecture already encourages.

6. Querying and monitoring operation status

After submitting an asynchronous request or a bulk request, the client receives a bulk_uuid in the HTTP response. The endpoint GET /rest/V1/bulk/:bulk_uuid/status allows querying the current processing state of every individual operation within that bulk run. Each operation carries a status from the constant set of OperationInterface: STATUS_TYPE_OPEN for messages not yet picked up, STATUS_TYPE_COMPLETE for successfully processed operations, STATUS_TYPE_RETRIABLY_FAILED for operations that failed due to a transient error and will be retried, and STATUS_TYPE_NOT_RETRIABLY_FAILED for operations that failed permanently.

External systems that talk to an asynchronous bulk API need a clear polling strategy to monitor status efficiently without overloading the server. Exponential backoff has proven effective, starting at around 2 seconds between polls and an upper bound of 30 to 60 seconds, combined with an overall timeout after which the run is flagged as stuck and checked manually. It is important that a PIM or ERP system never blindly interprets HTTP status 202 as success, but instead queries the actual operation status through the status endpoint before considering processing complete.

{
  "bulk_uuid": "3f9a1c2e-8b7d-4e21-9a4c-6d2f1e0c8a55",
  "operations_list": [
    {
      "id": 1021,
      "bulk_uuid": "3f9a1c2e-8b7d-4e21-9a4c-6d2f1e0c8a55",
      "topic_name": "mironsoft.pricesync.bulk.update",
      "status": 3,
      "result_message": "",
      "serialized_data": "{\"sku\":\"WSH-2000\",\"price\":\"49.9000\"}",
      "error_code": null
    },
    {
      "id": 1022,
      "bulk_uuid": "3f9a1c2e-8b7d-4e21-9a4c-6d2f1e0c8a55",
      "topic_name": "mironsoft.pricesync.bulk.update",
      "status": 4,
      "result_message": "SKU WSH-2001 does not exist",
      "serialized_data": "{\"sku\":\"WSH-2001\",\"price\":\"39.9000\"}",
      "error_code": 404
    }
  ]
}

The numeric value of status corresponds to the class constants: 1 stands for STATUS_TYPE_OPEN, 2 for STATUS_TYPE_COMPLETE, 3 is used in some Magento versions for STATUS_TYPE_RETRIABLY_FAILED, and 4 for STATUS_TYPE_NOT_RETRIABLY_FAILED, though the exact mapping can vary slightly by Magento minor version, so when in doubt always check against the class constants rather than hardcoded numbers.

7. RabbitMQ vs. MySQL as a message broker

Magento supports two broker implementations for the message queue: a MySQL-based queue that runs directly on the existing database tables without additional infrastructure, and RabbitMQ as a dedicated AMQP broker. For small installations with occasional asynchronous imports and low message volume, the MySQL queue is often sufficient, since no additional service needs to be operated and monitored. Configuration is simply a matter of not setting type="amqp" in queue.xml, letting the default database queue take over.

As soon as production bulk processing with multiple parallel consumer instances, high message frequency, or requirements for guaranteed delivery come into play, however, RabbitMQ becomes a practically mandatory prerequisite. RabbitMQ offers real message acknowledgement, message prioritization, dead-letter exchanges and considerably better scalability with many concurrent consumers, because the MySQL queue quickly becomes a locking bottleneck on the database under high parallelism. Configuration is centralized in app/etc/env.php under the queue key, where host, port, user, password and virtual host of the RabbitMQ server are stored.

<?php
// app/etc/env.php (excerpt)
return [
    'queue' => [
        'amqp' => [
            'host' => 'rabbitmq.internal.mironsoft.de',
            'port' => '5672',
            'user' => 'magento',
            'password' => 'REPLACE_WITH_SECRET',
            'virtualhost' => '/magento-prod',
            'ssl' => true,
        ],
    ],
];

Horizontal scaling of consumers works in both setups, but is considerably more reliable with RabbitMQ: multiple instances of the same consumer, started via bin/magento queue:consumers:start mironsoftPriceSyncConsumer --pid-file-path=/var/run/consumer1.pid on different processes or even different hosts, automatically share the messages from the same queue without any message being processed twice, as long as RabbitMQ acts as the broker. With the MySQL queue this guarantee is weaker, which is why RabbitMQ is the recommended choice for serious, production-grade asynchronous bulk API processing with multiple consumer workers.

8. Error handling, retries and idempotency in bulk operations

Every consumer registered via consumers.xml can be configured with the max_retries attribute, which determines how many times an operation marked retriable is redelivered before it is finally marked as STATUS_TYPE_NOT_RETRIABLY_FAILED. If this value is not set sensibly, failing messages can circulate endlessly between queue and consumer, tying up system resources without ever succeeding. Dead-letter behavior, where permanently failed messages are moved into a separate queue, can be implemented in RabbitMQ using dead-letter exchanges and is the recommended strategy for isolating such messages for later manual analysis rather than silently discarding them.

The most important aspect in bulk operations, however, is idempotency. Since a message can potentially be delivered multiple times on a retry, for example because the consumer crashes between successful processing and acknowledgement, every operation must be designed so that multiple executions do not produce incorrect results. In practice this means every operation should carry a unique idempotency id, for example the external order number from a PIM system, or a UUID generated by the calling system. Before actual processing, the consumer checks whether this id already exists in a dedicated tracking table and skips the operation if so.

A price update that writes the same record twice with the same value is inherently idempotent and harmless. A stock decrement, on the other hand, which reduces inventory by a fixed amount on every call, is dangerous without an idempotency check, because a retry would incorrectly reduce stock twice. Exactly such non-idempotent operations are the most common cause of hard-to-trace data inconsistencies in production systems that use asynchronous processing with retries without first carefully thinking through the idempotency of the underlying business logic.

9. Practical example and comparison: mass price update from a PIM system

A PIM system transferring 10,000 SKUs with new prices to Magento can be integrated in three fundamentally different ways. Using synchronous REST calls, the PIM system would send 10,000 individual PUT requests, each blocking a PHP-FPM worker for the duration of the database write, and at a realistic 80 to 150 milliseconds per request, a total runtime of 15 to 25 minutes results purely from HTTP communication, without even accounting for the additional load from indexer triggers. Using individual asynchronous requests with X-Asynchronous: true, the PIM system would still send 10,000 requests, but each one is answered immediately with HTTP 202, so the network load is reduced to the client-side connection setup, while actual processing runs in the background via consumers.

The third and, for this use case, optimal option is the bulk API: the PIM system sends all 10,000 price changes in a few large batches, for example 500 items each, to a bulk-capable endpoint. This results in only 20 HTTP requests instead of 10,000, each with its own bulk_uuid, while the server-side split into individual operations and their processing runs via consumers in parallel and independently of the original HTTP request. The following call shows how such a bulk request is triggered in practice using curl.

#!/usr/bin/env bash
set -euo pipefail

TOKEN="$(cat /run/secrets/magento_api_token)"
BASE_URL="https://shop.mironsoft.de/rest/V1"

# Trigger a bulk price update for a batch of 500 SKUs
# Note: X-Asynchronous is implicit for true bulk endpoints; shown here for clarity
curl -s -X POST "${BASE_URL}/products/bySku" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -H "X-Asynchronous: true" \
  -d @batch-001-of-20.json | tee response-001.json

# Extract bulk_uuid from the response for later status polling
bulk_uuid="$(jq -r '.bulk_uuid' response-001.json)"
echo "Batch 001 scheduled: ${bulk_uuid}"

# Poll status with simple exponential backoff
wait_seconds=2
for attempt in $(seq 1 10); do
  status_response="$(curl -s -H "Authorization: Bearer ${TOKEN}" \
    "${BASE_URL}/bulk/${bulk_uuid}/status")"
  echo "${status_response}" | jq -c '.operations_list[] | {id, status, error_code}'
  sleep "${wait_seconds}"
  wait_seconds=$(( wait_seconds < 30 ? wait_seconds * 2 : 30 ))
done

The table below summarizes the three approaches based on the criteria relevant to this practical example. It shows that choosing between synchronous REST API, asynchronous web API and bulk API is not a matter of taste, but depends directly on data volume, latency requirements and client architecture.

For the concrete example with 10,000 SKUs, this means: synchronous REST calls are unsuitable for this use case and risk timeouts as well as blocked workers. Individual asynchronous requests solve the blocking problem, but still generate 10,000 HTTP requests with corresponding overhead. The bulk API reduces the number of requests by the batch size factor, while server-side processing remains granularly traceable per operation, making it the technically and operationally superior choice for this scenario.

10. Summary

The asynchronous bulk API in Magento 2 solves a clearly defined problem: synchronous REST calls block PHP-FPM workers, run into HTTP timeouts on bulk data, and in the worst case impact checkout throughput. The X-Asynchronous: true header turns practically any existing endpoint into a non-blocking variant, while BulkManagementInterface and OperationInterface split entire arrays of items into individually traceable operations sharing a common bulk_uuid. Configuration through communication.xml, queue.xml and consumers.xml is pure declarative XML, with no need to change the actual business logic.

For production use, three points are decisive: RabbitMQ instead of the MySQL queue as soon as multiple consumer instances need to run in parallel under load, a properly configured retry and dead-letter behavior so that failing messages do not circulate endlessly, and above all idempotency at the level of every single operation, so that retries do not create duplicate effects in the database. Anyone who observes these three points can reliably handle mass imports, price updates and ERP integrations through the asynchronous bulk API without jeopardizing the stability of the running shop.

11. FAQ: Asynchronous and Bulk Web APIs in Magento 2