Idempotency and Retry for POST, PUT, PATCH and DELETE

1. What idempotency actually means

An operation is idempotent if it can be executed any number of times without the result changing after the first execution. The term comes from mathematics: a function f is idempotent if f(f(x)) = f(x). In the HTTP context this means: if the same request is sent n times, the server is in the same state after the first request as after the n-th request. The responses may differ (for example 200 on the first, 200 on the second DELETE), but the server state is identical.

Idempotency is not a safety guarantee. It is not about whether an operation changes data, but about whether a repetition causes damage. A PUT request that sets a user to {"name": "Alice"} is idempotent: executed ten times, the name stays Alice. A POST request that triggers a payment is not idempotent: executed ten times it triggers ten payments. This property is fundamental for error handling in distributed systems, because network failures and timeouts force retries, and retrying a non-idempotent operation can have devastating consequences.

HTTP explicitly distinguishes between idempotent and safe methods. A safe method does not change resources (GET, HEAD, OPTIONS). An idempotent method may change resources, but a repetition must not have any further effect (PUT, DELETE, GET). POST is neither safe nor idempotent. PATCH is not required to be idempotent by the specification, but it can be, depending on the semantics of the patch.

2. GET, PUT, DELETE, why idempotent?

GET is idempotent and safe: it reads data, it does not change it. Executing the same GET ten times changes nothing about the server state. PUT is idempotent because it sets a resource to a complete, defined state. PUT /users/7 with the body {"name": "Alice", "email": "alice@example.com"} replaces the user entirely with this state. A second identical PUT changes nothing, because the state is already identical. Important: PUT always sends the complete resource, not just the changed fields.

DELETE is idempotent in terms of server state: after the first DELETE, the resource is gone. A second DELETE on the same URL finds nothing left, but the state (resource not present) stays the same. The HTTP specification allows the second DELETE to return either 204 (No Content) or 404 (Not Found). For retry logic it makes sense not to treat 404 on DELETE as an error, but as confirmation that the goal has already been reached.

HEAD and OPTIONS are also idempotent and safe. CONNECT and TRACE are irrelevant in API contexts. Understanding the idempotency properties of every method is the foundation for a correct retry strategy: idempotent methods can safely be retried by a client on timeout or network failure. Non-idempotent methods must be secured through dedicated mechanisms.

3. The POST problem: duplicate execution and its consequences

The classic scenario: a client sends POST /orders with an order body. The network dies at the exact moment the server has saved the order and sends the 201 response. The client gets no response, interprets that as a failure, and retries the request. Result: two identical orders in the database, but the customer only ordered once. Or worse: the request was a payment, and the credit card was charged twice.

The problem is fundamental: on a timeout, the client does not know whether the server processed the request or not. For GET it does not matter, a retry simply reads again. For POST the retry can have catastrophic consequences. Naive solutions like "show the client an error message and let them click again manually" are not real system design, they push the problem onto the user instead of solving it in automated client-to-API communication.

# Unsafe POST without idempotency key -- duplicate order possible on retry
curl -X POST https://api.mironsoft.de/orders \
  -H "Content-Type: application/json" \
  -d '{"productId": 15, "quantity": 2, "customerId": 7}'

# Timeout -- client does not know whether the server saved the order
# Retry of the same request => possibly 2 orders in the DB

# Safe POST WITH idempotency key -- server detects the duplicate
IDEM_KEY=$(uuidgen)
curl -X POST https://api.mironsoft.de/orders \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $IDEM_KEY" \
  -d '{"productId": 15, "quantity": 2, "customerId": 7}'

# On timeout: reuse exactly the same key
curl -X POST https://api.mironsoft.de/orders \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $IDEM_KEY" \
  -d '{"productId": 15, "quantity": 2, "customerId": 7}'
# Server responds with the cached result of the first execution -- no duplicate

4. Idempotency keys: making POST safe

The idempotency key is the standard pattern for making POST requests idempotent. The client generates a one-time UUID before the first attempt and sends it as an HTTP header Idempotency-Key: <uuid>. The server stores the key together with the result of the operation in a fast store (Redis works excellently for this). On a retry with the same key, the server simply returns the cached result, without executing the operation again.

The server-side implementation must be atomic: checking whether the key already exists and storing the new key must happen in a single atomic step to avoid race conditions with parallel requests that use the same key. Redis provides SET key value NX EX ttl for exactly this: set the key only if it does not exist (NX), with a TTL. Stripe, Adyen and many other payment APIs use exactly this pattern. The key has a validity period (typically 24 hours), after which the same request can trigger a new operation again.

Important: the idempotency key protects against duplicate execution, not against incorrect inputs. If the first request with a given key creates an order for product 15, and a later request with the same key sends product 16, the server returns the cached result for product 15, and ignores the deviating body. Some implementations check whether body and key match, and respond with 422 on mismatch. That is the safer variant.

5. PATCH semantics: partial and idempotent at the same time?

PATCH, unlike PUT, sends only the fields to be changed. That makes PATCH more efficient for partial updates, but idempotency is not guaranteed. It depends on the semantics of the patch. A PATCH that sets a field to an absolute value ({"email": "new@email.com"}) is idempotent: executed ten times, the email address is still the same. A PATCH that increments ({"quantity": "+1"} or JSON Patch with "op": "add" on an array property) is not idempotent.

JSON Patch (RFC 6902) and JSON Merge Patch (RFC 7396) are the two standardized PATCH formats. JSON Merge Patch is the simpler one: it sends a JSON object that gets merged with the resource. Setting fields to null removes them. JSON Patch is more powerful and describes operations (add, remove, replace, move, copy, test) on the JSON document, similar to a Git diff. The test operation enables precondition checks: "only apply this patch if field X has value Y".

6. Retry strategies: exponential backoff and jitter

A retry that resends immediately after the first failure makes an overloaded server worse. The standard pattern for retries in distributed systems is exponential backoff: after the first failure the client waits 1 second, after the second 2 seconds, after the third 4 seconds, and so on. The wait time grows exponentially, up to a configured maximum (for example 60 seconds). That gives an overloaded server time to recover before new requests arrive.

Jitter prevents the thundering herd problem: if many clients get a failure at the same time and all retry after exactly 2 seconds, the server is hit again by many requests at once. Jitter adds a random variation to the wait time: instead of 2 seconds each client waits between 1 and 3 seconds. AWS recommends "full jitter": sleep(random(0, min(cap, base * 2^attempt))). Combined with idempotency keys on POST requests, exponential backoff with jitter is the standard solution for robust REST API clients.

#!/usr/bin/env bash
# Retry with exponential backoff + full jitter -- safe for idempotent methods
set -euo pipefail

retry_request() {
  local method="$1"
  local url="$2"
  local body="${3:-}"
  local max_attempts=5
  local base_wait=1
  local cap=60
  local attempt=0

  while (( attempt < max_attempts )); do
    http_code=$(curl -s -o /tmp/response_body -w "%{http_code}" \
      -X "$method" "$url" \
      ${body:+-H "Content-Type: application/json" -d "$body"})

    # Success: 2xx
    if [[ "$http_code" =~ ^2 ]]; then
      cat /tmp/response_body
      return 0
    fi

    # Client error: do NOT retry (400, 401, 403, 404, 422)
    if [[ "$http_code" =~ ^4 ]] && [[ "$http_code" != "429" ]]; then
      echo "[ERROR] Client error $http_code -- no retry" >&2
      return 1
    fi

    # 429 or 5xx: retry with exponential backoff + full jitter
    (( attempt++ ))
    max_sleep=$(( base_wait * (2 ** attempt) ))
    (( max_sleep > cap )) && max_sleep=$cap
    jitter=$(( RANDOM % max_sleep + 1 ))
    echo "[RETRY] Attempt $attempt/$max_attempts -- HTTP $http_code -- waiting ${jitter}s" >&2
    sleep "$jitter"
  done

  echo "[FAIL] Max retries exceeded" >&2
  return 1
}

# Safe to retry: GET is idempotent
retry_request GET "https://api.mironsoft.de/orders/42"

# Safe to retry: PUT is idempotent
retry_request PUT "https://api.mironsoft.de/orders/42" '{"status":"confirmed"}'

7. Optimistic locking with ETag and If-Match

ETags (entity tags) are an HTTP mechanism for cache validation and optimistic locking. On a GET request, the server returns an ETag header: a hash or version token of the resource. On a subsequent PUT or PATCH, the client sends this ETag along in the If-Match header. The server checks whether the ETag is still current. If yes, it executes the operation. If no, because another client has since changed the resource, it responds with 412 Precondition Failed.

This pattern prevents lost updates: if two clients simultaneously read, edit, and write back the same user, the second one does not silently overwrite the changes of the first. The second client gets 412 and must reload the resource, reapply its changes on top of the current version, and resend. ETags are also relevant for conditional GET: If-None-Match lets clients receive the full response only if the resource has changed, otherwise the server responds with 304 Not Modified without a body.

8. DELETE edge cases: treating 404 as success

The HTTP specification does not define how a repeated DELETE on an already-deleted resource should be answered. Some APIs respond with 204 No Content (resource is gone, all good), others with 404 Not Found. This is relevant for idempotent retry logic: a client that retries DELETE and gets 404 should treat that as success, not as a failure. The goal was to delete the resource, and it is gone.

The recommendation for API designers: DELETE should return 404 for an already-deleted resource, because that is the more honest status code. Clients should treat both 204 and 404 as success in their retry logic for DELETE. An alternative is for DELETE to always return 204, even if nothing was deleted. That simplifies client logic, but is less informative. In any case, retry logic for DELETE should not throw an error on 404.

9. HTTP methods compared for idempotency

The table makes it clear: GET and DELETE are unambiguous in their property as idempotent methods. PUT is idempotent, but optimistic locking with ETag additionally protects against concurrent changes. POST is fundamentally not idempotent, but can be made safe through the idempotency key mechanism. PATCH depends on the semantics of the concrete patch, increments are not idempotent, absolute value assignments are.

10. Summary

Idempotency is one of the most important properties for reliable REST APIs in distributed systems. GET, PUT and DELETE are inherently idempotent and can safely be retried after network failures or timeouts. POST is not idempotent and requires idempotency keys when retries need to be safe. PATCH depends on the concrete patch semantics. Exponential backoff with jitter is the standard pattern for retry logic that protects overloaded servers and avoids the thundering herd problem.

Implementing idempotency keys with Redis is not rocket science, but it requires atomic operations, a clear TTL strategy, and thoughtful error handling for body mismatches. Optimistic locking with ETag and If-Match additionally protects against lost updates under concurrent changes. Together, these patterns make REST APIs a reliable building block in microservice architectures, where network partitions and timeouts are part of everyday life.

11. FAQ: Idempotency and Retry in REST APIs