Idempotency Keys

Retrying reads is usually harmless. Retrying writes is dangerous because writes have side effects: money moves, inventory changes, seats become reserved, and downstream services receive messages. The nasty failure mode is not simply "the first request failed". It is "the first request may have succeeded, but the client cannot tell".

1. Client → POST /payments {orderId: "o_123", amount: 50}
2. API charges card successfully at the payment gateway
3. API response is lost because the network times out
4. Client retries the same POST /payments
5. API charges card again because it treats the retry as a new request

This is common in real systems because clients, load balancers, message queues, and job runners all retry. Mobile apps retry after spotty Wi-Fi. Browsers retry after a tab resumes. Queue consumers retry after a worker crashes. If the operation is expensive or irreversible, the server must be able to recognize that two HTTP requests are really one logical operation.

The client creates a high-entropy key, usually a UUID, before sending a mutation. The key travels in a header such as Idempotency-Key: 7b4c.... The server scopes the key to a caller and operation, atomically claims it, executes the work once, stores the response, and replays that response for later attempts.

Client chooses key K = uuid()

POST /orders
Idempotency-Key: K
Body: {cartId: "cart_9", paymentToken: "tok_abc"}

Server:
  1. Look up (tenant_id, endpoint, K)
  2. If completed:
       return stored status_code + response_body
  3. If in_progress:
       wait, poll, or return 409/202 so the client retries later
  4. If missing:
       atomically insert {K, request_hash, status: "in_progress"}
       create order and charge payment exactly once
       update {K, status: "completed", response_body, status_code}
       return the response

Store the result, not only the fact that the key existed

A good idempotency table stores the final HTTP status and response body. If the first attempt returns 201 Created with {orderId: "o_123"}, the retry should get that same response. That makes the client state machine simple: retry until it receives the result for the operation it already asked for.

Idempotency is only as strong as the atomic claim on the key. Two requests with the same key can arrive at the same millisecond, hit two API servers, and race. The storage layer must make exactly one of them the first executor.

-- One row per logical request.
CREATE TABLE idempotency_keys (
  scope          text NOT NULL,
  key            text NOT NULL,
  request_hash   text NOT NULL,
  status         text NOT NULL,
  response_json  jsonb,
  status_code    int,
  expires_at     timestamptz NOT NULL,
  PRIMARY KEY (scope, key)
);

-- Atomic claim. Only one concurrent request can insert this row.
INSERT INTO idempotency_keys(scope, key, request_hash, status, expires_at)
VALUES ($scope, $key, $hash, 'in_progress', now() + interval '24 hours')
ON CONFLICT DO NOTHING;

• Relational database: use a unique constraint on (scope, key). This is the most straightforward choice when the mutation also writes to the same database, because claiming the key and creating the order can live in one transaction.
• Redis: use SET key value NX PX ttl or a Lua script to atomically claim and update state. Redis is fast and useful for high-volume APIs, but think carefully about persistence and failover if the side effect is financial.
• Payment gateway keys: many providers accept their own idempotency key. Still store your local key too, so your order service and your gateway call share one business operation.

The hardest duplicate is not a retry minutes later; it is two identical attempts in flight at once. Maybe the user double-clicked, or an API gateway retried while the first request was still running. If both execute before either stores a result, idempotency has failed.

result = try_insert_key(scope, key, request_hash)

if result == "inserted":
    try:
        response = execute_mutation_once()
        mark_completed(scope, key, response)
        return response
    except Exception as error:
        mark_failed_or_delete_claim(scope, key, error)
        raise

existing = load_key(scope, key)
if existing.request_hash != request_hash:
    return 409 Conflict  # same key, different operation

if existing.status == "completed":
    return existing.stored_response

if existing.status == "in_progress":
    return 202 Accepted  # or wait briefly, then ask client to retry

Some systems block the second request until the first completes; others return 202 Accepted or 409 Conflict with a retry-after hint. The right choice depends on latency. For a sub-second order creation, waiting is friendly. For a multi-minute asynchronous workflow, returning a status endpoint is cleaner.

Keys are not globally magical. They need a scope, an expiry, and a rule for mismatched payloads. Without those details, a key from one user could collide with another user, storage would grow forever, or a buggy client could accidentally reuse a key for a different operation.

• Scope: store keys under something like tenant_id + user_id + endpoint + key. A key used for POST /orders should not affect POST /refunds.
• TTL: keep the record longer than all expected retries, queue redeliveries, and client reconnect windows. Payments often use at least 24 hours; business-critical workflows may keep keys for days.
• Payload hash: canonicalize the request body and store a hash. If the same key arrives with a different hash, return 409 Conflict instead of guessing which operation the client intended.
• Failure semantics: decide whether to cache failures. Validation errors can be replayed safely. Transient 500 errors may be better marked failed and retried through a recovery path.

Idempotency keys appear anywhere at-least-once delivery meets side effects. Stripe popularized the HTTP header for payments. Ecommerce systems use keys to protect order creation. Email providers use message IDs to avoid sending duplicates. Queue consumers use event IDs so a replayed Kafka or outbox event updates the database once.