A client sends POST /payments. The server charges the card, writes the row, and starts sending the response. Then the connection drops. From the client's point of view the request failed. From the server's point of view it succeeded. The client retries, and the customer is charged twice.
This is not an edge case. Mobile networks, load balancer timeouts, deploys and plain packet loss all produce it. Any API that creates something or moves money needs a clear answer to one question: what happens when the same request arrives twice?
What idempotent actually means
An operation is idempotent when performing it several times has the same effect on the server as performing it once. HTTP already defines GET, PUT and DELETE as idempotent methods: setting a resource to a value twice leaves it at that value, and deleting something twice leaves it deleted.
POST is the problem. "Create a payment" is naturally not idempotent; every call creates a new one. To make retries safe we have to give the server a way to recognise that two requests are the same logical operation.
Note that idempotency is about effects, not responses. The second DELETE might return 404 instead of 204; that is fine as long as nothing extra happened.
The idempotency key pattern
The client generates a unique key for each logical operation (a UUID is fine) and sends it with the request:
POST /payments HTTP/1.1
Idempotency-Key: 7f3c2a8e-0c4b-4b8e-9a51-3f1d2c6e9b10
Content-Type: application/json
{"amount": 4200, "currency": "EUR", "customer": "cus_123"}If the request is retried, the client sends the same key. The server stores the key together with the outcome of the first attempt and, on a repeat, returns that stored outcome instead of doing the work again.
Three rules make this work:
- The key is scoped. Store it per client or tenant, so two customers cannot collide.
- The key is bound to the request. Store a hash of the request body. If the same key arrives with a different body, that is a client bug and should fail with
422, not silently return the old result. - Keys expire. Retries happen within minutes or hours. Keeping keys for 24 hours is a common, reasonable window.
Handling concurrent duplicates
The subtle case is two identical requests arriving at the same time, for example a client that retries aggressively. Both check for the key, both see nothing, both do the work.
The fix is to claim the key atomically before doing the work, using the database's uniqueness guarantee:
CREATE TABLE idempotency_keys (
tenant_id uuid NOT NULL,
key text NOT NULL,
request_hash bytea NOT NULL,
status text NOT NULL DEFAULT 'in_progress', -- in_progress | completed
response_code int,
response_body jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (tenant_id, key)
);The handler inserts the row first. Exactly one request wins the insert; the others see a conflict and can either wait or return 409 Conflict telling the client to retry shortly.
func (s *Server) CreatePayment(ctx context.Context, tenant uuid.UUID, key string, req PaymentRequest) (Response, error) {
hash := sha256.Sum256(req.CanonicalJSON())
claimed, err := s.db.ClaimKey(ctx, tenant, key, hash[:])
if err != nil {
return Response{}, err
}
if !claimed {
existing, err := s.db.GetKey(ctx, tenant, key)
if err != nil {
return Response{}, err
}
switch {
case !bytes.Equal(existing.RequestHash, hash[:]):
return Response{Code: 422, Body: errorBody("idempotency key reused with a different request")}, nil
case existing.Status == "in_progress":
return Response{Code: 409, Body: errorBody("request with this key is still being processed")}, nil
default:
return Response{Code: existing.ResponseCode, Body: existing.ResponseBody}, nil
}
}
resp, err := s.chargeAndRecord(ctx, tenant, req)
if err != nil {
// Release the key so a retry can try again; the work did not complete.
_ = s.db.DeleteKey(ctx, tenant, key)
return Response{}, err
}
return resp, s.db.CompleteKey(ctx, tenant, key, resp.Code, resp.Body)
}ClaimKey is a single INSERT … ON CONFLICT DO NOTHING that reports whether a row was inserted.
Keep the side effects inside the transaction where you can
The example above still has a gap: if the process crashes after charging the card but before CompleteKey, the key stays in_progress and the outcome is unknown.
Where the work is purely database writes, the cleanest solution is to complete the key in the same transaction as the business writes. Either both commit or neither does.
When the work calls an external system, such as a payment provider, you cannot put it in your transaction. Two techniques help:
- Pass your idempotency key downstream. Many providers accept their own idempotency key. Deriving it from yours means the provider deduplicates your retries too.
- Record intent first, act later. Write the intended operation to an outbox table in the transaction, and let a worker perform the external call with retries. This is the transactional outbox pattern, and it turns "did it happen?" into "it will happen exactly once from the provider's point of view".
Idempotency beyond HTTP
The same thinking applies anywhere messages can be redelivered:
- Webhooks from providers are delivered at least once. Deduplicate on the provider's event ID before acting.
- Queue consumers may see a message again after a crash or a visibility timeout. Make the handler idempotent, or record processed message IDs.
- Batch imports that can be re-run should use upserts keyed on a natural identifier, not blind inserts.
In practice, "exactly once" delivery is rarely available. What you can build is at-least-once delivery plus idempotent processing, which has the same observable effect.
A checklist
- Every non-idempotent endpoint that clients may retry accepts an
Idempotency-Key. - Keys are scoped per tenant or client and bound to a hash of the request.
- The key is claimed atomically before work starts.
- Concurrent duplicates get a clear
409, mismatched bodies a422. - The stored response is returned on replay, including the original status code.
- Keys expire after a documented window.
- External side effects use downstream idempotency keys or an outbox.
- Webhook and queue consumers deduplicate on message IDs.
Designing this in from the first version costs very little. Retrofitting it after the first double charge is considerably more expensive, both in engineering time and in customer trust.



