Chapter 15: Idempotency and Duplicate Payments

Why This Exists

The internet is fundamentally unreliable. Wi-Fi drops, cell towers hand off connections, and servers occasionally crash. When a user clicks "Pay Now," that request travels across dozens of network nodes. If the connection drops, the user's browser doesn't know if the server received the request or not. The natural reaction is to click "Pay Now" again. Idempotency exists to guarantee that no matter how many times a system receives the exact same request, it only executes the financial transaction once, preventing the catastrophic error of charging a customer twice.

Real World Problem

A customer is buying a $2,000 laptop on their phone while on a train. They hit "Submit Order." The request reaches the server, the server successfully charges their credit card via Stripe, and the server tries to send the "Success" response back. At that exact moment, the train enters a tunnel, and the connection drops. The user's phone says "Network Error. Please try again." The user clicks "Submit Order" again. If the architecture is naive, the server processes this as a brand-new request and charges the user another $2,000.

Everyday Analogy

Think of an elevator button. If you walk up to an elevator and press the "Up" button, the elevator is called to your floor. If you are impatient and press the "Up" button 10 more times rapidly, the system doesn't call 10 elevators. The result of pressing it once is exactly the same as the result of pressing it 11 times. The elevator button is idempotent.

Beginner Explanation

Idempotency is a fancy word for a system having a memory of what it just did. Before the computer processes a payment, it checks a list: "Have I seen this exact request recently?" If the answer is No, it processes the payment and writes it down on the list. If the answer is Yes, it skips the payment and simply repeats the "Success" message it gave the first time.

Intermediate Explanation

In HTTP API design, some methods are naturally idempotent:

GET: Asking for product details 10 times doesn't change anything.
PUT / DELETE: Updating a user's name to "Bob", or deleting an item, has the same end state whether done once or 10 times.
POST: This is not naturally idempotent. POST /charge creates a new charge every time it is called.

To make a POST request idempotent, the client (the browser or mobile app) must generate a unique, random string (a UUID) called an Idempotency Key and send it in the HTTP Headers. The server uses this key to track the request.

Advanced Explanation

Implementing idempotency requires atomic database operations. When the server receives POST /checkout with Idempotency-Key: req_abc123:

The server attempts to insert req_abc123 into an idempotency_keys database table.
If the insert succeeds, the server knows this is a fresh request. It processes the payment, saves the final JSON response in the database alongside the key, and returns the response.
If the user retries, the server attempts to insert req_abc123 again. The database rejects it due to a UNIQUE constraint violation. The server catches this error, looks up the cached JSON response from the first attempt, and returns it immediately without hitting the payment gateway.

Real World Example

Stripe's API: Stripe handles billions of dollars and completely relies on idempotency. If your backend calls Stripe to charge a card, Stripe requires you to send an Idempotency-Key header. If your backend crashes, restarts, and sends the exact same charge request with the same key 5 minutes later, Stripe will not charge the card again. They will simply return the exact same HTTP response they generated the first time.

Architecture Design

Here is the flow of an Idempotent API request:

sequenceDiagram
    participant Client
    participant API
    participant DB(Idempotency Table)
    participant Gateway(Stripe)

    Client->>API: POST /charge (Header: Idem-Key=123)
    API->>DB(Idempotency Table): INSERT Key=123
    DB(Idempotency Table)-->>API: Success (New Key)
    
    API->>Gateway(Stripe): Charge Card
    Gateway(Stripe)-->>API: Success (Txn: 999)
    
    API->>DB(Idempotency Table): UPDATE Key=123 SET response='{"status":"ok"}'
    API-->>Client: 200 OK (Connection Drops here!)
    
    Note over Client,API: User clicks retry...
    
    Client->>API: POST /charge (Header: Idem-Key=123)
    API->>DB(Idempotency Table): INSERT Key=123
    DB(Idempotency Table)-->>API: Fails! (Duplicate Key)
    API->>DB(Idempotency Table): SELECT response WHERE Key=123
    DB(Idempotency Table)-->>API: Returns '{"status":"ok"}'
    API-->>Client: 200 OK (Without charging again)

Database Design

An Idempotency Table acts as a short-term memory cache.

CREATE TABLE idempotency_keys (
    key_hash VARCHAR(255) PRIMARY KEY,
    request_path VARCHAR(255), -- e.g., '/api/checkout'
    status VARCHAR(50), -- 'PROCESSING', 'COMPLETED', 'FAILED'
    response_code INT, -- e.g., 200
    response_body JSONB, -- The cached response to send back
    created_at TIMESTAMP,
    expires_at TIMESTAMP -- TTL, usually 24 hours
);

API Design

The API design doesn't change the URL or payload, it utilizes Headers.

Client Request:

POST /api/v1/orders HTTP/1.1
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
Content-Type: application/json

{ "cart_id": "12345" }

Production Considerations

Race Conditions on the First Request: If a user double-clicks a button, two identical requests hit the server at the exact same millisecond. If your code checks SELECT * FROM keys WHERE key=123 on both threads simultaneously, both will see it doesn't exist, and both will process the payment. You MUST use database-level locking (e.g., INSERT ... ON CONFLICT) or Redis SETNX to guarantee atomicity.
TTL (Time to Live): Idempotency keys should not be stored forever. They are designed to handle short-term network retries. Clear them out of the database after 24 hours to save space.

Security Considerations

Key Scoping: An Idempotency Key must be scoped to a specific User ID or Auth Token. If Hacker B guesses Hacker A's Idempotency Key, they shouldn't be able to retrieve Hacker A's cached checkout response containing their home address.

Common Mistakes

Relying purely on the Frontend: Disabling the HTML <button disabled="true"> after one click is good UX, but it does absolutely nothing to stop duplicate network requests caused by proxies, flaky connections, or malicious scripts. The backend must enforce idempotency.
Changing the Payload: If a client retries with the same Idempotency Key but changes the payload (e.g., changes the cart total from $50 to $10), the server must reject it with a 400 Bad Request. A key is bound strictly to the original payload.

Tradeoffs and Alternatives

Database Overhead vs. Safety: Checking an idempotency table adds 5-10 milliseconds of latency to every POST request. In read-heavy systems, this doesn't matter, but for high-frequency trading or massive ingest pipelines, it's a tradeoff. For e-commerce checkout, the safety against double-charging makes this overhead 100% mandatory.

Interview Questions

A user claims they were charged twice for the same order because they refreshed the page during checkout. Explain architecturally how you would prevent this.
What is an Idempotency Key, and who should generate it (the Client or the Server)?
Why is it dangerous to rely solely on frontend JavaScript to prevent duplicate form submissions?

Hands-On Exercise

Open a terminal and use curl to make a POST request to the Stripe API (using a test key) to create a customer, but include the header -H "Idempotency-Key: my_test_key_123".
Run the exact same curl command again.
Notice that Stripe does not create two customers, but returns the exact same customer object both times.

Key Takeaways

Idempotency guarantees that executing a request multiple times yields the same result as executing it once.
The Client must generate a unique Idempotency-Key (UUID) for critical POST requests.
The Server must use atomic database operations to catch duplicate keys, block the secondary execution, and return the cached original response.
Frontend button disabling is UX, not security or architecture.