Idempotency Explained: The Engineering Principle That Prevents Millions in Financial Losses

Why the same request must never charge a customer twice — and how to build systems that handle retries safely.

Melake Sisay

@melakesisay · about 3 hours ago · 30 min read

Idempotency Explained: The Engineering Principle That Prevents Millions in Financial Losses

Category: Backend Engineering · Distributed Systems · FinTech · Software Architecture
Reading time: ~34 minutes

"The customer clicked Pay only once."

Yet your system created:

Two orders
Two invoices
Two payment requests
Two emails
Two shipments

Somewhere between the user's click and your database, the same request was processed more than once. This is not a rare edge case. It is one of the most common — and expensive — problems in distributed systems.

The solution is a concept every backend engineer should master: idempotency.

If you build APIs, payment systems, banking applications, e-commerce platforms, booking systems, or any software that processes important actions, understanding idempotency is not optional. It is a fundamental engineering principle.

This article explores idempotency from first principles, explains why it matters, shows how it is implemented in production systems, and highlights the mistakes that even experienced developers make.

What Is Idempotency?

In simple terms:

An operation is idempotent if performing it multiple times has the same effect as performing it once.

Notice the wording. It does not mean the server only receives one request. It means processing the same logical request multiple times produces the same final state.

Consider turning on a light:

Turn ON
Turn ON
Turn ON
Turn ON

The light remains on. The final state does not change. That operation is idempotent.

Now imagine depositing money:

Deposit $100
Deposit $100
Deposit $100

The account balance increases every time. That operation is not idempotent.

The Mathematical Definition

Idempotency comes from mathematics. A function f is idempotent if applying it twice gives the same result as applying it once:

f(f(x)) = f(x)

In software, this translates to: executing the same operation repeatedly should converge to the same final state. The system may perform internal checks, but it must not produce side effects that accumulate with each retry.

Safe vs. Idempotent

A safe operation does not change state at all. GET is safe. An idempotent operation may change state, but repeating it does not add new effects. PUT and DELETE are idempotent but not safe. POST is neither safe nor idempotent by default.

Why Does This Matter?

Because networks are unreliable.

Applications crash.
Phones lose internet.
Browsers retry requests.
Load balancers resend packets.
Users double-click buttons.
Mobile apps retry automatically.
Cloud services occasionally timeout.
DNS resolution fails.
TLS handshakes stall.
Container orchestrators restart pods mid-request.
Wi-Fi drops in elevators.
Mobile networks switch between 4G and 5G.
CDN edge nodes retry origin requests.

Your backend will eventually receive duplicate requests. The real question is not will duplicate requests happen? The real question is: what happens when they do?

A Real Story

Imagine an online store. A customer buys a laptop. They click Pay. The request reaches the payment gateway. The gateway successfully charges the customer. But just before returning the response, the network connection drops.

The customer sees: Payment Failed.

Naturally, they try again. Now your system charges the customer twice. Not because your payment gateway failed. Not because your code had a syntax error. Because your application could not distinguish between a new payment and a repeated payment request.

The customer complains. The support team refunds the duplicate charge. The finance team reconciles the ledger. The engineering team writes a post-mortem. The marketing team manages a public apology. The cost of a single missing idempotency check ripples through the entire business.

That is exactly what idempotency solves.

The Cost of Duplicates

Duplicate operations create:

Financial loss — double charges, duplicate payouts, inflated liabilities.
Operational load — manual refunds, support tickets, reconciliation.
Trust erosion — customers lose confidence in the platform.
Compliance risk — regulators may question financial controls.
Engineering churn — teams firefight instead of building features.

A single high-profile duplicate transaction can generate more bad press than a hundred minor bugs.

HTTP Methods and Idempotency

Many developers first encounter idempotency while learning HTTP semantics. The HTTP specification defines idempotency as a property of request methods, not responses. A request method is idempotent if multiple identical requests have the same effect as a single request.

GET

GET /users/10

Whether you call it once or a thousand times, nothing changes. GET is safe and idempotent.

PUT

PUT /users/10
Content-Type: application/json

{
  "name": "John"
}

Calling it again produces the same final result. The resource is replaced with the same representation. PUT is idempotent.

However, PUT can be tricky if other clients are changing the same resource concurrently. If the representation you send does not include fields that another client updated, your PUT might overwrite those changes. For this reason, some teams prefer PATCH for partial updates with explicit field changes.

DELETE

DELETE /users/10
DELETE /users/10
DELETE /users/10

The resource is deleted. Deleting it again changes nothing. DELETE is idempotent. The server typically returns 204 on the first call and 404 on subsequent calls. Both are acceptable because the final state is the same.

POST

POST /orders
Content-Type: application/json

{
  "product_id": "123",
  "quantity": 1
}

This usually creates a new resource. Calling it twice often creates two orders. POST is not naturally idempotent.

That does not mean POST cannot be idempotent. It simply requires extra design, usually in the form of an idempotency key.

PATCH

PATCH is tricky. Some patch operations are idempotent, others are not. For example, replacing a field is idempotent:

PATCH /users/10
{ "name": "John" }

But incrementing a counter is not:

PATCH /users/10
{ "reputation": "+1" }

The second request would add another point. If you need idempotent increments, use a PUT with an absolute value or an idempotency key.

Method	Example	Idempotent?	Why
GET	`GET /users/10`	✅ Yes	Reading data does not change state.
PUT	`PUT /users/10` with `{ "name": "John" }`	✅ Yes	Repeated updates produce the same final state.
DELETE	`DELETE /users/10`	✅ Yes	The resource is deleted; deleting again changes nothing.
POST	`POST /orders`	❌ Not naturally	Each call may create a new resource.
PATCH	`PATCH /users/10` with partial updates	⚠️ Depends	Some patch operations are idempotent; others are not.

POST can be made idempotent, but it requires deliberate design.

The Hidden Enemy: Distributed Systems

In distributed systems, nothing is guaranteed. Consider this architecture:

Client
   │
   ▼
API Gateway
   │
   ▼
Order Service
   │
   ▼
Payment Service
   │
   ▼
Notification Service

Every network hop introduces uncertainty:

Did the request reach the payment service?
Did the payment service respond?
Did the response disappear?
Did Kafka publish the event?
Did RabbitMQ deliver twice?
Did the API gateway retry the request?
Did the load balancer send the same request to two different replicas?
Did the circuit breaker open and retry the call?

Nobody knows. Distributed computing is full of uncertainty. Idempotency brings certainty.

Failure Modes That Cause Duplicates

Client timeout with retry. The client waits 5 seconds, sees no response, and retries. Both requests reach the server.
Load balancer retries. Some proxies retry POST requests to a different backend if the first one fails to respond.
Message broker redelivery. Kafka, RabbitMQ, and SQS can deliver the same message more than once.
Database commit + response loss. The transaction commits, but the response to the caller is lost before delivery.
User impatience. A double-click or page refresh resubmits a form.
Mobile OS background retry. iOS and Android may retry network requests when the app returns to the foreground.
Webhook retries. External providers send webhooks multiple times if the first acknowledgment fails.

Each of these scenarios produces duplicate requests. Without idempotency, each duplicate creates a new side effect.

What Is an Idempotency Key?

This is where the magic begins.

Instead of relying on timing, clients send a unique identifier with each important request:

POST /payments
Idempotency-Key: 9c2fbb7d-f924-4f57-b8da-1f7b0e6fd340
Content-Type: application/json

{
  "amount": 10000,
  "currency": "USD",
  "order_id": "order-123"
}

The server stores it. When the same request arrives again, instead of executing the payment, the server simply returns the previous result. To the client, it appears as though the first response finally arrived.

Where Should the Key Come From?

The client generates the idempotency key. Common sources include:

UUIDv4 generated before the first request
A hash of the request payload plus a client-side nonce
A correlation ID already present in the frontend state
A deterministic key derived from the action, such as user-id:action:target:timestamp-slot

The key must be stable across retries. If the client generates a new key for each retry, idempotency cannot work.

Key Lifespan

A client should reuse the same key for the entire retry window. After the operation completes, the client may discard the key. The server, however, must keep the record for some period to handle late retries. Common retention windows:

Payment operations: 24 hours
Order creation: 24 to 72 hours
Webhook processing: 7 days
Email/SMS sending: 24 hours
Data imports: 30 days

A Typical Payment Flow

Without idempotency:

Click Pay → Payment Created → Timeout → Click Again → Second Payment Created

With idempotency:

Click Pay → Payment Created → Timeout → Retry → Same Idempotency Key → Return Existing Payment

Exactly one payment exists. No duplicate charge. No duplicate order. No duplicate shipment.

What Should Be Stored?

When processing an idempotent request, store more than just the key. A robust idempotency record looks like this:

Field	Purpose
`idempotency_key`	Unique request identifier supplied by the client.
`request_hash`	Hash of the payload to prevent the same key being reused with different data.
`response_body`	The exact response returned the first time.
`status_code`	The original HTTP status.
`user_id`	The request owner.
`created_at`	When the record was created.
`expires_at`	When the record can be safely deleted.

Storing the response is critical. Retries should receive the exact same response as the original request, including the same status code. This preserves the illusion that the first response finally arrived.

Why the Status Code Matters

A 201 Created response means the resource was created. A 200 OK response means the resource already existed. If a retry gets a different status code than the original, the client may behave incorrectly. For example, it might try to create a second resource because it no longer trusts the first response.

Request Hashing

Suppose someone sends:

{
  "key": "ABC123",
  "amount": 100
}

Later they send:

{
  "key": "ABC123",
  "amount": 500
}

Should that succeed? Absolutely not.

The server should compare a hash of the original request. If the payload changed, reject it. An idempotency key belongs to one logical request, not multiple different requests.

How to Compute the Hash

Normalize the request body and compute a stable hash:

import hashlib, json

def request_hash(payload: dict) -> str:
    canonical = json.dumps(payload, sort_keys=True, separators=(',', ':'))
    return hashlib.sha256(canonical.encode()).hexdigest()

Always use the same canonicalization rules. Different key ordering, whitespace, or number formatting will produce different hashes for the same logical payload.

What to Include in the Hash

Include every field that defines the logical operation. Do not include volatile metadata such as request timestamps, client trace IDs, or debug headers. If the hash changes because of a trace ID, the idempotency check becomes useless.

Database Design

A simple table for idempotency records might look like this:

CREATE TABLE idempotency_records (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  idempotency_key text NOT NULL,
  request_hash    text NOT NULL,
  response_body   jsonb NOT NULL,
  status_code     int NOT NULL,
  user_id         uuid NOT NULL REFERENCES auth.users(id),
  created_at      timestamptz NOT NULL DEFAULT now(),
  expires_at      timestamptz NOT NULL,
  UNIQUE (user_id, idempotency_key)
);

CREATE INDEX idx_idempotency_records_user_key
  ON idempotency_records(user_id, idempotency_key);

CREATE INDEX idx_idempotency_records_expires_at
  ON idempotency_records(expires_at)
  WHERE expires_at IS NOT NULL;

Key design decisions:

Unique constraint on (user_id, idempotency_key) — lets the same user retry safely, while different users can reuse the same key without collision.
JSONB response body — easy to replay the same response.
Expiration index — allows a cleanup job to remove stale records efficiently.

Handling Concurrent Duplicate Requests

Two duplicate requests may arrive at the same instant. The first one should execute the business logic; the second should wait and return the same result. A common pattern:

# Pseudocode
with db.transaction():
    row = insert_idempotency_record(
        key, request_hash, status='processing'
    )
    if row is None:
        # Another request is already processing.
        # Poll until it completes, then return its response.
        wait_for_existing_response(key)
        return existing_response

    try:
        result = execute_business_logic()
        store_response(key, result)
        return result
    except Exception:
        mark_failed(key)
        raise

The initial insert acts as a distributed lock. The first transaction wins; the second waits.

Retry Responses for In-Progress Requests

What if the first request is still processing when the retry arrives? The server should return a 409 Conflict with a clear message, or queue the retry to wait. Returning the same response too early is not possible because the result is not ready yet. A well-designed API documents this behavior.

Financial Systems Depend on It

Banks cannot afford duplicate transactions. Imagine:

Transfer → Timeout → Retry → Second Transfer

Without idempotency, the customer loses money. With idempotency, the bank simply returns:

Transfer Already Completed

Exactly once. Always.

Real-World Consequences

Duplicate financial operations cause:

Customer trust erosion
Regulatory scrutiny
Chargeback fees
Manual reconciliation work
Negative media coverage
Engineering hours spent on post-mortems
Potential fines from payment networks

A single idempotency bug can cost more than a year of senior engineering salary. The investment in proper idempotency design is tiny compared to the cost of getting it wrong.

Case Study: The Double-Withdrawal Bug

In 2018, a major bank suffered a public outage when a retry storm caused duplicate ATM withdrawals. Customers saw their balances debited multiple times for a single transaction. The root cause was a missing idempotency check on the withdrawal endpoint. The incident cost millions in remediation, customer compensation, and regulatory penalties.

Idempotency Beyond Payments

Many developers think idempotency is only for payment gateways. It is useful almost everywhere:

Creating orders — prevent duplicate purchases.
Booking flights — prevent two identical bookings.
Hotel reservations — avoid double reservations.
Food delivery — prevent restaurants receiving duplicate orders.
Sending emails — avoid customers receiving ten confirmation emails.
SMS notifications — prevent multiple OTP messages.
Invoice generation — ensure invoice numbers remain unique.
User registration — prevent duplicate accounts during retries.
Webhook processing — ignore duplicate webhook deliveries.
Inventory reservation — prevent overselling the same item twice.
Subscription changes — avoid billing a customer twice for the same upgrade.
Data imports — safely re-run import jobs without creating duplicates.
Push notifications — prevent the same alert from being sent repeatedly.
Loyalty point awards — do not credit a customer twice for the same action.
Referral code redemption — prevent a code from being used twice by the same user.

Case Study: Hotel Double-Booking

A booking platform receives a reservation request. The customer sees a spinner for ten seconds and clicks the button again. Without idempotency, two reservations are created. The hotel is overbooked. The customer is charged twice. The support team has to cancel one booking and refund one payment. All of this is avoided with a single idempotency key tied to the booking session.

Event-Driven Systems

Imagine RabbitMQ publishing a User Registered event:

User Registered → Email Sent

What if RabbitMQ delivers the event twice?

Without idempotency:

Welcome Email
Welcome Email

With idempotent consumers:

Already Processed → Ignore

Message brokers generally provide at-least-once delivery, not exactly-once delivery. Your consumers must be prepared to handle duplicate events safely.

Idempotent Consumer Pattern

A common approach for event consumers:

# Pseudocode
for event in message_queue:
    dedupe_key = f"{event.type}:{event.id}"
    if already_processed(dedupe_key):
        acknowledge(event)
        continue

    with db.transaction():
        insert_processed_event(dedupe_key)
        handle(event)

    acknowledge(event)

The key is to insert the dedupe record in the same transaction as the business effect. Then the event is processed exactly once from the system's perspective, even if the broker delivers it multiple times.

SQS and Visibility Timeout

Amazon SQS uses a visibility timeout. If a consumer does not process and delete a message within the timeout, SQS makes the message visible again and another consumer may pick it up. This is a classic source of duplicate processing. Idempotent consumers are essential when using SQS.

The Transactional Outbox Pattern

Idempotency becomes even more important when you need to publish events atomically with database changes. The transactional outbox pattern solves this:

Write the business update and the outgoing event to an outbox table in the same database transaction.
A separate relay process reads the outbox and publishes the event to the message broker.
The relay uses idempotency to avoid publishing the same event twice.

BEGIN;

INSERT INTO orders (id, total, status) VALUES ('order-123', 10000, 'confirmed');
INSERT INTO outbox (id, topic, payload) VALUES ('evt-456', 'orders.created', '{...}');

COMMIT;

Because both inserts happen in one transaction, the event is guaranteed to be published if and only if the business update succeeds. The relay then ensures the event is published exactly once, using idempotency keys and broker deduplication.

Why Not Just Publish Directly?

If you publish the event before committing the database transaction, the event might be published but the transaction rolls back. If you publish after committing, the service might crash before publishing. The outbox pattern removes this dilemma by making the event a durable part of the same transaction.

Common Mistakes

Mistake 1: Using timestamps as keys

2026-08-03-10-00

Not unique enough. Two requests in the same second will collide.

Mistake 2: Generating keys on the server

The client should generate and reuse the key for retries. If the server generates the key, the client cannot safely retry. The server may return a key in the response, but that is for tracking, not for retry safety.

Mistake 3: Only storing the key

Store the response, too. Otherwise, the retry will not receive the same result as the original request.

Mistake 4: Never expiring old keys

Old keys should eventually be cleaned up based on your business requirements. A 24-hour window is common for payments; longer for other systems. Without cleanup, the idempotency table grows indefinitely and queries slow down.

Mistake 5: Ignoring request hashes

The same key must not be reused with a different request body. Always validate the request hash.

Mistake 6: Forgetting the lock

Two duplicate requests may arrive at the exact same moment. Use a short-lived database lock or atomic insert to ensure only one of them executes the business logic.

Mistake 7: Idempotency keys scoped globally

A key should be scoped to the user or tenant. Two different users might legitimately send the same UUID as their key. If you enforce global uniqueness, you create a subtle bug where unrelated users block each other.

Mistake 8: Returning different responses for the same key

If the business logic changed between the first request and the retry, the retry might get a different result. This breaks the contract. Store the response and replay it.

Mistake 9: Not testing retries

Many teams implement idempotency but never test the retry path. The code looks correct but fails in production because of a race condition or a missing hash check. Always include retry tests in your test suite.

Mistake 10: Forgetting idempotency in callbacks

External payment providers send callbacks. If your callback handler is not idempotent, a delayed callback may trigger a duplicate shipment or a second email. The callback itself should be deduplicated just like any other request.

"Exactly Once" Is Mostly a Myth

Many systems advertise exactly-once delivery. In reality, distributed systems rarely guarantee true end-to-end exactly-once processing.

Instead, engineers combine:

Retries
Idempotent APIs
Idempotent consumers
Deduplication
Transactional outbox patterns
Reliable messaging
Fenced producers in Kafka

Together, these techniques produce behavior that is effectively "exactly once" from the user's perspective.

Kafka and Idempotency

Kafka 0.11 introduced an idempotent producer. The producer attaches a unique producer ID and sequence number to each message. The broker discards duplicates caused by producer retries. This gives exactly-once semantics at the producer-broker level, but consumers still need to be idempotent because they may be restarted or rebalanced.

Database Unique Constraints

Sometimes the simplest idempotency mechanism is a unique constraint. If an order has a unique client_order_id, the database rejects duplicates naturally. This is not a replacement for idempotency keys in all cases, but it is a powerful complementary tool.

Designing for Failure

One of the biggest mindset shifts in backend engineering is this:

Do not design systems assuming everything works. Design them assuming everything can fail.

Requests will timeout. Networks will partition. Servers will restart. Messages will be duplicated. Databases will become temporarily unavailable. Third-party APIs will return 500 errors at 2 AM on a Saturday. Container orchestrators will kill pods mid-transaction.

Idempotency is not about preventing failures. It is about ensuring failures do not create inconsistent outcomes.

The Resilience Triangle

Reliable systems combine three ideas:

Retry — try again when something fails.
Timeout — give up when something takes too long.
Idempotency — make retries safe.

Without idempotency, retries become dangerous. Without retries, temporary failures become permanent errors. All three are needed.

Retry Strategies

Not all retries are created equal. Use exponential backoff with jitter to avoid thundering herds. Combine retries with circuit breakers to prevent cascading failures. And always ask: if this retry succeeds, is the system still in a correct state?

# Exponential backoff with jitter
import random, time

for attempt in range(max_retries):
    try:
        return call_api()
    except TransientError:
        sleep = (2 ** attempt) + random.uniform(0, 1)
        time.sleep(sleep)

The Philosophy Behind Idempotency

Junior engineers often ask: "How do I stop duplicate requests?"

Experienced engineers ask: "How can my system behave correctly even if duplicate requests happen?"

That is the essence of resilient software. Instead of fighting reality, you design for it.

A Practical Design Checklist

Before shipping a write endpoint, ask:

If you can answer yes to the relevant questions, your endpoint is well-prepared for the realities of distributed systems.

Final Thoughts

Idempotency is not just a payment gateway feature. It is not just an HTTP concept. It is not just another backend buzzword.

It is one of the foundational ideas behind reliable software.

Every time you build a payment API, create an order, process a webhook, consume a message queue, or expose a public endpoint, ask yourself one question:

If this exact request arrives again in the next second, will my system still produce the correct outcome?

If the answer is yes, you have built more than an API. You have built a system that users — and businesses — can trust.

Because in distributed systems, success is not measured by how often things go right. It is measured by how gracefully your software behaves when things inevitably go wrong.

Keep building resilient systems.

How Real APIs Implement Idempotency

Stripe

Stripe's API is a canonical example of idempotency done right. Every write request accepts an Idempotency-Key header. The first request executes the operation. Subsequent requests with the same key return the same response without executing the operation again.

Stripe stores the response for at least 24 hours. If a retry arrives while the original request is still processing, Stripe returns a 409 Conflict. The client can then poll or retry later.

The key takeaway: the idempotency key belongs to the client, and the response is stored for replay.

AWS

Many AWS APIs support client tokens for idempotency. For example, EC2's RunInstances accepts a ClientToken parameter. If you call it twice with the same token, AWS creates only one set of instances. This prevents accidental duplicate infrastructure deployments.

AWS documentation explicitly states that the client must provide the token and must reuse it across retries. The server does not invent the token on the client's behalf.

PayPal

PayPal's REST API includes an PayPal-Request-Id header for idempotency. Payment, payout, and order operations use this header. PayPal recommends that the client generate a unique UUID for each distinct request and reuse the same UUID for retries.

What They All Have in Common

The client generates the key.
The server stores the response.
The server rejects mismatched payloads.
There is a defined retention window.
In-progress requests return a distinct status.

These patterns form a gold standard for idempotent API design.

A Production-Ready Implementation Walkthrough

Let us design a simple but robust idempotent payment endpoint. The goal is to allow a client to retry a payment without creating duplicates.

Step 1: Define the API Contract

POST /payments
Idempotency-Key: <client-generated-uuid>
Content-Type: application/json

{
  "amount": 10000,
  "currency": "USD",
  "source": "card_123"
}

Response on success:

HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": "pay_abc",
  "status": "succeeded",
  "amount": 10000,
  "currency": "USD"
}

Response on retry:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": "pay_abc",
  "status": "succeeded",
  "amount": 10000,
  "currency": "USD"
}

Notice the status code difference. The original request returns 201. The retry returns 200. Both bodies are identical.

Step 2: Store the Request

When the server receives the request, it first attempts to create an idempotency record with a status of processing. If another request with the same key is already processing, the server waits. If a completed record exists, the server returns the stored response.

Step 3: Execute the Business Logic

If the server wins the lock, it executes the payment. This typically involves calling a payment processor, updating the order status, and creating ledger entries. All of these operations should be wrapped in a single database transaction where possible.

Step 4: Store the Response

After the business logic completes, the server stores the response in the idempotency record. The status changes from processing to completed. Now any retry will receive the stored response.

Step 5: Handle Failures

If the business logic fails, the server must decide whether to remove the processing record or leave it with a failure status. Removing the record allows the client to retry safely. Leaving it with a failure status requires the client to use a new key.

A common pattern is to mark the record as failed and include the error response. The client can then either retry with the same key (if the failure was transient) or use a new key for a fresh attempt.

Testing Idempotency

Testing idempotency requires more than happy-path unit tests. You need to simulate the failure modes that cause duplicate requests.

Unit Tests

Same key, same payload → returns stored response.
Same key, different payload → returns 409 or 422.
Different key, same payload → executes normally.
Missing key → executes normally or rejects depending on policy.

Concurrency Tests

Two requests with the same key arrive simultaneously. Only one should execute the business logic. The other should wait and return the same response.

Integration Tests

Kill the server mid-transaction. Restart it. Retry the same key. Verify the response is consistent and no duplicate side effects occurred.

Load Tests

Send a burst of identical requests under load. Verify that the system creates exactly one resource and returns consistent responses.

Chaos Tests

Introduce network latency and dropped responses. Simulate timeouts and retries. Confirm that the final state remains correct.

Security Considerations

Idempotency keys are not security controls, but they interact with security in important ways.

Scope Keys to the User

A global key namespace can be exploited. If user A sends a key that user B already used, and your system enforces global uniqueness, user B's retry might be blocked. Always scope idempotency keys to the authenticated user or tenant.

Do Not Expose Internal State

When returning a response for a retry, do not leak internal processing details. The response should be identical to the original response. Do not expose transaction IDs, internal error messages, or stack traces.

Validate the Payload

Always validate the request body before checking idempotency. A malformed request should fail validation even if the key matches a previous successful request. Idempotency is not a bypass for input validation.

Rate Limiting

Idempotency does not replace rate limiting. A malicious client could flood your API with retries even if the operation is idempotent. Apply rate limits at the API gateway or application layer.

Regulatory Context: Payments and PSD2

In Europe, the Revised Payment Services Directive (PSD2) places strong requirements on payment initiation and execution. While PSD2 does not mention idempotency by name, its requirements for secure execution, strong customer authentication, and transaction integrity make idempotency a practical necessity.

A duplicate payment under PSD2 can trigger:

Customer complaints to the national regulator
Reversal requests under the payment services rules
Reputational damage to the payment institution
Audit findings during compliance reviews

Financial regulators expect that institutions have controls to prevent duplicate transactions. Idempotency keys are one of the most direct technical controls.

Microservices and Idempotency

In a microservices architecture, idempotency must be considered at every boundary. A single user action may traverse multiple services, and each service may retry calls to its dependencies.

Synchronous Calls

When service A calls service B synchronously, both should use idempotency keys. Service A generates a key for the overall operation. Service B may accept that key or generate its own. The important thing is that retries do not cause duplicate effects.

Asynchronous Workflows

In asynchronous workflows, each event should carry an idempotency key. Consumers use that key to deduplicate processing. This works well with the transactional outbox pattern.

Sagas and Compensating Transactions

Long-running business processes often use sagas. Each step in a saga should be idempotent so that retries do not corrupt the overall state. If a step fails, the saga may execute a compensating transaction. That compensation should also be idempotent.

Partial Failures and Idempotency

A partial failure is one of the hardest scenarios to handle. Imagine a payment that succeeds but the notification email fails. Should the retry reprocess the payment? Should it only resend the email?

Idempotency Records Help

If the idempotency record already contains a successful payment response, the retry should not reprocess the payment. It can, however, trigger the missing notification as a separate concern. This is where separation of concerns matters: the payment is idempotent; the notification is a separate operation with its own retry logic.

Reversibility

Some operations are not naturally idempotent but can be made reversible. For example, a reservation can be canceled if the follow-up fails. However, the cancellation itself must also be idempotent. The goal is always to converge to a known good state.

Observability and Monitoring

You cannot manage what you cannot measure. Add observability around idempotency to detect issues before they become incidents.

Metrics to Track

Idempotency hits — requests matched to an existing record.
Idempotency misses — new idempotency keys.
Hash mismatches — same key but different payload.
Concurrent collisions — two requests for the same key arriving simultaneously.
In-progress retries — retries that hit a still-processing record.
Expired key lookups — retries after the retention window.
Duplicate events processed — duplicate messages in consumers.

Alerts

Alert on sudden spikes in hash mismatches or concurrent collisions. These may indicate a bug in the client, a replay attack, or an unexpected retry pattern.

Tracing

Include the idempotency key in distributed traces. This makes it easy to follow a single logical request across services, retries, and message brokers.

Frequently Asked Questions

Should every API endpoint accept an idempotency key?

No. Only endpoints that create or mutate important state need idempotency keys. Read endpoints are naturally idempotent. Trivial state changes, like updating a session timestamp, may not need them.

What happens if a client loses the idempotency key?

The client should generate a new key and treat the operation as a fresh request. This may create a duplicate if the previous operation actually succeeded, but it is the best the client can do without the key.

Can I use the database primary key as the idempotency key?

Only after the resource has been created. Before creation, the resource does not have a primary key, so you still need a client-generated key. After creation, the primary key can serve as a natural deduplication mechanism.

Should I delete idempotency records immediately after the response?

No. Retries may arrive later, especially from mobile clients or message brokers. Keep records for a retention window that matches your business requirements.

Is idempotency the same as deduplication?

They are closely related. Deduplication is the act of removing duplicates. Idempotency is the property of an operation that makes duplicates safe. You often implement idempotency using deduplication techniques.

A Final Checklist for Engineers

Before you ship your next write endpoint, run through this checklist one more time:

If you follow this checklist, you will prevent the kind of duplicate-operation incidents that cost companies millions and erode customer trust.

Keep building resilient systems.

Code Example: Building an Idempotent Endpoint

Let us walk through a concrete implementation using a Python-like pseudocode. The same principles apply in Node.js, Go, Java, or any backend language.

The Database Layer

from dataclasses import dataclass
from typing import Optional
import json
import hashlib

@dataclass
class IdempotencyRecord:
    key: str
    user_id: str
    request_hash: str
    response_body: dict
    status_code: int
    status: str  # 'processing' | 'completed' | 'failed'

class IdempotencyStore:
    def create_or_get(self, key: str, user_id: str, request_hash: str) -> Optional[IdempotencyRecord]:
        """
        Try to insert a processing record. Return None if a record already exists.
        """
        try:
            db.execute(
                """
                INSERT INTO idempotency_records
                (idempotency_key, user_id, request_hash, response_body, status_code, status, expires_at)
                VALUES (%s, %s, %s, '{}', 0, 'processing', now() + interval '24 hours')
                ON CONFLICT (user_id, idempotency_key) DO NOTHING
                RETURNING *
                """,
                (key, user_id, request_hash)
            )
            return IdempotencyRecord(**db.fetchone())
        except UniqueViolation:
            return None

    def get(self, key: str, user_id: str) -> Optional[IdempotencyRecord]:
        row = db.execute(
            "SELECT * FROM idempotency_records WHERE user_id = %s AND idempotency_key = %s",
            (user_id, key)
        ).fetchone()
        return IdempotencyRecord(**row) if row else None

    def save_response(self, key: str, user_id: str, response_body: dict, status_code: int):
        db.execute(
            """
            UPDATE idempotency_records
            SET response_body = %s, status_code = %s, status = 'completed'
            WHERE user_id = %s AND idempotency_key = %s
            """,
            (json.dumps(response_body), status_code, user_id, key)
        )

    def save_failure(self, key: str, user_id: str, response_body: dict, status_code: int):
        db.execute(
            """
            UPDATE idempotency_records
            SET response_body = %s, status_code = %s, status = 'failed'
            WHERE user_id = %s AND idempotency_key = %s
            """,
            (json.dumps(response_body), status_code, user_id, key)
        )

The Request Handler

import uuid
from flask import Flask, request, jsonify

app = Flask(__name__)
store = IdempotencyStore()

@app.route('/payments', methods=['POST'])
def create_payment():
    user_id = get_authenticated_user_id()
    idempotency_key = request.headers.get('Idempotency-Key')

    if not idempotency_key:
        return jsonify({"error": "Idempotency-Key header is required"}), 400

    try:
        uuid.UUID(idempotency_key)
    except ValueError:
        return jsonify({"error": "Idempotency-Key must be a valid UUID"}), 400

    payload = request.get_json()
    request_hash = compute_request_hash(payload)

    # Check for an existing record.
    existing = store.get(idempotency_key, user_id)
    if existing:
        if existing.request_hash != request_hash:
            return jsonify({"error": "Idempotency key reused with different payload"}), 422

        if existing.status == 'completed':
            return jsonify(existing.response_body), existing.status_code

        if existing.status == 'processing':
            return jsonify({"error": "Request is still being processed"}), 409

        if existing.status == 'failed':
            # Allow retry of failed requests by removing the failed record.
            store.delete(idempotency_key, user_id)

    # Try to claim the idempotency lock.
    record = store.create_or_get(idempotency_key, user_id, request_hash)
    if record is None:
        # Another request is processing. Wait briefly and return the stored result.
        for _ in range(10):
            time.sleep(0.2)
            existing = store.get(idempotency_key, user_id)
            if existing and existing.status == 'completed':
                return jsonify(existing.response_body), existing.status_code
            if existing and existing.status == 'failed':
                return jsonify({"error": "Original request failed"}), 409
        return jsonify({"error": "Request is still being processed"}), 409

    try:
        # Execute the business logic.
        payment = process_payment(payload)
        response_body = {
            "id": payment.id,
            "status": payment.status,
            "amount": payment.amount,
            "currency": payment.currency,
        }
        store.save_response(idempotency_key, user_id, response_body, 201)
        return jsonify(response_body), 201
    except TransientError as e:
        store.save_failure(idempotency_key, user_id, {"error": str(e)}, 503)
        return jsonify({"error": str(e)}), 503
    except Exception as e:
        store.save_failure(idempotency_key, user_id, {"error": str(e)}, 500)
        return jsonify({"error": str(e)}), 500

Key Observations

The handler always validates the key format and payload hash before checking idempotency.
The idempotency insert acts as a distributed lock.
Completed responses are replayed exactly.
Failed requests can be retried with the same key or a new one, depending on the desired policy.
In-progress requests return 409 so the client knows to wait or retry later.

This pattern is not tied to Python or Flask. The same logic can be expressed in TypeScript with Express, Go with Gin, Java with Spring, or Rust with Axum.

Idempotency and Ethiopian Fintech

For developers building payment systems in Ethiopia, idempotency is especially important. Mobile money integrations, bank transfers, and POS systems often operate over networks with variable reliability. A merchant's payment request may timeout, the customer may retry, and the gateway may send delayed callbacks.

Without idempotency, the same purchase can result in multiple charges on Telebirr, CBE Birr, or bank accounts. This creates reconciliation nightmares for merchants and erodes trust in digital payments. Implementing idempotency keys from day one is a small upfront cost that prevents large operational debt.

When integrating with local gateways, always ask:

Does the gateway provide an idempotency mechanism?
If not, can you build one on your side using an order ID or transaction reference?
Are callbacks deduplicated?
Do retries use the same reference number?

These questions separate production-ready integrations from fragile ones.

Keep building resilient systems.

Comments · 0

Be the first to comment.