Your Idempotency Check Has a Race in It

Safe to retry means nothing if two retries run at the same time.

Almost every team I talk to has an idempotency story, and it's almost always the same one. Give each operation a key. Look the key up. If you've seen it, return what you returned last time. If you haven't, do the work and save the result.

It's a good design. It also passes every test you'll write for it, because your tests run one request at a time.

Then production sends you two copies of the same message, half a millisecond apart. Both look up the key before either has written anything.

The check and the write are two different moments 🔗

Here is the sequence, with two threads instead of one.

Thread 1: SELECT ... WHERE key = 'abc'   -> not found
Thread 2: SELECT ... WHERE key = 'abc'   -> not found
Thread 1: process, then INSERT
Thread 2: process, then INSERT           <- charged the card twice

Neither thread did anything wrong. Both followed the design exactly. The design has a gap in the middle of it. It sits between the moment you check and the moment you act.

This has a name that predates messaging by decades. Time-of-check to time-of-use, TOCTOU. It shows up anywhere you read state, decide, then write on that decision without holding anything in between. A distributed cache has it. A database read followed by a write has it. Any read-modify-write that isn't a single atomic operation has it.

The window is small. It's also wide open, thousands of times a day, in a system doing real traffic.

This happens more often than the odds suggest 🔗

The usual reaction is that two retries landing in the same millisecond sounds unlikely. It would be, if retries were spread evenly. They aren't.

Duplicates arrive in bursts, because the things that cause them are bursty. A consumer's visibility timeout expires while it's still working. The broker hands that same message to a second consumer, mid-flight. A Kafka consumer group rebalances and reassigns partitions that were already being processed. A deployment rolls, and new instances pick up work the old ones hadn't acknowledged.

Every one of those produces duplicates that are concurrent by construction. They don't spread out politely over the next few minutes. Retry storms have the same shape. When a downstream service recovers, every client that was backing off comes back at once.

Scale the consumer out to ten instances and the effect multiplies. You didn't make duplicates more likely. You made them more likely to overlap, and overlap is the thing your idempotency check can't survive.

Make the check and the write one operation 🔗

The fix isn't a bigger lock or a longer transaction. Stop treating the check and the write as two steps that happen to sit near each other.

The simplest version is a unique constraint. Put one on the idempotency key and let the database referee the race.

INSERT INTO processed_messages (message_id, processed_at)
VALUES (@messageId, @now)
ON CONFLICT (message_id) DO NOTHING;

Then look at how many rows changed. One means you won and should do the work. Zero means someone else got there first, and the right thing to do is nothing.

var inserted = await connection.ExecuteAsync(InsertProcessedMessage, new { messageId, now });

if (inserted == 0)
{
    // Another delivery of this message is already being handled.
    return;
}

await DoTheActualWork();

That's the whole trick. The database does the check and the write in one indivisible step. No second thread can slip through the gap. Write it as ON CONFLICT DO NOTHING in Postgres. Or a unique index that throws in SQL Server, or SET NX in Redis. The property is the same: one operation, not two.

Optimistic concurrency reaches the same place from another direction. Include a version number or an ETag in the write, and require it to match what you read. If someone moved the row underneath you, your update affects zero rows and you know you lost. This fits better when you're updating existing state rather than recording that a message was seen.

One version of this looks right and isn't. Wrapping the SELECT and the INSERT in a transaction doesn't close the gap on its own. At read-committed isolation, two transactions can both read "not found" happily. You need the constraint, a stricter isolation level, or an explicit lock. The transaction alone isn't the atomicity you're looking for.

Losing the race is a normal outcome 🔗

Once the write is atomic, roughly half your concurrent duplicates will lose. You have to decide what losing looks like to the caller.

Returning the stored result of the first execution is the friendliest option, and the most expensive. You keep the response body around, keyed by the idempotency key, for as long as anyone might retry. Stripe does this, which is why their idempotency layer is a small state machine rather than a unique index.

Returning a 409 Conflict is honest and cheap. It hands the decision back to the caller. That's reasonable for a synchronous API, and unhelpful for a message handler with nobody to hand anything to.

Doing nothing at all is correct more often than people expect. Say your handler's only job is moving an order from pending to paid. The second delivery has no work left. Silence is the right response.

The rule underneath all three: the caller must not be able to tell whether it won or lost. Same status, same shape, same effect on the world. The moment a duplicate returns something visibly different, you've leaked your implementation into your contract.

Keys can't live forever 🔗

An idempotency table with no expiry policy grows until someone notices it in a disk alert.

Pick a retention window covering your realistic retry horizon, then delete beyond it. A day handles transport-level redelivery comfortably. A week covers manual replays and a long weekend of nobody watching the dead letter queue. Longer than that, and you're storing rows against a retry nobody will send.

The forgotten part is that expiry is a guarantee with an end date. After the TTL passes, the same key is a brand new request and the operation runs again. That's a business decision as much as a storage one. It belongs in your API documentation, next to everything else you promise.

What this means in practice 🔗

• Never SELECT then INSERT. Use a unique constraint, an atomic upsert, or a compare-and-set. One operation.
• Assume your duplicates are concurrent. Timeout expiry, rebalances, and deploys all produce overlapping deliveries.
• Decide what losing returns, and make it indistinguishable from winning.
• Give keys a TTL, then document when the guarantee expires.
• Reach for the stored-response state machine only when one logical operation spans several side effects. A unique constraint covers the rest.

If you're on MassTransit or NServiceBus, most of this exists already. Both frame their outbox support as a replacement for hand-rolled idempotency checks, not something you bolt on beside one. That's one less race condition to write yourself.

Closing 🔗

Idempotency is usually taught as a property of an operation. It's more useful as a property of an operation under contention. That's the only condition where it does any work.

A handler that's safe to retry sequentially is a handler that has never been tested. Run two copies at once and see what your database says.

How are you handling this in your stack?

idempotency concurrency messaging distributed-systems dotnet