deskthebuildDISTRIBUTED SYSTEMS
← All notes · 5 min read

Duplicate events are creating repeated orders. How would you design idempotency?

Suppose a hypothetical order service consumes messages from a broker that provides at-least-once delivery. A worker creates an order and commits it, then crashes before acknowledging the message. The broker redelivers the event, and another worker creates a second order.

The broker is behaving correctly. Redelivery is how it avoids losing work. The consumer must make repeated delivery of one logical event safe.

Why the common answer is incomplete

Most candidates answer: “Deduplicate messages in Redis.”

A cache can reduce repeated work, but checking it and writing the order are separate operations. A crash between those operations still creates a duplicate or incorrectly suppresses work. Cache eviction and failover can also forget older keys while delayed messages remain deliverable. Correctness must be tied transactionally to the business mutation.

A production design gives each event a stable identity and records consumption in the same consistency boundary as the resulting state change.

A production approach

1. Define the logical event identity

Producers assign an immutable event ID when the business event is created, not each time it is published. Retries preserve that ID. Include aggregate identity and version when ordering matters. Do not deduplicate from the entire payload hash alone; equivalent serialization can change, and two legitimate events may contain the same values.

2. Persist producer intent with an outbox

Write the business state and an outbox event in one database transaction. A relay publishes the outbox record and marks delivery progress. Publishing may happen more than once, which is acceptable because consumers are idempotent. The outbox closes the gap where a producer commits an order update but crashes before sending its event.

3. Use a consumer inbox in the business transaction

Before applying an event, insert its ID into an inbox table with a uniqueness constraint in the same transaction as the business mutation. If the insert conflicts, return the previously recorded outcome and acknowledge the duplicate. If the transaction rolls back, neither the inbox marker nor the mutation remains, so redelivery can safely try again.

flowchart LR A[Producer transaction] --> B[Business state plus outbox] B --> C[At least once publish] C --> D[Consumer receives event] D --> E{Inbox ID exists?} E -->|Yes| F[Acknowledge duplicate] E -->|No| G[Inbox plus business mutation] G --> H[Commit transaction] H --> I[Acknowledge event]

4. Make handlers deterministic from durable state

Validate schema and authorization context, then derive the mutation from the event and current business state. Avoid external calls inside a long database transaction. When another service must be notified, write another outbox record as part of the transaction. This extends reliable intent without pretending a distributed transaction exists across unrelated systems.

5. Handle ordering independently from duplication

Deduplication prevents the same event twice; it does not prevent version 7 arriving before version 6. Store the last applied aggregate version and reject, buffer, or reconcile gaps according to the domain. Partitioning broker traffic by aggregate can preserve order within a partition, but replays and producer bugs still require version checks.

6. Define the deduplication retention window

Keep inbox records at least as long as messages can be retried, replayed, or restored from backup. For permanent business identifiers such as payment requests, the idempotency record may need to live with the business record. Archive old technical metadata only after confirming a delayed duplicate cannot recreate the effect.

7. Reconcile downstream reality

Periodically compare produced events, inbox consumption, and resulting business state. Detect outbox records never published, events stuck in dead-letter handling, and business objects without their expected follow-up. Idempotency prevents duplicates but does not guarantee every logical event completed. Reconciliation is the repair mechanism for gaps and software defects.

8. Test crashes at every boundary

Inject failure before and after outbox commit, publish, inbox insert, business mutation, commit, and acknowledgement. Replay events concurrently and out of order. Verify one logical order exists and every duplicate returns the same result. Happy-path integration tests rarely reveal the narrow crash windows that idempotency is designed to close.

Engineering trade-offs

Inbox and outbox tables add writes, indexes, cleanup, and operational monitoring. Using the business database gives transactional correctness but may increase load. A separate deduplication store scales independently while requiring a distributed consistency strategy. Ordered buffering improves correctness for some domains but can block later events behind one missing version.

“Exactly once” processing is often a system-level outcome built from at-least-once delivery and idempotent effects, not a broker flag. Some analytics operations are naturally commutative and can tolerate duplicates through aggregation design. Orders, payments, and notifications usually require explicit identity and durable deduplication.

Common mistakes

  • Creating a new event ID on every publish retry.
  • Checking a cache before the mutation without a transaction.
  • Acknowledging the broker before durable business commit.
  • Assuming deduplication also solves out-of-order delivery.
  • Calling another service inside the consumer transaction and assuming rollback undoes it.
  • Deleting inbox keys sooner than the broker’s replay window.
  • Testing duplicate events without testing crashes between commit and acknowledgement.

Failure handling and observability

Trace event ID, producer, aggregate and version, broker offset, handler version, inbox result, transaction outcome, attempt count, and downstream outbox IDs. Monitor redelivery rate, deduplication hits, version gaps, handler latency, transaction conflicts, dead-letter age, outbox lag, and reconciliation discrepancies. A rising duplicate rate may indicate unstable consumers even when business state remains correct.

On transient failure before commit, do not acknowledge; let bounded redelivery occur. On invalid schema or unrecoverable business conflict, record a quarantined outcome and route it for review rather than retrying forever. If acknowledgement fails after commit, redelivery should hit the inbox record and complete cheaply. Recovery procedures must preserve event IDs during replay.

Interview insight

The interviewer is not testing whether you know a broker’s delivery setting. They are testing whether you understand atomic state transitions across failure boundaries. A staff-level answer accepts redelivery, couples deduplication to the business effect, separates ordering from duplication, and builds reconciliation for missing outcomes.