Queues, streams and idempotency
Asynchronous work is at-least-once in practice, so every consumer must be idempotent and every pipeline needs a poison-message plan.
Use a queue when each message is a task for one worker and order mostly does not matter. Use a log/stream when many consumers need the same ordered history, or you need replay. In both, delivery is at-least-once: the network can lose an acknowledgement after the work was done.
Make consumers idempotent
- Give every operation a stable idempotency key (client-generated request id, or event id).
- Record processed keys in the same transaction as the side effect (an inbox table), or make the side effect naturally idempotent (
SET status = 'paid', notbalance = balance - 10). - For outbound events, use the transactional outbox: write the event to an outbox table in the same transaction as the state change, then relay it.
Exactly-once, honestly
"Exactly-once" usually means effectively once: at-least-once delivery plus idempotent processing. Some streaming systems offer transactional exactly-once within the system; it does not extend to your email provider or a payment API. Saying this precisely is a reliable Staff signal.
Operations
- Dead-letter queue with an alert and a replay tool.
- Backpressure: bounded queues, consumer autoscaling on lag (KEDA on Kubernetes scales on queue depth or consumer lag).
- Ordering: only within a partition or session key; choose that key on purpose.
Practice
Written for this site to drill the lesson. Not reported interview questions.
- Design order-placed to email-and-invoice processing so a retried message never sends two invoices.
- A consumer falls 6 hours behind after a deploy. Walk through detection, mitigation and the fix.
Check yourself
What does the transactional outbox solve?
The dual-write problem: state change and event publication commit together, so neither can happen without the other.
What does "exactly-once" usually mean in practice?
At-least-once delivery plus idempotent processing.