Design a distributed task queue
A system design prompt about distributing work, recovering from failures, and making retries safe.
The prompt
Design a task queue that accepts work from producers, distributes it across workers, and recovers when a worker fails. Tasks can take seconds or minutes. Producers should receive an acknowledgement once a task has been durably accepted.
Solution recording
No recording has been added to this example. A published walkthrough appears here once its YouTube video ID is included in the entry, and adds one square to the system design tracker.
Questions to establish first
- How many tasks arrive per second, and how bursty is the traffic?
- Does ordering matter globally, per customer, or not at all?
- How long can a task run, and how quickly must a failure be retried?
- Can tasks safely execute more than once?
A starting architecture
Producers write to a durable queue. Workers lease tasks for a limited period, execute them, and acknowledge completion. An expired lease makes a task available again.
Producer → API → Durable queue → Workers
↑ │
└── Retry ────┘The tradeoff to explain
An expired lease does not prove the previous worker stopped. At-least-once delivery requires duplicate-safe processing, especially when tasks call external services.
What to measure
Queue age, queue depth, execution duration, retry count, and tasks that exhaust their retry budget. Queue age is particularly useful: it describes how long real work is waiting.