Managing Long Running Tasks hard
For a task like 'Email all 1 million users', how do you avoid a single worker timing out?
- A Increase timeout to 24 hours.
- B Partition/Split the task. The initial worker creates 1000 sub-tasks (pages of 1000 users each) and pushes them to the queue. 1000 workers then process them in parallel.
- C Use a bigger server.
- D Use a GPU.
Answer: B. Partition/Split the task. The initial worker creates 1000 sub-tasks (pages of 1000 users each) and pushes them to the queue. 1000 workers then process them in parallel.
Divide and Conquer. Keeps individual tasks small enough to fit within visibility timeouts.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- What is the primary purpose of a Dead Letter Queue (DLQ)?
- The architectural pattern where one service creates tasks and another processes them is called:
- In a Priority Queue system (High vs Low), what is 'Starvation' and how do you fix it?
- How do you atomically update your local database AND publish a message to a queue without losing sync if the process crashes in between?
- What does 'Fire and Forget' mean?
- Why might you 'Throttle' (limit the speed of) your background workers?