Multi Step Processes hard
In Kafka, what happens if your message processing logic is too heavy and takes longer than the `max.poll.interval.ms`?
- A Kafka pauses the partition.
- B The consumer is presumed dead. The Broker triggers a 'Rebalance', kicking the consumer out of the group and assigning its partitions to others.
- C The message is skipped.
- D The consumer crashes.
Answer: B. The consumer is presumed dead. The Broker triggers a 'Rebalance', kicking the consumer out of the group and assigning its partitions to others.
This is a common production outage cause. If you take too long, the group coordinator thinks you died. It reshuffles partitions, pausing everyone. If the next node also takes too long, the loop repeats forever (Rebalance Storm).
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- A user registration process involves two steps: inserting a row into the 'Users' database table and inserting a row into the 'Profiles' table. If the first insert succeeds but the second fails, the database is left in an inconsistent state. What mechanism prevents this?
- In a multi-step workflow across different services (Order Service -> Payment Service), you cannot use a single database transaction. If the Payment fails after the Order was created, how do you fix the Order state?
- You are writing a script to process 10,000 images. The script runs on a server that might restart unexpectedly. To ensure you don't have to start over from image #1 after a crash, what should you implement?
- A client app sends a 'Pay $50' request to your API. The server processes the payment but the network response is lost. The client automatically retries the request. What prevents the user from being charged $100?
- A user uploads a large video file that needs to be transcoded (converted), which takes 5 minutes. Why is it bad practice to keep the HTTP connection open while the server processes the video?
- How does CDC (reading the database transaction log) solve the 'Dual Write' problem better than the Application-level Outbox pattern?