Multi Step Processes hard
How does Kafka achieve 'Exactly-Once Processing' (EoS) when consuming from one topic and producing to another (read-process-write)?
- A It uses idempotent producer IDs combined with transactions that write the output message and the consumer offset commit marker to the log atomically.
- B It uses a two-phase commit between the Consumer and the Producer.
- C It relies on the database to deduplicate messages.
- D It deletes the message from the source topic immediately after reading.
Answer: A. It uses idempotent producer IDs combined with transactions that write the output message and the consumer offset commit marker to the log atomically.
Kafka EoS writes the 'result message' and the 'I am done with input message X' marker in a single transaction. If the crash happens, both or neither are visible to downstream consumers.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- A user registration process involves two steps: inserting a row into the 'Users' database table and inserting a row into the 'Profiles' table. If the first insert succeeds but the second fails, the database is left in an inconsistent state. What mechanism prevents this?
- In a multi-step workflow across different services (Order Service -> Payment Service), you cannot use a single database transaction. If the Payment fails after the Order was created, how do you fix the Order state?
- You are writing a script to process 10,000 images. The script runs on a server that might restart unexpectedly. To ensure you don't have to start over from image #1 after a crash, what should you implement?
- A client app sends a 'Pay $50' request to your API. The server processes the payment but the network response is lost. The client automatically retries the request. What prevents the user from being charged $100?
- A user uploads a large video file that needs to be transcoded (converted), which takes 5 minutes. Why is it bad practice to keep the HTTP connection open while the server processes the video?
- How does CDC (reading the database transaction log) solve the 'Dual Write' problem better than the Application-level Outbox pattern?