Kafka medium
How does the Kafka Group Coordinator distinguish between a 'slow processing' consumer and a 'dead' consumer instance?
- A It pings the consumer's IP address every 10 seconds to check for OS-level responsiveness.
- B It uses two separate timeouts: `session.timeout.ms` monitors the heartbeat thread (connection health), while `max.poll.interval.ms` monitors the actual message processing frequency (application health).
- C It checks the consumer's CPU usage via the Prometheus JMX exporter.
- D It requires the Consumer to send a 'Finish' signal after every single message batch.
Answer: B. It uses two separate timeouts: `session.timeout.ms` monitors the heartbeat thread (connection health), while `max.poll.interval.ms` monitors the actual message processing frequency (application health).
Separating heartbeats from processing prevents 'false rebalances' if a consumer is doing heavy computations. If heartbeats arrive, the broker knows the node is 'up'. if `poll()` stops, it knows the application code is 'stuck' or processing too slowly.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- When implementing 'Exactly-Once Semantics' (EOS) in a Kafka stream, what is the role of the `isolation.level=read_committed` setting on the Consumer?
- Historically, which external service was required by Kafka to manage cluster hierarchy, metadata, and leader election status?
- Once a message is successfully appended to a Kafka partition log, what is its mutability status?
- In a Kafka cluster, what is the role of a 'Broker'?
- What is the primary indicator of a bottleneck in a Kafka consumer's processing logic?
- What is the primary architectural role of Apache Kafka in a distributed system?