Kafka medium
Modern Kafka stores consumer bookmarks (offsets) in an internal compacted topic. What is the technical name and rationale for this topic?
- A `_consumer_history`: it provides a searchable audit log for legal compliance.
- B `consumer_cache`: it is stored purely in RAM to maximize speed.
- C `__consumer_offsets`: Storing state within Kafka itself provides better scalability and durability than legacy methods (like ZooKeeper), while compaction ensures only the latest bookmark per consumer is kept.
- D `system_offsets`: to allow the Producers to know exactly where the consumers are.
Answer: C. `__consumer_offsets`: Storing state within Kafka itself provides better scalability and durability than legacy methods (like ZooKeeper), while compaction ensures only the latest bookmark per consumer is kept.
This was a major architectural shift. By treating offsets as just another Kafka topic, the system can handle the massive write-load of millions of consumers committing their progress simultaneously across a large cluster.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- When implementing 'Exactly-Once Semantics' (EOS) in a Kafka stream, what is the role of the `isolation.level=read_committed` setting on the Consumer?
- Historically, which external service was required by Kafka to manage cluster hierarchy, metadata, and leader election status?
- Once a message is successfully appended to a Kafka partition log, what is its mutability status?
- In a Kafka cluster, what is the role of a 'Broker'?
- What is the primary indicator of a bottleneck in a Kafka consumer's processing logic?
- What is the primary architectural role of Apache Kafka in a distributed system?