Sharding hard
In a decentralized, leaderless sharded system (like Apache Cassandra), how do physical nodes keep track of which servers own which parts of the hash ring?
- A They rely on a centralized SQL configuration database that acts as the single source of truth for the cluster.
- B They use a 'Gossip Protocol' where nodes periodically exchange state information with random neighbors, allowing cluster changes to propagate epidemically.
- C The ring topology is fixed at the time of installation and cannot be changed without reformatting the disks.
- D They use a hardware-based multicast signal that is broadcast over the datacenter's internal network.
Answer: B. They use a 'Gossip Protocol' where nodes periodically exchange state information with random neighbors, allowing cluster changes to propagate epidemically.
Gossip is a highly robust, peer-to-peer communication method. Every second, each node picks a random partner and says 'Here is what I know about the cluster'. This ensures that even if several nodes fail or the network splits, the latest topology will eventually reach every node. The trade-off is 'Eventual Consistency' of the cluster state; for a few seconds after adding a node, some servers might still try to send requests to the old locations until the 'Gossip' reaches them and updates their local routing map.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In an E-Commerce platform, why might you choose to shard the `Orders` table by `user_id` rather than the more obvious `order_id`?
- In the context of database scaling, what is the fundamental difference between 'Vertical Partitioning' and 'Horizontal Partitioning' (Sharding)?
- If Shard A fails in a large cluster, and its incoming traffic is automatically re-routed to Shard B, what is the risk of a 'Cascading Failure'?
- In a standard 'Modulo Sharding' setup (`key % N`), what is the primary consequence of adding a new server to the cluster (increasing N from 10 to 11)?
- From an engineering perspective, when should you choose 'Vertical Scaling' (a bigger server) over 'Horizontal Scaling' (Sharding)?
- Which sharding strategy is mathematically optimized for efficient 'Range Scans' (e.g., retrieving all records created between January and March)?