Sharding hard
To migrate a live dataset from 5 shards to 10 without downtime, engineers often use a 'Dual-Write' strategy. What is the correct sequence of operations to ensure zero data loss during the cutover?
- A 1. Begin writing to both Old and New locations (Dual Write); 2. Backfill historical data from Old to New; 3. Verify consistency; 4. Switch Reads to New; 5. Stop writing to Old.
- B 1. Stop all writes to the cluster; 2. Perform a file-system rsync from Old to New; 3. Update the DNS records; 4. Resume traffic on the New cluster.
- C 1. Copy all historical data to the New cluster first; 2. Enable Dual-Writes to keep them in sync; 3. Run a validation script; 4. Instantly switch all traffic.
- D 1. Provision New servers; 2. Update the Application Shard Map to include New servers; 3. Allow the database's auto-balancer to move data; 4. Delete the Old servers.
Answer: A. 1. Begin writing to both Old and New locations (Dual Write); 2. Backfill historical data from Old to New; 3. Verify consistency; 4. Switch Reads to New; 5. Stop writing to Old.
The order is critical. You must start 'Dual Writing' *before* the backfill begins to ensure that any new data added during the migration is captured in the new shard. If you copy the data first (Option C) and then enable writes, you will lose every record added during the 'Copying' window. Once the backfill is complete, the New shards are a superset of the Old ones. After a 'Verification' phase to account for edge cases, you safely flip the Read traffic. This 'Continuous Migration' pattern is the gold standard for high-availability systems where maintenance windows are not an option.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In an E-Commerce platform, why might you choose to shard the `Orders` table by `user_id` rather than the more obvious `order_id`?
- In the context of database scaling, what is the fundamental difference between 'Vertical Partitioning' and 'Horizontal Partitioning' (Sharding)?
- If Shard A fails in a large cluster, and its incoming traffic is automatically re-routed to Shard B, what is the risk of a 'Cascading Failure'?
- In a standard 'Modulo Sharding' setup (`key % N`), what is the primary consequence of adding a new server to the cluster (increasing N from 10 to 11)?
- From an engineering perspective, when should you choose 'Vertical Scaling' (a bigger server) over 'Horizontal Scaling' (Sharding)?
- Which sharding strategy is mathematically optimized for efficient 'Range Scans' (e.g., retrieving all records created between January and March)?