Vector Search hard
A multi-tenant SaaS stores 2B vectors for 20K tenants, and every query is scoped to one tenant. One tenant holds 40% of the vectors and sends bursty traffic; the rest are small. Vectors are hash-sharded by document ID across 32 shards, so every query fans out to all shards with a tenant filter. Small tenants see latency spikes during the large tenant's bursts. Which sharding change fits best?
- A Keep hash sharding; it spreads every tenant evenly, so the large tenant's bursts are already diluted across the cluster.
- B Add read replicas of every hash shard so the bursty tenant's QPS spreads out, leaving the shard key unchanged.
- C Hash on tenant ID so each tenant maps to exactly one shard, isolating tenants and limiting every query to one shard.
- D Shard by tenant, packing small tenants together and giving the large tenant its own dedicated set of shards.
Answer: D. Shard by tenant, packing small tenants together and giving the large tenant its own dedicated set of shards.
With hash sharding, every tenant-scoped query fans out to all 32 shards. One tenant's burst therefore eats capacity every other tenant needs, and filtered per-shard search wastes work. **Sharding by tenant** isolates noisy neighbors and limits fan-out to the tenant's own shards, at the cost of uneven shard sizes and rebalancing as tenants grow. Hashing on tenant ID (C) puts the 40% tenant on one oversized hot shard. Hash sharding remains the better choice when queries span the whole corpus.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are building an IVF index on 20 million vectors. A teammate asks why the build needs a 'training' step before any vectors can be added. What is the reason?
- An HNSW index serves a marketplace where about 30% of listings are updated or removed each week. Each update is applied as a delete plus an insert, and deletes mark tombstones. Over a month, recall@10 drifts from 97% to 89% and p99 latency rises at the same ef_search. What is happening, and what should you do?
- Users delete documents in your primary database, but the deleted documents keep appearing in vector search results for hours. The vector index is rebuilt nightly. What is the best fix?
- Your embedding model outputs unnormalized vectors whose magnitudes vary widely. A teammate wants to switch the index from cosine similarity to dot product because it is cheaper to compute. What is the consequence?
- You are setting an SLO for a new ANN-backed recommendation search over 50 million vectors. A product manager asks for 100% recall@10 with single-digit millisecond latency. What is a realistic response?
- A B2B SaaS app serves 500 customers, and every query searches only one customer's documents. One large customer runs heavy batch queries that slow down search for everyone. Which sharding approach fits best?