Scalability

Performance vs Scalability

People use these two words as if they mean the same thing. They do not.

A system has a performance problem when it is slow for one user. A system has a scalability problem when it is fast for one user but breaks when many users arrive. You can have one without the other.

Performance optimization makes individual operations faster. Scalability engineering makes the system handle more operations. The techniques often overlap, but the goals are distinct. A hand-tuned SQL query is a performance win. Sharding your database across nodes is a scalability play.

Latency vs Throughput

Latency is how long a single request takes: the time between "I asked" and "I got an answer." Throughput is how many requests the system handles per unit of time.

In most systems, you are aiming for maximum throughput with acceptable latency. These are often at odds. Batching requests improves throughput (you process more per cycle) but increases latency for individual requests (each one waits for the batch).

Where it gets tricky

Under low load, latency and throughput seem independent. Under high load, they become coupled. When throughput approaches capacity, latency spikes. Requests start to queue, and each one waits longer. This is why load testing matters: your system's latency profile at 10% utilization tells you almost nothing about how it behaves at 80%.

Vertical vs Horizontal Scaling

There are two fundamental approaches to adding capacity:

Vertical scaling is appealing because your code does not change. You run it on a bigger box. But there is a ceiling. Hardware has physical limits, single machines are single points of failure, and the cost curve is not linear (a 2x bigger machine costs more than 2x the price).

Horizontal scaling is harder up front. Your application needs to handle distribution: load balancing, data partitioning, consensus. But it offers near-unlimited growth and better fault tolerance. Most serious production systems end up here.

Where Bottlenecks Hide

Systems do not scale uniformly. You will find constraints in specific layers:

• CPU-bound: Compute-heavy work like image processing, encryption, or ML inference. More cores or faster CPUs help.
• I/O-bound: Waiting on disk, network, or database calls. Async processing, caching, and connection pooling help.
• Memory-bound: Working set exceeds available RAM, causing cache misses or swap. More memory or data partitioning helps.
• Network-bound: Bandwidth or latency between services. CDNs, compression, and co-location help.

The first job in any scalability effort is identifying which bottleneck you are actually hitting. Optimizing CPU when you are I/O-bound is wasted work.