Availability & Consistency

When servers die or disagree, what does your system do? The trade-offs behind every system with more than one machine.

CAP Theorem

The CAP theorem says a system spread across many machines can promise at most two of these three things at the same time:

Consistency
Availability
Partition Tolerance
In a network partition, you must choose: reject requests (consistent but unavailable) or serve potentially stale data (available but inconsistent).
You always need partition tolerance in distributed systems, so the real choice is C vs A.

Here is the thing most explanations gloss over: partition tolerance is not optional. Networks fail. Packets get dropped. Data centers lose connectivity. If you are building a distributed system, partitions will happen. So the real decision is between consistency and availability during a partition.

CP systems: consistency and partition tolerance

A CP system will refuse to serve requests rather than return stale data. When nodes cannot communicate, the system blocks or errors until the partition heals.

When to choose CP: Financial transactions, inventory systems, anything where serving wrong data is worse than serving no data. If your bank balance shows the wrong number, that is a serious problem. Better to show an error page.

AP systems: availability and partition tolerance

An AP system keeps serving requests during a partition, even if different nodes return different values. Data converges once connectivity is restored.

When to choose AP: Social media feeds, DNS, shopping cart contents, view counters. If two users see slightly different like counts for a few seconds, nobody notices. But if the site goes down, everyone notices.

CAP is a spectrum, not a switch. Real systems do not fit neatly into CP or AP boxes. Most databases let you tune consistency per-query. DynamoDB defaults to eventually consistent reads but offers strongly consistent reads at higher cost. The trade-off is continuous, not binary.

Consistency Patterns

Strong Consistency

Every read returns the most recent write. After a write completes, all later reads, from any node, see that write. This is the simplest mental model but the hardest to implement at scale, because it requires coordination between nodes on every operation.

Examples: Traditional RDBMS with synchronous replication, Google Spanner (uses TrueTime to achieve global strong consistency).

Eventual Consistency

If no new writes occur, all replicas will converge to the same value in the end. The system does not guarantee when. In practice, convergence usually happens in milliseconds, but under high load or network issues, it can take longer.

Examples: DNS propagation, DynamoDB default reads, Cassandra with low consistency levels.

Weak Consistency

After a write, there is no guarantee that subsequent reads will see it. The system makes a "best effort" but provides no convergence promise. This is common in caches and real-time systems where speed matters more than accuracy.

Examples: Memcached, video chat (you do not replay dropped frames), multiplayer game state.

Pattern Guarantee Latency Use case
Strong Reads always current Higher Banking, inventory
Eventual Converges over time Lower Social feeds, analytics
Weak Best effort Lowest Caches, real-time streams

Availability Patterns

Fail-over

Fail-over means having a standby ready to take over when the primary fails. There are two flavors:

Active-Passive

One active server handles traffic. A passive standby monitors via heartbeat and takes over if the primary dies.

Primary
heartbeat →
Standby
Active-Active

Both servers handle traffic simultaneously. If one fails, the other absorbs the full load.

Server A
← traffic →
Server B
Active-passive wastes a standby. Active-active is more efficient but harder to manage state.

Active-passive is simpler but you are paying for a machine that sits idle. There is also a brief downtime during failover (the detection + switchover window). Active-active uses all machines efficiently but requires careful handling of state: both nodes need to agree on data, or you need to partition the workload cleanly.

Replication

Replication keeps copies of data across multiple nodes. The two primary patterns:

  • Leader-follower (also called primary-replica): One node handles writes, followers replicate and serve reads. Simple, but the leader is a bottleneck and a single point of failure unless you add automatic leader election.
  • Multi-leader (also called multi-primary): Multiple nodes accept writes. Higher write throughput and availability, but you need conflict resolution, what happens when two nodes get different writes to the same key simultaneously?

Measuring Availability

Availability is usually expressed as "nines":

Level Uptime Downtime/year
99% (two 9s) 99% 3.65 days
99.9% (three 9s) 99.9% 8.77 hours
99.99% (four 9s) 99.99% 52.6 minutes
99.999% (five 9s) 99.999% 5.26 minutes
Availability is multiplicative. If your app server is 99.9% and your database is 99.9%, the combined availability is 99.9% × 99.9% = 99.8%. Every dependency you add erodes total availability. This is why reducing the number of synchronous dependencies in your critical path matters so much.