Availability & Consistency

The fundamental trade-offs that shape every distributed system, and how to reason about them.

CAP Theorem

The CAP theorem states that a distributed system can deliver at most two out of three guarantees simultaneously:

Consistency
Availability
Partition Tolerance
In a network partition, you must choose: reject requests (consistent but unavailable) or serve potentially stale data (available but inconsistent).
You always need partition tolerance in distributed systems — so the real choice is C vs A.

Here's the thing most explanations gloss over: partition tolerance isn't optional. Networks fail. Packets get dropped. Data centers lose connectivity. If you're building a distributed system, partitions will happen. So the real decision is between consistency and availability during a partition.

CP Systems — Consistency + Partition Tolerance

A CP system will refuse to serve requests rather than return stale data. When nodes can't communicate, the system blocks or errors until the partition heals.

When to choose CP: Financial transactions, inventory systems, anything where serving wrong data is worse than serving no data. If your bank balance shows the wrong number, that's a serious problem — better to show an error page.

AP Systems — Availability + Partition Tolerance

An AP system keeps serving requests during a partition, even if different nodes return different values. Data converges once connectivity is restored.

When to choose AP: Social media feeds, DNS, shopping cart contents, view counters. If two users see slightly different like counts for a few seconds, nobody notices. But if the site goes down, everyone notices.

CAP is a spectrum, not a switch. Real systems don't fit neatly into CP or AP boxes. Most databases let you tune consistency per-query. DynamoDB defaults to eventually consistent reads but offers strongly consistent reads at higher cost. The trade-off is continuous, not binary.

Consistency Patterns

Strong Consistency

Every read returns the most recent write. After a write completes, all subsequent reads — from any node — reflect that write. This is the simplest mental model but the hardest to implement at scale, because it requires coordination between nodes on every operation.

Examples: Traditional RDBMS with synchronous replication, Google Spanner (uses TrueTime to achieve global strong consistency).

Eventual Consistency

If no new writes occur, all replicas will converge to the same value — eventually. The system doesn't guarantee when. In practice, convergence usually happens in milliseconds, but under high load or network issues, it can take longer.

Examples: DNS propagation, DynamoDB default reads, Cassandra with low consistency levels.

Weak Consistency

After a write, there's no guarantee that subsequent reads will see it. The system makes a "best effort" but provides no convergence promise. This is common in caches and real-time systems where speed matters more than accuracy.

Examples: Memcached, video chat (you don't replay dropped frames), multiplayer game state.

Pattern Guarantee Latency Use case
Strong Reads always current Higher Banking, inventory
Eventual Converges over time Lower Social feeds, analytics
Weak Best effort Lowest Caches, real-time streams

Availability Patterns

Fail-over

Fail-over means having a standby ready to take over when the primary fails. There are two flavors:

Active-Passive

One active server handles traffic. A passive standby monitors via heartbeat and takes over if the primary dies.

Primary
heartbeat →
Standby
Active-Active

Both servers handle traffic simultaneously. If one fails, the other absorbs the full load.

Server A
← traffic →
Server B
Active-passive wastes a standby. Active-active is more efficient but harder to manage state.

Active-passive is simpler but you're paying for a machine that sits idle. There's also a brief downtime during failover (the detection + switchover window). Active-active uses all machines efficiently but requires careful handling of state — both nodes need to agree on data, or you need to partition the workload cleanly.

Replication

Replication keeps copies of data across multiple nodes. The two primary patterns:

  • Leader-follower (master-slave): One node handles writes, followers replicate and serve reads. Simple, but the leader is a bottleneck and a single point of failure unless you add automatic leader election.
  • Multi-leader (master-master): Multiple nodes accept writes. Higher write throughput and availability, but you need conflict resolution — what happens when two nodes get different writes to the same key simultaneously?

Measuring Availability

Availability is usually expressed as "nines":

Level Uptime Downtime/year
99% (two 9s) 99% 3.65 days
99.9% (three 9s) 99.9% 8.77 hours
99.99% (four 9s) 99.99% 52.6 minutes
99.999% (five 9s) 99.999% 5.26 minutes
Availability is multiplicative. If your app server is 99.9% and your database is 99.9%, the combined availability is 99.9% × 99.9% = 99.8%. Every dependency you add erodes total availability. This is why reducing the number of synchronous dependencies in your critical path matters so much.