Distributed Systems Intuition

In Lesson 03 you learned to read a system by asking where data flows and what fails first. This lesson goes inside the data layer, because that's where the strangest bugs at scale are born: a comment that vanishes on refresh, one server on fire while its siblings idle, two data centers both convinced they're in charge. None of these are freak accidents — they are the predictable consequences of three moves every large system makes: keeping copies of data, splitting data, and deciding who must agree before anything counts.¹ No math today — just the mental models. They're what staff+ system-design interviews actually probe.

Replication: why copies, and who's in charge

You replicate data — keep the same data on several machines — for three reasons: so a dead machine doesn't lose it, so reads can be served by many machines instead of one, and so data can live near the users who read it.¹ The most common arrangement is leader/follower: all writes go to one machine (the leader), and the followers copy its changes. Think of a newspaper: one printing press sets the edition, and trucks carry copies to newsstands across the city. Anyone can read at any newsstand — but if you want to publish, you go to the press.

Here's the trade-off you already know from Lesson 02: does the leader wait for followers to confirm each write? Wait for all of them and every write crawls; wait for none and followers fall behind. Almost everyone picks speed — which means followers are always slightly stale. That gap is replication lag: reading a follower is like reading yesterday's newspaper. Nothing in it is wrong — it was all true when it was printed — it's just not current. Usually the lag is milliseconds and nobody notices. Under load it stretches to seconds or minutes, and then users notice.¹

What they notice has names. A user posts a comment (written to the leader), refreshes (read from a lagging follower) — their comment is gone. They didn't lose data; they read yesterday's paper one second after publishing today's. The guarantee that prevents this is read-your-own-writes: after I write something, my reads must see it — a cheap, targeted promise, typically kept by routing a user's reads to the leader for a short window after they write. The cost: more load on the leader and routing complexity.¹ A cousin bug: two refreshes hit two differently-lagged followers, and the user sees time move backwards — fixed by pinning each user to one replica, at the cost of lumpier load. Notice the pattern: every fix is a trade, never a free lunch.

Partitioning: why split, and where it burns

Replication copies all the data to each machine — which stops working when the data or the write traffic no longer fits on any single machine. Partitioning (sharding) is the other move: split the data by key, so each machine owns a slice. Like an encyclopedia in volumes — A–F on one shelf, G–M on the next — every record has exactly one home, found by its key. Real systems do both at once: partition the data, then replicate each partition.²

Partitioning's signature failure is the hot partition. Splitting helps only if load spreads evenly — and real-world keys refuse to. One celebrity's posts, one viral product on sale day, one enterprise customer's launch: suddenly every request wants the same volume of the encyclopedia, and one shard melts while its siblings sit idle. Adding more shards does nothing — the hot key still lives on one of them. Real mitigations reshape the key (split the hot key into sub-keys, cache it in front, or isolate the big tenant), and each buys relief by adding complexity you'll pay for later.²

Consistency: what the user experiences

"Consistency" sounds abstract until you phrase it as a user experience. Strong consistency means the system behaves as if there were one copy: once a write completes, everyone sees it. Eventual consistency means copies converge eventually: reads are fast and keep working through failures, but may be stale in the meantime.¹ Neither is "better" — it's Lesson 02 again, a straight trade⁴: strong consistency spends latency and availability (machines must coordinate, and must sometimes refuse to answer rather than answer wrong) to buy simplicity for users and developers. The Principal-level move is choosing per feature, not per system: a like-counter can be a day-old newspaper; an account balance or an inventory decrement cannot. And often the honest requirement is smaller than "strong" — read-your-own-writes is exactly such a scoped promise, much cheaper than global strong consistency.¹

Why consensus is hard: the split brain

One puzzle remains. Leader/follower assumes everyone agrees who the leader is. So the leader dies, a standby takes over — easy? Here's the trap: from inside a network, a dead machine and a slow network look identical.³ If a standby promotes itself whenever the leader goes quiet, then one network blip creates two leaders, both accepting writes that contradict each other. That's split brain — like two store co-managers whose radio link drops: each assumes the other is gone, both start giving orders, and the staff execute both sets. And notice why "just use a leader to decide" begs the question — choosing the leader requires everyone to agree on the choice, and agreement-under-failure is the very problem we're solving. The real answer is majority voting: nothing counts unless a majority of nodes confirms it, so of two rival leaders, at most one can ever hold a majority — the minority side is forced to stand down. It works, and it costs: extra machines, extra coordination on every decision, and a minority partition that refuses writes entirely.³ Which is why the standing advice is: don't build consensus yourself — lean on systems that have it (your managed database's failover, ZooKeeper, etcd), and know what they're protecting you from.

🧪 Lab: the diagnosis sheet

Time to run these models against realistic incidents — the exact skill of a Principal in an incident channel or a system-design interview. Below are four incidents. For each one, write three lines in a doc: (a) the distributed-systems phenomenon, named precisely; (b) why it happens, mechanically; (c) one mitigation and the trade-off it introduces.

1. The vanishing comment. A user posts a comment, sees it appear, refreshes — it's gone. Sixty seconds later it's back. Support has 40 tickets calling the app "haunted." Writes go to the primary database; reads go to read replicas.
2. One shard on fire. Mid-flash-sale, checkout latency spikes. Dashboard: database shard 7 of 16 is at 100% CPU; the other fifteen are nearly idle. The catalog is sharded by product ID, and one doorbuster product is 80% of today's traffic.
3. Two primaries. After a 90-second network outage between two data centers, monitoring shows both DCs' databases accepted writes for eight minutes. Some orders now exist in one DC, some in the other, and twelve exist in both — with different totals.
4. The report that doesn't add up. A nightly finance report reads each region's totals from that region's read replica, then sums them. The sum disagrees with the global total by a small, different amount every night. No writes are missing the next morning.

Then bring the sheet back to me in chat. I'll review it against a Principal-level rubric: did you name the mechanism (not just "lag" but which guarantee broke), does your mitigation state its cost (a fix with no trade-off named is a red flag at this level), and would your explanation survive an interview follow-up ("okay — and why doesn't adding more shards fix incident 2?"). File the sheet in your evidence trail — Lesson 12 turns these into interview stories, and "I can diagnose distributed failures on sight" is one of the strongest.

Check yourself — diagnose from the model

Three scenarios. Reason from the mental models — newspapers, encyclopedias, co-managers — not from memory. Wrong picks stay live.