Lesson 07 · Operating at Scale
Reliability as a product decision with a number on it — and the budget that turns "is it reliable enough?" from an argument into arithmetic.
Ask most teams "how reliable should this service be?" and you'll get a religious answer: as reliable as possible. That answer is how reliability stays an ops afterthought — something engineers plead for after launches, in the language of fear. Google's SRE organization made one move that changed the whole conversation: treat reliability as a product decision, with a number on it, chosen deliberately and defended like any other feature trade-off.1 This lesson gives you that vocabulary — three acronyms and one budget — and it's the vocabulary Principal interviews expect you to speak fluently.
People blur SLI, SLO, and SLA constantly, and the blur is expensive — they belong to different owners and drive different behavior. Keep them apart:1
| Term | What it is | Owned by | Example |
|---|---|---|---|
| SLI — indicator | A measurement of one aspect of service level | Engineering | % of requests answered successfully in < 300 ms, measured at the load balancer |
| SLO — objective | The target you choose for an SLI | Product + engineering, jointly | 99.9% of requests good, over a rolling 30 days |
| SLA — agreement | A contract with customers, with penalties for missing it | Business / legal | Below 99.5% monthly uptime → 10% service credit |
The ordering matters: you measure an SLI, you target an SLO, and the business sells an SLA that sits looser than the SLO — your internal target trips first, so you're fixing things before you're writing refund cheques.1 As an engineer you will rarely write SLAs; you will absolutely be expected to write SLIs and SLOs. One rule dominates everything else: a good SLI measures what users experience — a request they sent, a page they loaded — not a proxy that's merely easy to collect. CPU utilization is not an SLI; your server can be at 40% CPU while every user request fails.3
The counterintuitive core of the whole discipline: 100% is the wrong reliability target for basically everything.2 Two reasons. First, past a certain point users cannot tell the difference — their phone, their Wi-Fi, their ISP fail more often than you do, and your extra nine disappears into that noise. Second, each additional nine costs roughly ten times more — and you pay for it in the currency you care most about: features you can't ship, because every deploy is a risk you can no longer afford.1 Look at what the nines actually buy:
| Target | Allowed downtime per 30-day month |
|---|---|
| 99% | 7 h 12 m |
| 99.9% | 43 m |
| 99.99% | 4 m 19 s |
| 99.999% | 26 s |
Read that table as a Principal: 99.999% means no human is in the loop — 26 seconds is less time than it takes to open a laptop. That's an architecture (automated failover, multi-region), not an aspiration, and it's why the right question is never "how reliable can we be?" but "what's the cheapest reliability our users will be happy with?"2 The full nines table, the SLI menu, and the math live in this course's SLO cheat sheet — printable, keep it next to your monitor.
Here's where it stops being definitions and becomes an operating system. If your SLO is 99.9%, then 0.1% of events are allowed to fail. That allowance is your error budget — and the move is to treat it as a resource you spend, not a failure you apologize for.2 Concretely: 10 million requests a month at a 99.9% SLO is a budget of 10,000 bad requests. Spend it on velocity — risky deploys, experiments, planned maintenance. A service comfortably inside its SLO with budget left over isn't "doing great"; it's underspending, and should be shipping faster.4
Notice what the error budget really is: an org-alignment tool. Product is paid for velocity; reliability engineers are paid for stability — structurally opposed incentives that usually get resolved by politics, seniority, or who shouts loudest in the launch review.2 The error budget replaces that with a self-policing contract: product agreed to the number, so when the budget is gone, the freeze isn't the engineers punishing anyone — it's the policy both sides signed. This is the lesson where reliability becomes something you negotiate — in a document, once — instead of something you plead for, in every meeting, forever. Walking into a room of product and engineering leads and brokering that number is leverage in its purest form, and it's a story promotion committees recognize instantly.
This is the lab this whole module builds on — the same exercise Google's Customer Reliability Engineering team runs in its Art of SLOs workshop, pointed at a service you actually know.3 Pick one real service you own or know well and write a one-page worksheet:
Feedback loop: bring it back to me in chat. I'll review it against a Principal-level rubric — is each SLI measurable as specified, today (or does the worksheet honestly name the gap), is each SLO tied to user experience rather than current performance, and is the error-budget policy an actual decision rule someone could enforce on a bad Tuesday. This worksheet joins your evidence trail — Lesson 12 folds it into the promotion packet.
Three scenarios. Diagnose from the mental model — don't scroll up. Wrong picks stay live.
Scenario A
Your checkout service has a 99.9% availability SLO but has actually delivered 99.99% for six straight months. Your manager suggests raising the SLO to 99.99%, "since we're clearly already there." What's the Principal-level response?
Scenario B
Mid-quarter, a string of bad deploys exhausts the error budget. The freeze policy says feature releases stop — but product's flagship launch is scheduled this week, and the VP asks you to "be pragmatic." What do you do?
Scenario C
A team proposes two SLIs for their public API: "average CPU stays below 70%" and "zero failed internal health checks." Both are cheap to collect and easy to alert on. You're reviewing the proposal. What's the core problem?
Hand-picked follow-ups. None are required — the primary source above comes first.