Lesson 07 · Operating at Scale

SLOs & Error Budgets

Reliability as a product decision with a number on it — and the budget that turns "is it reliable enough?" from an argument into arithmetic.

Principal skill · negotiate reliability, don't plead for it
🎧 Listen to this lesson · ~8 min · narrated audiobook edition

Ask most teams "how reliable should this service be?" and you'll get a religious answer: as reliable as possible. That answer is how reliability stays an ops afterthought — something engineers plead for after launches, in the language of fear. Google's SRE organization made one move that changed the whole conversation: treat reliability as a product decision, with a number on it, chosen deliberately and defended like any other feature trade-off.1 This lesson gives you that vocabulary — three acronyms and one budget — and it's the vocabulary Principal interviews expect you to speak fluently.

Three letters, three different things

People blur SLI, SLO, and SLA constantly, and the blur is expensive — they belong to different owners and drive different behavior. Keep them apart:1

TermWhat it isOwned byExample
SLI — indicatorA measurement of one aspect of service levelEngineering% of requests answered successfully in < 300 ms, measured at the load balancer
SLO — objectiveThe target you choose for an SLIProduct + engineering, jointly99.9% of requests good, over a rolling 30 days
SLA — agreementA contract with customers, with penalties for missing itBusiness / legalBelow 99.5% monthly uptime → 10% service credit

The ordering matters: you measure an SLI, you target an SLO, and the business sells an SLA that sits looser than the SLO — your internal target trips first, so you're fixing things before you're writing refund cheques.1 As an engineer you will rarely write SLAs; you will absolutely be expected to write SLIs and SLOs. One rule dominates everything else: a good SLI measures what users experience — a request they sent, a page they loaded — not a proxy that's merely easy to collect. CPU utilization is not an SLI; your server can be at 40% CPU while every user request fails.3

Why 100% is the wrong target

The counterintuitive core of the whole discipline: 100% is the wrong reliability target for basically everything.2 Two reasons. First, past a certain point users cannot tell the difference — their phone, their Wi-Fi, their ISP fail more often than you do, and your extra nine disappears into that noise. Second, each additional nine costs roughly ten times more — and you pay for it in the currency you care most about: features you can't ship, because every deploy is a risk you can no longer afford.1 Look at what the nines actually buy:

TargetAllowed downtime per 30-day month
99%7 h 12 m
99.9%43 m
99.99%4 m 19 s
99.999%26 s

Read that table as a Principal: 99.999% means no human is in the loop — 26 seconds is less time than it takes to open a laptop. That's an architecture (automated failover, multi-region), not an aspiration, and it's why the right question is never "how reliable can we be?" but "what's the cheapest reliability our users will be happy with?"2 The full nines table, the SLI menu, and the math live in this course's SLO cheat sheet — printable, keep it next to your monitor.

The error budget — the genius move

Here's where it stops being definitions and becomes an operating system. If your SLO is 99.9%, then 0.1% of events are allowed to fail. That allowance is your error budget — and the move is to treat it as a resource you spend, not a failure you apologize for.2 Concretely: 10 million requests a month at a 99.9% SLO is a budget of 10,000 bad requests. Spend it on velocity — risky deploys, experiments, planned maintenance. A service comfortably inside its SLO with budget left over isn't "doing great"; it's underspending, and should be shipping faster.4

The decision rule Budget remaining → ship: launches, deploys, and experiments proceed. Budget exhausted → freeze and fix: feature releases stop, and engineering effort goes to reliability until the budget recovers. Agreed in advance, in writing, by product and engineering — so nobody has to win an argument on the day.2

The Principal angle: negotiation, not pleading

Notice what the error budget really is: an org-alignment tool. Product is paid for velocity; reliability engineers are paid for stability — structurally opposed incentives that usually get resolved by politics, seniority, or who shouts loudest in the launch review.2 The error budget replaces that with a self-policing contract: product agreed to the number, so when the budget is gone, the freeze isn't the engineers punishing anyone — it's the policy both sides signed. This is the lesson where reliability becomes something you negotiate — in a document, once — instead of something you plead for, in every meeting, forever. Walking into a room of product and engineering leads and brokering that number is leverage in its purest form, and it's a story promotion committees recognize instantly.

The mental model in one line An SLI is what you measure, an SLO is what you promise yourself, an SLA is what you promise customers — and the error budget is the gap between your promise and perfection, spent deliberately on moving fast.

🧪 Lab: the SLO worksheet

This is the lab this whole module builds on — the same exercise Google's Customer Reliability Engineering team runs in its Art of SLOs workshop, pointed at a service you actually know.3 Pick one real service you own or know well and write a one-page worksheet:

  1. The critical user journey — one sentence: who the users are and the one interaction that, if it breaks, makes them consider leaving.
  2. Two SLIs, with exact measurement definitions — for each: where is it measured (client, load balancer, server?) and what precisely counts as a good event (which endpoints, which status codes, what latency threshold)?
  3. A defensible SLO for each — the target and window, plus one "why this number" sentence tied to what users notice, not to what the dashboards currently show.
  4. The monthly error budget it implies — in minutes or bad requests — and one sentence stating what the team does when it's exhausted.

Feedback loop: bring it back to me in chat. I'll review it against a Principal-level rubric — is each SLI measurable as specified, today (or does the worksheet honestly name the gap), is each SLO tied to user experience rather than current performance, and is the error-budget policy an actual decision rule someone could enforce on a bad Tuesday. This worksheet joins your evidence trail — Lesson 12 folds it into the promotion packet.

Check yourself — spend the budget

Three scenarios. Diagnose from the mental model — don't scroll up. Wrong picks stay live.

Scenario A

Your checkout service has a 99.9% availability SLO but has actually delivered 99.99% for six straight months. Your manager suggests raising the SLO to 99.99%, "since we're clearly already there." What's the Principal-level response?

Scenario B

Mid-quarter, a string of bad deploys exhausts the error budget. The freeze policy says feature releases stop — but product's flagship launch is scheduled this week, and the VP asks you to "be pragmatic." What do you do?

Scenario C

A team proposes two SLIs for their public API: "average CPU stays below 70%" and "zero failed internal health checks." Both are cheap to collect and easy to alert on. You're reviewing the proposal. What's the core problem?

Primary source — read this
The founding text of the discipline, by the people who invented the practice — this chapter is where SLI/SLO/SLA and target-setting are defined. Pair it with Ch. 3, "Embracing Risk" for the error-budget argument, and the Art of SLOs workshop (CC-BY, free) for the hands-on version of this lesson's lab.
Your one tangible win You can now put a defensible number on reliability: name the SLI, justify the SLO from user experience, compute the budget it implies, and state the decision rule for when it runs out. That's a complete answer to "how reliable should it be?" — in a promo doc, a design review, or a Principal interview.
I'm your teacher — ask me anything. Not sure whether your service's "good event" definition holds up? Torn between measuring at the load balancer or the client? Bring your SLO worksheet to chat — pressure-testing the "why this number" sentence is exactly what I'm for.

Recommended learning

Hand-picked follow-ups. None are required — the primary source above comes first.

References

  1. Betsy Beyer et al. (eds.), Site Reliability Engineering, Ch. 4 "Service Level Objectives" (Google/O'Reilly, 2016; free online) — SLI/SLO/SLA definitions, target selection, keeping SLOs looser than SLAs.
  2. Marc Alvidrez, Site Reliability Engineering, Ch. 3 "Embracing Risk" — why 100% is the wrong target; the error budget as resolution of the velocity/stability incentive conflict.
  3. Google Customer Reliability Engineering, "The Art of SLOs" workshop (CC-BY) — user-centric SLI specification and the SLO worksheet method.
  4. Steven Thurgood & David Ferguson, The Site Reliability Workbook, Ch. 2 "Implementing SLOs" — SLIs per service type, stakeholder agreement, and spending surplus budget on velocity.