SLO Cheat Sheet — Principal Engineer Track

The three terms

Term	Definition	Owner	Remember it as
SLI Service Level Indicator	A quantified measurement of one aspect of service level, defined over real user interactions (e.g., proportion of HTTP requests returning 2xx/3xx in < 300 ms, measured at the load balancer).	Engineering	What you measure
SLO Service Level Objective	The target value (or range) you choose for an SLI, over a window (e.g., 99.9% of requests good, rolling 30 days). Internal; chosen from user needs, not current performance.	Product + engineering	What you promise yourself
SLA Service Level Agreement	A contract with customers with consequences (credits, penalties) for missing it. Always set looser than the SLO so the internal target trips first.	Business / legal	What you promise customers

The SLI menu

Pick 1–3 per critical user journey. Every SLI is a ratio: good events ÷ valid events. Specify where it's measured and what counts as good — or it isn't an SLI yet.

SLI type	Question it answers	Example specification
Availability	Did the service respond successfully?	% of requests to `/api/*` returning non-5xx, at the load balancer
Latency	Did it respond fast enough?	% of successful requests completing in < 300 ms (use a threshold ratio or a high percentile — never the average)
Throughput	Can it keep up with demand?	% of minutes where processed events ≥ 95% of enqueued events
Correctness	Was the answer right?	% of records where pipeline output matches a golden/known-good result
Freshness	Is the data recent enough?	% of reads served from data updated within the last 5 minutes

The nines

SLO	Allowed downtime / 30-day month	Roughly means
99% ("two nines")	7 h 12 m	Humans fix it during business hours
99.5%	3 h 36 m	Humans fix it, with urgency
99.9% ("three nines")	43 m 12 s	On-call with paging; the common serious-service target
99.95%	21 m 36 s	Paging + fast, rehearsed mitigation
99.99% ("four nines")	4 m 19 s	Automated failover — humans are too slow

Rule of thumb: each extra nine costs ~10× more and delivers benefit only if users can perceive it — beyond their own device/ISP reliability, they can't.

Error-budget math — one worked example

Service: checkout API · SLO: 99.9% availability · rolling 30 days Error budget = 100% − 99.9% = 0.1% of events may fail.
Traffic: 10,000,000 requests/month → budget = 10,000,000 × 0.001 = 10,000 failed requests/month. As time: 30 days × 0.001 = 43.2 minutes of full downtime (or 86 minutes at 50% failure, 7.2 hours at 10% — partial outages spend the budget pro-rata).
A bad deploy that fails 25,000 requests burns 2.5× the monthly budget on its own → feature freeze until the rolling window recovers. Budget remaining → ship; budget exhausted → freeze and fix. That policy is agreed by product + engineering in advance, in writing.

Five SLO-setting pitfalls

Setting the SLO to current performance. Dashboards tell you what you deliver, not what users need. Targets based on "what we happen to achieve" ratchet up forever and cost 10× per nine for zero added user happiness.
Targeting 100% (or one nine too many). Users can't perceive reliability beyond their own phone/Wi-Fi/ISP; every extra nine is paid for in features you can't ship.
Measuring proxies instead of users. CPU, memory, and internal health checks can all be green while every real request fails. Measure real user interactions, as close to the user as practical — and use ratios/percentiles, never averages.
Too many SLIs and SLOs. Ten indicators per service is noise nobody acts on. One to three user-centric SLIs per critical journey force real prioritization.
No error-budget policy. An SLO with no agreed consequence is decoration. Write the decision rule — who freezes what, when, agreed by product and engineering before the budget is ever spent.