Observability, Performance & Cost

⏱ ~10 min read · 🎧 9 min listen · ✎ 3 quizzes · 🧪 ~45 min lab

Here's the pattern that separates Principal-level architecture from senior-level architecture: seniors design the system, then bolt on monitoring, tune performance when someone complains, and discover the cloud bill in a panicked Slack thread. Principals treat all three as architecture decisions made up front — because retrofitting any of them is an order of magnitude more expensive than designing for them.¹ This lesson gives you the working mental model for each, and a lab that turns them into a single audit you can run on a real service this week.

Observability: can you answer a question you didn't predict?

Start with the sharpest distinction in the field. Monitoring answers known questions — you predicted a failure mode ("CPU might spike", "error rate might climb") and built a dashboard or alert for it. Observability answers unknown questions — the failure you never predicted, arriving as a weird customer complaint. The test, per Observability Engineering: can you understand any state your system gets into, without shipping new code to do it?¹ Forty green dashboards and a customer saying "it's broken only for us, only since Tuesday" is the moment that test gets graded.

You'll hear observability sold as "three pillars": metrics, logs, and traces. Be able to argue both sides of that framing. It's genuinely useful as an inventory of telemetry types — it's how OpenTelemetry organizes its signals, and how most vendors package their products. The critique, argued hard by Majors, Fong-Jones, and Miranda: pillars describe data types, not the capability — buy three siloed tools and you'll still be eyeballing timestamps trying to correlate them during an incident.¹ What actually buys you answers to unknown questions is high-cardinality, high-dimensionality data: events that carry fields like customer_id, app_version, and build_id — exactly the fields traditional metrics systems choke on, and exactly the ones that isolate "which customers on which version".¹

Performance: the p99 is a user; the average is nobody

The senior-level performance mistake is reporting averages. An average of 60ms can hide the fact that one request in a hundred takes eight seconds — and no actual user experienced 60ms.⁵ Principals speak in percentiles: the p99 is a real request from a real user, and often your most valuable user — the customer with the most data makes the heaviest queries.² That's why Amazon famously set internal latency targets on the 99.9th percentile: the slowest requests belong to the accounts that matter most.²

Then comes the effect that makes tails an architecture problem: tail latency amplification. When one user request fans out to many backend calls — as almost every microservice page load does — the user waits for the slowest one. If just 1% of backend calls are slow and a page touches 100 backends, roughly 63% of page loads hit at least one slow call. Your rare tail becomes your typical experience.²³ This is why Google treats tail-tolerance as a design discipline — hedged requests, tried-and-true replication tricks — not a tuning pass.³ And it connects straight back to Lesson 03: performance work that isn't aimed at the system's actual bottleneck is decoration. Find the constraint first; percentiles tell you whether you found it.

Cost: the third column in every trade-off table

Cloud cost is the concern most engineers treat as someone else's job — and the one where a Principal who shows up with numbers gets instant credibility with leadership. The move is Lesson 02's move, unchanged: cost is just another dimension in the trade-off table, weighed against latency, reliability, and team speed — never optimized in isolation, never ignored. AWS's cost pillar defines the goal the same way: deliver business value at the lowest price point that still meets your requirements — a trade-off statement, not a minimization.⁴

Know the three classic gotchas. Egress: moving data out of a cloud or between regions often costs more than storing it — a "simple" multi-region or multi-cloud design can hemorrhage money in transfer fees. Storage lifecycle: data written once and read never, sitting on hot expensive tiers forever. The observability bill itself: high-cardinality telemetry is powerful and priced accordingly — observability spend is a top-line item at scale, and the 2nd edition of Observability Engineering devotes chapters to managing it.¹⁴ And the metric that makes cost legible to the business: cost per request, or cost per tenant. "$40k/month" starts a fight; "our cost per order doubled since March" starts a strategy conversation.⁴

🧪 Lab: the operability audit

This lab produces a one-page operability audit memo for one real service you work on — the same three questions, asked honestly. It's promotion evidence ("found and framed an operability gap nobody owned") and a ready-made interview story. Write:

1. The unknown-question test. Invent a weird, specific customer complaint — "uploads fail for our account only, only on Android, only since Tuesday." Now trace, step by step, how you'd investigate with today's tooling. Can you slice by customer and app version, or does the trail dead-end at an aggregate dashboard? Write down where it dead-ends.
2. The percentile test. What is the service's p99 latency — not the average — right now? If you can't answer, that's the finding. Then: if p99 doubled tomorrow, would any alert fire or any human notice before a customer did? Name the alert, or name the gap.
3. The cost test. Identify the service's single biggest cost driver (compute, storage, egress, telemetry — go look at the bill). Propose one lever to cut it, and — non-negotiable — state what that lever sacrifices: durability, latency, debuggability, team time.

Feedback loop: bring the memo back to me in chat. I'll review it against a Principal-level rubric — did you test the audit with a concrete unknown-question scenario rather than asserting "we have good logging", is every latency claim percentile-based with no averages anywhere, and does your cost lever honestly name its trade-off. This memo goes into your Lesson 12 promotion packet — operability audits are rare, legible, org-level evidence.

You are a seasoned Principal Engineer reviewing my work: a one-page operability audit memo for a real service, covering the unknown-question test, the percentile test, and the cost test. Grade it against this rubric:

- The unknown-question test uses a concrete, specific scenario — a weird customer complaint traced step by step to where the tooling dead-ends — not an assertion like "we have good logging."
- Every latency claim is percentile-based (p99 or similar), with no averages anywhere; the memo says whether anyone would notice a doubled p99 before a customer did.
- The cost test names the single biggest cost driver, proposes one lever, and honestly states what that lever sacrifices — durability, latency, debuggability, or team time.

Grade each criterion Strong / Adequate / Missing with one sentence of evidence. Be skeptical — challenge the weakest claim first. Then ask me 2–3 follow-up questions a design-review panel or interviewer would ask. Finish with the single highest-leverage improvement.

My operability audit memo follows below.

Check yourself — diagnose with the model

Three scenarios. Diagnose from the mental models — unknown questions, percentiles, trade-offs — not from memory of the text.