Observability

"You can't manage what you can't measure, and you can't debug what you can't see. In a distributed system, seeing is the whole job." — an SRE proverb, paraphrasing Peter Drucker

Welcome to Part VI. The last five parts got your code shipped: packaged into a container, wired through a pipeline, scheduled onto a cluster, and running on cloud infrastructure defined as code. Now it's live — and "live" means dozens of pods across a handful of nodes, a service mesh routing between them, a managed database somewhere, a queue, a cache, three availability zones, and a hundred thousand strangers hammering it at once. When something in that machine misbehaves at 2 a.m. — and it will — you cannot SSH into a box and read the logs. There is no box. There are forty of them, and the guilty one changed since you started looking.

This part is about running systems: seeing into them (this chapter), setting reliability targets and defending them with error budgets, going on-call, and learning from failure without blame. It's the Site Reliability Engineering discipline, and it opens where it must — with the ability to see. You met the app-engineer's half of this in Backend & APIs (Volume 8): how a single service emits structured logs, counts its own requests, and threads a request id through its own code. This chapter zooms out to the whole platform. The question is no longer "what did my service do?" but "what is the cluster doing, and how do I find the one failing hop among forty services I didn't write?"

Monitoring is not observability

These words get used interchangeably and they are not the same thing. The distinction is the spine of this entire chapter.

Monitoring is watching known signals for known failure modes. You predicted a problem — "the node might run out of disk," "the error rate might spike" — so you built a dashboard and an alert for it in advance. Monitoring answers questions you already knew to ask. It catches the known unknowns: things you knew could break, and did.

Observability is a property of the system — how much you can infer about its internal state purely from the outputs it emits. A system is observable when you can ask a question you never anticipated, after the fact, and get an answer from telemetry you already have. It catches the unknown unknowns: the first-time, never-seen failure that no dashboard was built for, because nobody imagined it.

Here's why the difference is existential in a distributed system. You can pre-build a dashboard for "CPU high" or "5xx rate up." You cannot pre-build a dashboard for "checkout is slow, but only for users in the EU, only on the mobile client, only when the recommendations service falls back to its cold cache, and only since the 09:14 deploy of an unrelated service." Nobody predicts that. Monitoring will tell you that checkout is slow. Only observability — rich, high-cardinality, correlatable telemetry — lets you interrogate your way to that specific, novel cause without shipping new code first.

The three pillars — and the thread that ties them

Observability is conventionally assembled from three kinds of telemetry, each answering a different question. Volume 8 introduced them at the single-service level; here we care about how they work across a fleet.

They are complementary, not competitors. A metric tells you that p99 latency spiked cluster-wide; a trace tells you which service in the request path ate the time; the logs on that service tell you exactly what went wrong. The whole superpower — the reason this part opens here — is that all three can be stitched together by a shared trace_id, so you can jump from a spiking graph to the one guilty request to the log line that explains it, across machines you'll never log into.

Logs: centralized, structured, and shipped off the box

At app level (Volume 8) you learned the discipline: emit structured JSON, one object per line, with a level, a trace_id, and context — never prose, because at scale nobody reads logs, machines query them. The platform problem is different and additive: your logs are scattered across forty ephemeral pods that get rescheduled, evicted, and deleted. A pod dies and its local logs die with it. So the cluster rule is: never let logs live on the node. A collector agent (Fluent Bit, Fluentd, Vector, or the OTel Collector) runs on every node, tails each container's stdout, and ships the lines to a central store — Loki, Elasticsearch/OpenSearch (the ELK/EFK stack), or CloudWatch Logs — where they're indexed and queryable long after the pod is gone.

The what-not-to-log rule from Volume 8 gets more dangerous at platform scale, not less: a secret logged by one pod is now fanned out through a shipping pipeline, indexed in a shared store, replicated to backups, and readable by everyone with dashboard access across the org. Never log credentials, tokens, connection strings, full card numbers, or raw request bodies. Redact at the source. This is a security boundary we treat fully in Part VII (DevSecOps) — here, internalize that centralization multiplies the blast radius of a leaked secret.

Metrics: Prometheus scrapes, Grafana shows

Metrics are cheap, numeric, and aggregatable, which makes them the raw material of dashboards and alerts. The three instrument types you met in Volume 8 carry over unchanged — counters (only ever up: total requests, total errors), gauges (up and down: in-flight requests, queue depth, memory), and histograms (distributions, above all request latency, so you can read percentiles). What changes at platform scale is the collection model.

The dominant open-source model is Prometheus, and its defining choice is pull, not push: each service exposes a plain /metrics HTTP endpoint, and a central Prometheus server scrapes it on an interval, discovering targets automatically from Kubernetes. You don't configure forty endpoints by hand; you tell Prometheus how to find pods:

# prometheus.yml — scrape every pod that opts in via annotations (show-only)
global:
  scrape_interval: 15s          # pull metrics from every target every 15s
scrape_configs:
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:        # service discovery: ask the K8s API for targets
      - role: pod
    relabel_configs:
      # only scrape pods that asked to be scraped
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      # use the port the pod advertised
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)

Prometheus stores everything as time series — a metric name plus a set of labels (http_requests_total{service="checkout", method="POST", status="500"}) — and you query it with PromQL (rate(http_requests_total{status=~"5.."}[5m]) for the 5xx rate). Grafana sits on top and turns those queries into dashboards and alerts. Prometheus + Grafana is the default RED/USE stack of the cloud-native world.

Two measurement recipes tell you what to graph — both from Volume 8, both worth re-anchoring here because at platform scale you apply them per-service and to shared infrastructure. RED — Rate, Errors, Duration — for anything that serves requests. USE — Utilization, Saturation, Errors — for anything that is a resource (nodes, disks, the DB connection pool, a queue). Rule of thumb: RED for the things that serve, USE for the things they consume.

And the percentile rule, which interviewers adore and which is even truer across a fleet: graph p50/p95/p99, never the average. The mean is a liar with a friendly face — a handful of slow requests inflate it while it still looks "fine," masking a tail that's ruining real users. The p50 is the typical experience; the p99 is your unluckiest 1%, which at a million requests a day is ten thousand people. The deeper platform gotcha: you cannot average percentiles across servers. Ten pods' p99s meaned together is a meaningless number — a percentile is a property of a distribution, not a value you re-aggregate. This is exactly why latency is stored as a histogram (buckets), so the p99 can be computed over the combined samples. We derived percentiles from first principles in System Design (Volume 5) and computed them in Volume 8; here just remember the fleet-wide trap.

Traces: finding the slow hop across services

Metrics say p99 spiked. Logs say what one service saw. But a single user request in a microservice system fans out — gateway → checkout → inventory → pricing → database → cache — and when it's slow, which hop is guilty? Distributed tracing answers exactly that.

A trace is one request's whole journey rendered as a tree of spans. Each span is a timed unit of work (one service call, one DB query) and carries a trace id shared by every span in the request, plus a span id and a parent span id that reconstruct the tree. When one service calls another over HTTP, it propagates the trace context in a header — the W3C standard is traceparent — so the callee attaches its spans to the same trace. The payoff is a waterfall: nested bars showing exactly where the milliseconds went, so you can read at a glance that inventory's DB span ate 810ms of an 900ms request. The tail latency suddenly has an address. Jaeger, Grafana Tempo, and Zipkin are the common open-source trace backends; the instrumentation that generates the spans is increasingly always OpenTelemetry.

OpenTelemetry: instrument once, send anywhere

Historically, every observability vendor shipped its own agent and SDK, and adopting one meant hard-wiring your code to it — switch vendors, re-instrument everything. OpenTelemetry (OTel) ended that. It is the vendor-neutral standard (a CNCF project, merging the old OpenTracing and OpenCensus efforts) for generating and exporting all three signals — logs, metrics, and traces. You instrument your code once against the OTel API, and you can send the data to any backend: Prometheus, Jaeger, Loki, Datadog, Honeycomb, whatever. The lingua franca is the thing to learn — not one vendor's proprietary agent.

The other half of OTel is the Collector — a standalone process (often one per node, plus a central gateway) that receives telemetry from your apps, processes it (batch, filter, redact, sample, add cluster metadata), and exports it to one or more backends. It decouples your apps from your vendors: apps speak OTLP (the OpenTelemetry protocol) to the Collector, and the Collector fans out. Change backends by editing the Collector config, not your code.

# otel-collector.yaml — receive OTLP, batch, and fan out to three backends (show-only)
receivers:
  otlp:
    protocols:
      grpc:                       # apps push telemetry here (OTLP/gRPC)
processors:
  batch: {}                       # group before export — fewer, bigger writes
  attributes:                     # redaction happens centrally, before storage
    actions:
      - key: http.request.header.authorization
        action: delete
exporters:
  prometheus:                     # metrics → Prometheus scrape endpoint
    endpoint: "0.0.0.0:8889"
  otlp/jaeger:                    # traces → Jaeger
    endpoint: "jaeger:4317"
  loki:                           # logs → Loki
    endpoint: "http://loki:3100/loki/api/v1/push"
service:
  pipelines:
    traces:  { receivers: [otlp], processors: [batch], exporters: [otlp/jaeger] }
    metrics: { receivers: [otlp], processors: [batch], exporters: [prometheus] }
    logs:    { receivers: [otlp], processors: [batch, attributes], exporters: [loki] }

That pipeline — app → collector → backend → dashboard — is the canonical shape of modern observability. Learn it once and every vendor's product is a variation on it.

Cardinality and the cost of seeing

Telemetry is not free, and this is where platform observability gets an economics problem app engineers rarely feel. The killer concept is cardinality: the number of distinct label combinations a metric can produce. Prometheus stores a separate time series for every unique combination of label values. A metric http_requests_total{service, method, status} with 10 services × 5 methods × 6 statuses is 300 series — fine. Add user_id as a label and, with a million users, you've just created a million series per metric — a "cardinality explosion" that can OOM your Prometheus and blow your bill overnight. The rule: labels must be low-cardinality (bounded sets like status code, region, service name) — never unbounded ones like user id, email, request id, or full URL path.

So where do high-cardinality fields go? Into logs and traces, which are designed for it — a user_id or trace_id belongs on a log line or a span, not on a metric label. This division of labor is the whole trick: metrics stay cheap and low-cardinality for dashboards and alerts; the high-cardinality detail lives in logs/traces and is reached by correlation when you need to drill in. (Newer "wide-event" tools like Honeycomb push high-cardinality querying much further — a different, more expensive storage model.)

The other cost levers are sampling and retention. Tracing every request is expensive at high traffic, so teams sample — keep, say, 1% of traces, but bias toward keeping slow and errored ones ("tail-based sampling") so you never lose the interesting ones. And you keep raw data for days-to-weeks, then downsample to cheaper long-term rollups. Observability is a budget line; senior engineers design it deliberately.

Dashboards and the four golden signals

If RED and USE tell you what to measure per component, Google's SRE book gives the canonical top-level view of any user-facing service — the four golden signals, the first thing to put on a dashboard and the first thing an interviewer at Google expects you to name:

Note the overlap: latency + traffic + errors is RED; saturation is the headline of USE. The golden signals are the executive summary; RED and USE are how you fill it in. A good service dashboard shows these four at the top, per service, and lets you drill from any spike into traces and logs beneath it. Keep dashboards few and purposeful — a wall of 200 unwatched graphs is not observability, it's decoration.

Debugging an incident with observability

Put it all together in the shape a real on-call shift takes. It's 02:14. Vary this scenario in your head — it's not the single-service story from Volume 8, it's cluster-wide, and the culprit isn't even the service that's paging:

1. Alert. You're paged not on a raw internal ("CPU high") but on an SLO burn-rate alert: checkout is burning its error budget fast enough to exhaust the month in hours. Symptom-based, user-facing, actionable.
2. Dashboard. The checkout golden-signals board shows p99 latency up 10×, errors climbing, saturation high — what, where, when: it started at 02:05, right after a deploy.
3. Trace. You click an exemplar — a link from the spiking latency graph straight to a representative slow trace — and read the waterfall. Checkout is fine. It spent 1.9s waiting inside the pricing service's DB span. The tail has an address, and it's a service you didn't touch.
4. Log. You copy that request's trace_id, query centralized logs across all services for it, and find pricing's line: "pool exhausted, waiting for connection","waited_ms":1880. Root cause: a pricing deploy at 02:05 shipped a config that halved its DB connection pool. Roll back pricing; budget stops burning.

Alert → dashboard → trace → log → root cause. Aggregate to specific, symptom to cause, in minutes — and only possible because a single trace_id threads through metrics exemplars, traces, and logs alike. That thread is the entire investment; everything else is tooling around it.

A runnable taste: cluster-wide RED from raw samples

Everything above is infrastructure that can't run in a sandbox. But the math under a RED panel is pure logic, so here's a small, self-contained version: latency samples collected from three different pods of one service, pooled into a fleet-wide distribution (you pool the raw samples — you do not average per-pod percentiles), plus the error rate. Run it and read the summary.

Read the contrast at the bottom: the pooled p99 (260) is the honest tail; the average of per-pod p99s is a different, meaningless number. That's the fleet-scale percentile trap made concrete.

The tooling landscape

You'll wire up mature tools, not build these. Know them by what they do:

The strategic advice: instrument against OpenTelemetry, store metrics in Prometheus and traces in a Jaeger/Tempo, visualize in Grafana, and you can swap or add a commercial backend later without touching application code.

Common Mistakes

• Confusing monitoring with observability. Believing a wall of pre-built dashboards means you're "observable." Dashboards answer known questions; observability is the ability to ask new ones about novel failures. Optimize for queryability, not graph count.
• Cardinality explosions. Putting user_id, request_id, email, or a raw URL path on a metric label and OOM-ing Prometheus. High-cardinality data goes in logs and traces; metric labels stay low-cardinality and bounded.
• Averaging latency, and re-averaging percentiles. Reporting a mean (hides the tail), or meaning per-pod p99s into a "cluster p99" (meaningless). Pool the raw distribution, then take the percentile.
• Logs that die with the pod. Relying on local container logs in an ephemeral cluster. Ship every line to a central store (Loki/ELK/CloudWatch) so it outlives the pod.
• Logging secrets or PII into a central store. The leak is now fanned out, indexed, and backed up org-wide. Redact at the source (or in the Collector). See Part VII.
• No correlation id / trace context propagation. Beautiful logs from ten services with nothing linking them. Generate a trace_id at the edge and propagate it (via traceparent) to every hop.
• Alerting on causes, not symptoms. Paging on "CPU > 80%" produces noise nobody actions. Alert on user-facing SLO burn rate; an alert that's normal to ignore is worse than none.
• Instrumenting for one vendor. Hard-wiring a proprietary agent and getting locked in. Instrument against OpenTelemetry and stay portable.

Summary

• Monitoring ≠ observability. Monitoring watches known signals for known failures (known unknowns); observability is the ability to ask new questions about novel failures from telemetry you already emit (unknown unknowns). In a distributed system you need both.
• Three pillars: logs (structured JSON, shipped off ephemeral pods to a central store — Loki/ELK/CloudWatch), metrics (Prometheus scrapes numeric time series, Grafana visualizes), traces (distributed spans across services — Jaeger/Tempo — to find the slow hop). A shared trace_id correlates all three.
• Measure with RED (Rate/Errors/Duration for services) and USE (Utilization/Saturation/Errors for resources); Google's four golden signals (latency, traffic, errors, saturation) are the dashboard summary. Report p95/p99, never averages — and never average percentiles across pods; pool the distribution.
• OpenTelemetry is the vendor-neutral standard: instrument once, export anywhere; the Collector (app → collector → backend → dashboard) decouples your apps from your vendors.
• Cardinality is cost. Metric labels must be low-cardinality (status, region, service) — never user_id/request_id/path. High-cardinality detail lives in logs and traces; use sampling and retention to control spend.
• Debug by drilling aggregate-to-specific: alert → dashboard → trace → log → root cause, threaded by the correlation id — the one investment that makes cross-service debugging fast.

Next we make "reliable" a number instead of a feeling: SLIs, SLOs, and the error budget — the contract that turns the signals you just learned to see into a policy for when to ship features and when to stop and fix.