Chapter 25 of 33

Monitoring, Logging & Observability

It is 9 PM on the first day of a big festive sale. Customers start complaining on social media that payments are failing. Your system has 40 microservices, 300 servers, a cache cluster, two databases and a payment gateway. Which one is broken? Is it all users or only UPI users? Did it start after the 8 PM deployment?

Without good monitoring, answering these questions means guessing and SSH-ing into random servers. With good monitoring, a dashboard shows the error spike, an alert has already woken the on-call engineer, a trace shows that the payment service is waiting 3 seconds on an external gateway, and the logs for that trace show the exact error. Everything we designed in earlier chapters (caches, queues, replicas, gateways) is useless if we cannot see when it breaks. This chapter is about seeing.

Monitoring is collecting and watching predefined measurements of a system to detect known kinds of problems. Observability is the property of a system that lets you understand its internal state, including problems you did not predict, from the data it emits: mainly metrics, logs and traces.


Monitoring vs Observability

Monitoring

Observability

Answers "is something wrong?"

Helps answer "why is it wrong?"

Watches known failure modes: CPU high, error rate high, disk full

Lets you investigate new, unexpected failures by slicing data in ways you did not plan

Dashboards and alerts

Rich, connected telemetry: high-detail events, traces, correlated logs

Something you do

A property the system has (because it emits good data)

They are not rivals. Monitoring tells you there is a fire; observability helps you find where it started. In a monolith on one server, monitoring is often enough. In a distributed system where one request touches ten services, you need observability.


The Three Pillars: Metrics, Logs and Traces

Signal

What it is

Example

Best for

Metrics

Numbers measured over time, aggregated

requests/sec = 4,200; p99 latency = 180 ms

Dashboards, alerts, trends; very cheap to store

Logs

Timestamped records of individual events

"ERROR payment failed order=ORD-1042 reason=timeout"

The detailed story of one event; debugging

Traces

The path and timing of one request across services

checkout 3.2 s = auth 20 ms + cart 40 ms + payment 3.1 s

Finding which service or call is slow or failing

A typical investigation uses all three in order: a metric alert fires (error rate up), a trace of a failing request shows which service is at fault, and that service's logs for the same trace ID show the exact error. (Continuous profiling, which shows which functions use CPU and memory, is often called a fourth signal; OpenTelemetry's profiles signal entered public alpha in 2026.)


Metrics

Metric Types

  • Counter: a number that only goes up (or resets to 0 on restart), e.g. total requests served, total errors. You almost always look at its rate: requests per second over the last 5 minutes.

  • Gauge: a value that goes up and down, e.g. current memory use, queue length, active connections.

  • Histogram: counts of observations in buckets (e.g. how many requests took ≤ 50 ms, ≤ 100 ms, ≤ 250 ms...), from which percentiles can be estimated across many servers.

  • Summary: percentiles calculated on each server itself. Cheap to read but cannot be correctly combined across servers (you cannot average p99s), so histograms are usually preferred.

Labels and the Cardinality Trap

Metrics carry labels (also called tags or dimensions):

http_requests_total{service="payment", method="POST", status="500"}  1027

Each unique combination of label values is a separate time series. Labels like status (a handful of values) are fine. A label like user_id with 5 crore values creates 5 crore time series, which can crash the metrics system. This is called a high-cardinality problem. Rule: per-user or per-request details belong in logs and traces, not in metric labels.

Push vs Pull

  • Pull model (Prometheus): each service exposes a /metrics HTTP endpoint, and the monitoring server scrapes it every 15 or 30 seconds. A failed scrape itself tells you the target is down.

  • Push model (StatsD, many cloud agents, OpenTelemetry's OTLP export): services send metrics to a collector. Better for short-lived jobs and for services behind firewalls.

Prometheus queries use the language PromQL. Two very common queries:

# Fraction of requests that failed (5xx) over the last 5 minutes
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# p99 latency estimated from a histogram
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Why Averages Lie: Use Percentiles

In Latency, Throughput & Availability we met percentiles. They matter most in monitoring. Here is a concrete demonstration, plus an error budget calculation and a structured log line (both explained in the following sections):

import json
import math

# 1. Latency: why the average lies
# 100 requests: 90 fast ones, 8 slower ones, 2 very slow ones (ms)
latencies = [20] * 90 + [150] * 8 + [2000] * 2


def percentile(values, p):
    """Nearest-rank percentile: smallest value with at least p% of data <= it."""
    s = sorted(values)
    rank = math.ceil(p / 100 * len(s))
    return s[rank - 1]


print(f"average = {sum(latencies) / len(latencies):.1f} ms")
for p in (50, 95, 99):
    print(f"p{p}     = {percentile(latencies, p)} ms")

# 2. Error budget for an availability SLO over 30 days
total_requests = 50_000_000
slo = 0.999
budget = total_requests * (1 - slo)
failed_so_far = 32_000
print(f"\nerror budget = {budget:,.0f} failed requests allowed")
print(f"used         = {failed_so_far / budget:.0%}")
print(f"remaining    = {budget - failed_so_far:,.0f}")

# 3. One structured log line (JSON), carrying the trace id
log = {
    "ts": "2026-11-02T10:15:30.120Z",
    "level": "ERROR",
    "service": "payment-service",
    "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
    "order_id": "ORD-1042",
    "msg": "UPI gateway timeout",
    "latency_ms": 3001,
}
print("\n" + json.dumps(log))

Output:

average = 70.0 ms
p50     = 20 ms
p95     = 150 ms
p99     = 2000 ms

error budget = 50,000 failed requests allowed
used         = 64%
remaining    = 18,000

{"ts": "2026-11-02T10:15:30.120Z", "level": "ERROR", "service": "payment-service", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "order_id": "ORD-1042", "msg": "UPI gateway timeout", "latency_ms": 3001}

The average of 70 ms describes nobody: 90% of users saw 20 ms and 1 in 50 waited 2 full seconds. The p99 exposes that pain. At scale, "1% of requests" means thousands of unhappy users every minute, and a single page load that makes 50 backend calls is very likely to hit at least one slow one. That is why SLOs and alerts are written on percentiles (p95, p99), not averages.


Logs

Log Levels

Level

Use for

DEBUG

Detailed internal state, usually off in production

INFO

Normal important events: "order placed", "service started"

WARN

Something unusual but handled: a retry, a slow dependency

ERROR

A request or operation failed

FATAL / CRITICAL

The process cannot continue

Structured Logging

Compare these two log lines:

Unstructured:  Payment failed for order ORD-1042 after 3001ms (UPI timeout)
Structured:    {"level":"ERROR","service":"payment-service","order_id":"ORD-1042",
                "latency_ms":3001,"msg":"UPI gateway timeout","trace_id":"4bf9..."}

Humans can read both, but only the structured one can be reliably searched and aggregated by a machine: "count ERROR logs where service = payment-service grouped by msg", or "show all logs where order_id = ORD-1042". Always log in a structured format (usually JSON) in production systems.

Centralised Logging

With 300 servers, logging into each one to read files is hopeless, and when a container is replaced its local logs disappear. Logs are shipped to a central system:

App instances ──stdout──→ Log agent (on each node) ──→ (optional) Kafka buffer
                                                                │
                                                                ▼
                                        Log store + index (Elasticsearch/OpenSearch, Loki, cloud logging)
                                                                │
                                                                ▼
                                                   Search UI and dashboards

A well-known stack is ELK (Elasticsearch, Logstash, Kibana); Elasticsearch indexes log text with the inverted index from Search Systems & Inverted Indexes. Grafana Loki takes a cheaper approach: it indexes only labels and stores the raw log text compressed.

Logs Are Expensive

A quick estimate: 20,000 requests per second, each producing about 500 bytes of logs:

20,000 × 500 B × 86,400 s/day = 864 × 10^9 bytes ≈ 864 GB per day
30-day retention ≈ 26 TB (before replication)

So teams control volume: log at INFO or above in production, sample repetitive success logs, keep errors in full, and move old logs to cheap object storage (see Object Storage & File Storage).

Never log secrets or personal data: passwords, OTPs, tokens, full card numbers, Aadhaar numbers. Logs are copied to many systems and read by many people. Mask such values before logging.


Distributed Tracing

When a checkout request goes through the gateway, the order service, the inventory service and the payment service, each writes its own logs. Tracing ties them together.

  • A trace represents one end-to-end request and has a unique trace ID.

  • A span is one unit of work inside it (an HTTP call, a DB query), with a start time, a duration, a span ID and its parent span's ID.

Trace 4bf92f35...   POST /checkout                     total 3,240 ms
├─ api-gateway                    [■■                              ]   15 ms
├─ order-service                  [ ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■] 3,200 ms
│  ├─ inventory-service  check    [  ■■                            ]   40 ms
│  ├─ mysql INSERT orders         [    ■                           ]   12 ms
│  └─ payment-service    charge   [     ■■■■■■■■■■■■■■■■■■■■■■■■■■■] 3,120 ms
│     └─ UPI gateway HTTP call    [     ■■■■■■■■■■■■■■■■■■■■■■■■■■ ] 3,001 ms  ✗ timeout

One look at this "waterfall" shows the problem: almost all the time is spent waiting on the external UPI gateway call.

Context Propagation

For this to work, every service must pass the trace ID to the next one. The standard way is the W3C Trace Context traceparent HTTP header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                │
             │  trace-id (16 bytes, 32 hex)      parent span id   flags (01 = sampled)
             version                             (8 bytes, 16 hex)

Each service reads the header, creates its own span as a child, and sends a new traceparent (same trace ID, its own span ID) on outgoing calls. The trace ID should also go into every log line, as in our structured log example, so that you can jump from a trace to its logs.

Sampling

Recording every span of every request is costly. Head-based sampling decides at the start of a request (e.g. keep 1%). It is simple, but may throw away the rare failing request. Tail-based sampling buffers the spans of a trace and decides at the end, so it can keep all errors and slow requests while dropping most healthy ones, at the cost of more infrastructure.


OpenTelemetry: The Standard Toolkit

OpenTelemetry (OTel), a CNCF project, is the vendor-neutral standard for producing telemetry. It provides APIs and SDKs for many languages, automatic instrumentation for common frameworks, a wire protocol called OTLP, and the OpenTelemetry Collector, which receives, processes and exports data.

Services (OTel SDK) ──OTLP──→ OTel Collector ──→ metrics backend (e.g. Prometheus)
                                    │        ├──→ trace backend   (e.g. Jaeger, Tempo)
                                    │        └──→ log backend     (e.g. Loki, OpenSearch)
                       (batching, sampling, removing sensitive fields)

The big benefit: you instrument your code once and can change backends or vendors later by reconfiguring the collector, not rewriting code.


What Should You Measure?

Three popular checklists:

Method

Signals

Apply to

Four Golden Signals (Google SRE book)

Latency, Traffic, Errors, Saturation

Any user-facing service

RED

Rate, Errors, Duration

Request-driven services (APIs, microservices)

USE (Brendan Gregg)

Utilisation, Saturation, Errors

Resources: CPU, memory, disks, network, connection pools

Saturation means "how full" a resource is: queue length, thread pool usage, DB connections in use out of the maximum. It often warns you before latency and errors rise. Add business metrics too: orders per minute, payment success rate, signups. A sudden drop in orders is sometimes the first sign of a bug that no technical metric catches.


SLIs, SLOs, SLAs and Error Budgets

  • SLI (Service Level Indicator): a measured number that reflects user happiness, e.g. "percentage of checkout requests that succeed within 1 second".

  • SLO (Service Level Objective): the internal target for that SLI, e.g. "99.9% over 30 days".

  • SLA (Service Level Agreement): a contract with customers with penalties (refunds, credits) if missed. SLAs are usually looser than SLOs, so you notice trouble before you owe money.

The error budget is what the SLO allows you to fail: 100% − SLO. In our code example, 50 million requests at 99.9% allow 50,000 failures; having used 64% of that, the team should be careful with risky releases until the window recovers. When the budget is exhausted, many teams pause feature launches and focus on reliability.

SLO

Allowed downtime per 30 days

Per year

99%

432 minutes (7.2 hours)

87.6 hours

99.9%

43.2 minutes

8.76 hours

99.99%

4.32 minutes

about 53 minutes


Alerting Without Burning Out

  • Alert on symptoms users feel (error rate, latency SLO at risk), not on every cause. "CPU at 85%" is not an emergency if users are fine; it belongs on a dashboard.

  • Every page must be actionable. If the on-call engineer cannot do anything about it, it should not wake them up. Too many useless alerts cause alert fatigue, and real alerts get ignored.

  • Use burn rate alerts. Burn rate = current error rate ÷ error rate allowed by the SLO. With a 99.9% SLO (0.1% allowed), an error rate of 1.44% is a burn rate of 14.4, which would use up a 30-day budget in about 2 days (30 ÷ 14.4 ≈ 2.1). Alert fast on high burn rates, and with a ticket (not a page) on slow ones.

  • Attach a runbook to each alert: what it means, which dashboard to open, and the first steps to try.

  • Use health checks (liveness and readiness endpoints) so load balancers and orchestrators can remove broken instances automatically (see Load Balancing).


Common Mistakes

  • Monitoring averages instead of percentiles, which hides tail latency.

  • High-cardinality labels (user ID, request ID, full URL with IDs) on metrics.

  • Unstructured logs that cannot be searched, and no trace ID in log lines.

  • Logging secrets or personal data such as passwords, OTPs and tokens.

  • Alerting on everything, leading to alert fatigue.

  • Monitoring only infrastructure (CPU, RAM) and missing user-facing or business failures.

  • Running the monitoring system on the same infrastructure it watches, so it fails together with the system. The monitor needs to be more reliable than what it monitors.


Interview Questions

1. What are the three pillars of observability and when do you use each?

Metrics (aggregated numbers over time) for dashboards, trends and alerts; logs (individual event records) for detailed debugging; traces (the path and timing of a single request across services) for finding which component is slow or failing. They are connected through shared identifiers such as the trace ID.

2. Why should you alert on p99 latency rather than average latency?

The average hides the slow tail. A small fraction of very slow requests can leave the average looking healthy while many real users wait seconds. Percentiles show what the slowest users actually experience.

3. What is the difference between SLI, SLO and SLA?

An SLI is a measured indicator (e.g. success rate). An SLO is the internal target for it (e.g. 99.9% over 30 days). An SLA is an external contract with penalties, usually looser than the SLO.

4. How much downtime does a 99.95% monthly (30-day) SLO allow?

30 × 24 × 60 = 43,200 minutes; 0.05% of that is 21.6 minutes.

5. How does a trace ID travel across microservices?

Through context propagation: the caller adds a header such as the W3C traceparent header containing the trace ID and its span ID; the callee creates a child span with the same trace ID and passes it on in its own outgoing calls.

6. Why should user_id not be a metric label?

Each unique label value creates a new time series. Millions of users would create millions of series, exploding memory and storage in the metrics system. Put user IDs in logs and traces instead.


Summary

  • Monitoring detects known problems; observability lets you understand new ones from the data the system emits.

  • Metrics (counters, gauges, histograms) are cheap and power dashboards and alerts; keep label cardinality low.

  • Logs should be structured, centralised, free of secrets, and carry the trace ID.

  • Traces show a request's path through services; context is propagated with the traceparent header.

  • OpenTelemetry is the vendor-neutral standard for producing and collecting telemetry.

  • Measure golden signals / RED / USE, use percentiles, define SLOs, and spend error budgets wisely.

  • Alert on user-facing symptoms with burn rates; every alert should be actionable and have a runbook.

Metrics tell you something is wrong, traces tell you where, and logs tell you why.