Chapter 05 of 33

Latency, Throughput & Availability

"The app is slow." "The site was down for an hour." "We can't handle the IPL final traffic." These everyday complaints map to three precise ideas that every system designer must measure: latency (how long one request takes), throughput (how many requests are handled per second) and availability (what fraction of time the system works at all).

In the Scalability: Vertical vs Horizontal chapter we added servers to handle more load. This chapter gives you the vocabulary and the maths to say how well a system performs: why averages lie, what p99 means, how latency and throughput are linked, what "four nines" really allows, and how SLIs, SLOs and SLAs turn all this into promises.

Latency      = how long one request waits       (milliseconds)
Throughput   = how many requests per second      (requests/s, MB/s)
Availability = fraction of time it works         (99.9%, 99.99%, ...)

Latency

Latency is the time between sending a request and receiving its response, as seen by the one who is waiting.

Strictly speaking, some books separate latency (time spent waiting before the work starts, such as network travel and queuing) from response time (everything the client sees: network + queuing + processing). In system design discussions the two words are often used interchangeably; just be clear about what you are measuring and from where (the user's phone, the load balancer, or inside the server).

Where the time goes

User taps ──→ network to server ──→ wait in queue ──→ app logic ──→ DB / cache ──→ network back
             (propagation,          (server busy)    (CPU)        (disk, memory)
              TLS, etc.)

To reason about which parts matter, you should know rough orders of magnitude. The exact values depend on hardware and change over the years, so treat this table as a feel for scale, not exact numbers.

Operation

Rough time

Relative scale

CPU L1 cache access

~1 ns

1

Main memory (RAM) access

~100 ns

~100×

Random read from an NVMe SSD

tens of µs (~10-100 µs)

~10,000× and more

Round trip inside one data centre

well under 1 ms (often ~0.1-0.5 ms)

~100,000×+

Hard disk (HDD) seek

several ms (~5-10 ms)

~millions×

Round trip between cities in the same country

tens of ms

~tens of millions×

Round trip India ↔ US east coast

~150-250 ms

~hundreds of millions×

Lessons from this table that drive real designs:

  • Memory is far faster than disk, which is why caches (Redis, in-process caches) exist.

  • Network round trips dominate. Five sequential calls to another service can easily cost more than all the CPU work. Batch calls or run them in parallel.

  • Distance cannot be optimised away. Light in fibre travels about 200,000 km/s, so serving Indian users from a US-only data centre adds well over 100 ms per round trip. That is why CDNs and multi-region deployments exist.


Why Averages Lie: Percentiles

Suppose we record the response times of 20 requests to an API. Most are about 40 ms, but two are very slow. Let us compare the average with percentiles.

The p-th percentile (written pXX) is the value below which XX% of the measurements fall. p50 is the median; p99 means 99% of requests were at least this fast and 1% were slower.

import math
import statistics

# Response times (ms) of 20 requests to an API
latencies = [42, 38, 45, 40, 41, 39, 44, 43, 37, 46,
             40, 41, 38, 42, 47, 39, 44, 1200, 43, 950]

def percentile(data, p):
    """Nearest-rank percentile: smallest value with at least p% of data at or below it."""
    s = sorted(data)
    rank = math.ceil(p / 100 * len(s))
    return s[rank - 1]

print(f"mean = {statistics.mean(latencies):.1f} ms")
for p in (50, 90, 95, 99):
    print(f"p{p}  = {percentile(latencies, p)} ms")

Output:

mean = 144.9 ms
p50  = 42 ms
p90  = 47 ms
p95  = 950 ms
p99  = 1200 ms

The mean (145 ms) describes no real request: 18 requests took under 50 ms and two took about a second. The percentiles tell the true story: the typical user (p50) waits 42 ms, but 1 in 10 users waits about a second (p95 jumps to 950 ms). Real systems therefore set targets like "p99 latency under 300 ms", not "average under 300 ms".

Note: there are several slightly different ways to compute percentiles (nearest-rank, linear interpolation); monitoring tools also estimate them from histograms. With thousands of samples the methods agree closely.

Tail latency and fan-out

The slow end of the distribution (p99, p99.9) is called the tail. Why care about the slowest 1%? Two reasons:

  1. Your most valuable users often hit it. Users with the most data (largest carts, most followers) tend to have the slowest requests.

  2. Fan-out multiplies it. A single page may call many backend servers in parallel and must wait for all of them. If each server is slow on just 1% of requests, the chance that a page hits at least one slow server grows quickly.

# One user request fans out to n backend servers and must wait for all of them.
# Each server is "slow" (slower than its own p99) on 1% of requests.
for n in (1, 10, 50, 100):
    p_slow = 1 - 0.99 ** n
    print(f"fan-out {n:3d} -> {p_slow * 100:5.1f}% of user requests hit at least one slow server")

Output:

fan-out   1 ->   1.0% of user requests hit at least one slow server
fan-out  10 ->   9.6% of user requests hit at least one slow server
fan-out  50 ->  39.5% of user requests hit at least one slow server
fan-out 100 ->  63.4% of user requests hit at least one slow server

With a fan-out of 100, the "rare" 1% slowness hits 63% of user requests. Common fixes: reduce fan-out, set timeouts, and send a hedged request (if a reply is late, send the same request to another replica and use whichever answers first).


Throughput

Throughput is the amount of work a system completes per unit of time, for example requests per second (RPS or QPS), transactions per second, or megabytes per second.

Bandwidth is the maximum rate a link or component could carry; throughput is what it actually achieves. A 1 Gbps network card may give 400 Mbps of throughput because the disk behind it is slower: the slowest component in the path is the bottleneck.

Latency and throughput are different

A highway analogy: latency is how long one car takes to travel from A to B; throughput is how many cars pass per hour. Adding lanes (parallelism) increases throughput but does not make any single car faster. Batching is a similar trade-off: grouping 100 database writes into one batch raises throughput but makes the first write wait longer.

Change

Effect on latency

Effect on throughput

Add more servers (when not overloaded)

Same

Higher

Cache a slow query

Lower

Higher (less work per request)

Batch writes

Higher for individual items

Higher

Move servers closer to users

Lower

Same

Little's Law

A beautifully simple rule connects the three quantities for any stable system:

L = λ × W

L = average number of requests in the system (in flight)
λ = average arrival rate (requests per second) = throughput when stable
W = average time each request spends in the system (latency)

Example: an API receives 2,000 requests/s and each takes 150 ms on average. Then L = 2,000 × 0.150 = 300 requests are in progress at any moment. If each server thread handles one request at a time, you need at least 300 threads (or connections, or async slots) across the fleet, plus headroom. Little's law is a quick way to size thread pools and connection pools in an interview.

Why latency explodes near full load

As a server gets busier, requests start waiting in a queue. A classic queueing model (M/M/1: one server, random arrivals, service rate μ) gives the average time in the system as W = 1 / (μ − λ). Let us see what happens to a server that can process 1,000 requests/s (1 ms each):

# M/M/1 queue: one server, service rate mu = 1000 req/s (1 ms per request)
mu = 1000
print("load(rho)  arrivals/s  avg time in system")
for lam in (100, 500, 800, 900, 950, 990):
    w = 1 / (mu - lam)                      # seconds
    print(f"{lam / mu:8.2f}  {lam:10d}  {w * 1000:10.1f} ms")

Output:

load(rho)  arrivals/s  avg time in system
    0.10         100         1.1 ms
    0.50         500         2.0 ms
    0.80         800         5.0 ms
    0.90         900        10.0 ms
    0.95         950        20.0 ms
    0.99         990       100.0 ms

At 50% load the average time is 2 ms; at 99% load it is 100 ms, fifty times worse, though the server is "only" 2× busier. Real servers are not exactly M/M/1, but the shape of the curve is always the same. That is why capacity plans keep servers well below 100% utilisation, as we did with the 60% CPU target in the scalability chapter.


Availability

Availability is the fraction of time (or of requests) for which a system is working and able to serve users correctly.

Availability = uptime / (uptime + downtime)

Request-based version: successful requests / total valid requests

The "nines"

Availability is usually quoted in "nines". Each extra nine cuts the allowed downtime by 10×.

YEAR, MONTH, DAY = 365 * 24 * 3600, 30 * 24 * 3600, 24 * 3600

def fmt(sec):
    if sec >= 3600:
        return f"{sec / 3600:.2f} h"
    if sec >= 60:
        return f"{sec / 60:.2f} min"
    return f"{sec:.2f} s"

print(f"{'availability':<14}{'down/year':>12}{'down/month':>13}{'down/day':>12}")
for a in [0.99, 0.999, 0.9999, 0.99999]:
    down = 1 - a
    label = f"{a * 100:g}%"
    print(f"{label:<14}{fmt(down * YEAR):>12}{fmt(down * MONTH):>13}{fmt(down * DAY):>12}")

Output:

availability     down/year   down/month    down/day
99%                87.60 h       7.20 h   14.40 min
99.9%               8.76 h    43.20 min    1.44 min
99.99%           52.56 min     4.32 min      8.64 s
99.999%           5.26 min      25.92 s      0.86 s

So "three nines" (99.9%) allows about 8.8 hours of downtime a year or 43 minutes a month. "Five nines" allows only about 5 minutes a year, which is less time than it takes most humans to even notice and respond to an alert. Systems with that target need automatic failover everywhere.

Availability of combined components

Real systems chain components together. Two rules (assuming failures are independent):

  • In series (the request needs all of them): multiply availabilities. A = A1 × A2 × ... The total is lower than the weakest part.

  • In parallel (any one copy is enough): A = 1 − (1 − A1) × (1 − A2) × ... The total is higher than any single copy.

Example: a load balancer at 99.99%, an app server at 99.9% and a database at 99.9%.

def series(*parts):
    """All parts must work: multiply availabilities."""
    total = 1.0
    for a in parts:
        total *= a
    return total

def parallel(*parts):
    """At least one copy must work: 1 - P(all copies down)."""
    all_down = 1.0
    for a in parts:
        all_down *= (1 - a)
    return 1 - all_down

lb, app, db = 0.9999, 0.999, 0.999

print(f"LB + app + DB (one of each) : {series(lb, app, db) * 100:.4f}%")
print(f"two app servers in parallel : {parallel(app, app) * 100:.4f}%")
print(f"LB + 2 apps + 2 DBs         : {series(lb, parallel(app, app), parallel(db, db)) * 100:.4f}%")

mtbf, mttr = 1000, 1        # hours
print(f"MTBF {mtbf} h, MTTR {mttr} h    : {mtbf / (mtbf + mttr) * 100:.4f}%")

Output:

LB + app + DB (one of each) : 99.7901%
two app servers in parallel : 99.9999%
LB + 2 apps + 2 DBs         : 99.9898%
MTBF 1000 h, MTTR 1 h    : 99.9001%
One of each:     [LB 99.99%] → [App 99.9%] → [DB 99.9%]            ≈ 99.79%

With redundancy: [LB 99.99%] → ┌[App 99.9%]┐ → ┌[DB 99.9%]┐        ≈ 99.99%
                               └[App 99.9%]┘   └[DB 99.9%]┘

One copy of each gives only about 99.79% (roughly 18 hours of downtime a year). Duplicating the app server and database raises it to about 99.99%. The remaining weak spot is now the single load balancer, so it should be duplicated too. This is the mathematical reason behind the rule "no single points of failure".

Two honest caveats: the parallel formula assumes copies fail independently (two servers in the same rack on the same power supply do not), and that failover is instant and automatic. That is why copies are spread across availability zones and why failover must be tested.

MTBF and MTTR

Another way to see availability uses two averages:

MTBF = Mean Time Between Failures   (how long it usually runs before failing)
MTTR = Mean Time To Repair/Recover  (how long it takes to get back up)

Availability = MTBF / (MTBF + MTTR)

With MTBF = 1,000 hours and MTTR = 1 hour, availability ≈ 99.90% (last line of the output above). This formula shows two ways to improve availability: fail less often (higher MTBF), or recover faster (lower MTTR). In practice, lowering MTTR (automatic failover, quick rollbacks, good monitoring) is often the cheaper path.

Availability vs reliability vs durability

Term

Question it answers

Example

Availability

Can I use it right now?

The site responds to 99.95% of requests

Reliability

Does it work correctly, without failure, over time?

No wrong balances, no crashes for months

Durability

Once saved, will my data still exist later?

An uploaded photo is never lost, even if disks fail

A system can be available but unreliable (it responds, but with wrong data), or durable but temporarily unavailable (your files are safe but you cannot reach them during an outage).

Failover patterns

Pattern

How it works

Trade-off

Active-passive

One node serves traffic; a standby takes over if it fails

Simple; standby capacity sits idle; brief downtime during switch-over

Active-active

All nodes serve traffic; if one fails, the others absorb its load

Uses all capacity; each node must have headroom; data sync is harder


SLI, SLO, SLA and Error Budgets

Teams turn these measurements into targets and promises using three related terms popularised by Google's Site Reliability Engineering practice:

Term

Meaning

Example

SLI (Service Level Indicator)

A carefully defined measurement of service behaviour

Fraction of requests answered successfully in under 300 ms

SLO (Service Level Objective)

An internal target for an SLI over a time window

99.9% of requests succeed in under 300 ms, measured over 30 days

SLA (Service Level Agreement)

A contract with customers that states consequences (such as refunds or credits) if a promised level is missed

If monthly availability drops below 99.9%, customers get a service credit

A quick test: if missing the target has no external consequence, it is an SLO, not an SLA. SLAs are usually set looser than internal SLOs, so the team gets warned before the contract is broken.

Error budget

An SLO of 99.9% also means you are allowed 0.1% failure. Over a 30-day month that is 43.2 minutes of full downtime (or the same amount of failed requests spread out). This allowance is the error budget.

  • Budget left? The team can ship features and take reasonable risks.

  • Budget used up? Slow down releases and focus on reliability until it recovers.

The error budget turns "how reliable should we be?" from an argument into a number. It also explains why 100% is the wrong target: every extra nine costs much more and slows development, while users often cannot tell the difference because their own Wi-Fi or mobile network is less reliable than that.


Common Mistakes

  • Reporting only the average latency. Always look at p50, p95 and p99.

  • Averaging percentiles. The average of five servers' p99 values is not the p99 of the whole system. Combine the raw data or histograms instead.

  • Confusing bandwidth with throughput, or latency with throughput. More servers raise throughput, not single-request speed.

  • Planning for 100% utilisation. Queueing makes latency shoot up near full load.

  • Adding components in series and expecting high availability. Every extra required component lowers the total.

  • Assuming two copies are independent when they share a rack, power supply, region or a buggy deployment.

  • Calling an internal target an SLA. No contract, no SLA.


Interview / GATE-Style Questions

Q1. A service must reach 99.99% availability in a 30-day month. How much downtime is allowed?

0.01% of 30 × 24 × 60 = 43,200 minutes = 4.32 minutes.

Q2. Three components with availability 99.9%, 99.95% and 99.99% are in series. What is the total?

0.999 × 0.9995 × 0.9999 ≈ 0.99840, i.e. about 99.84%, lower than the weakest component.

Q3. Two independent replicas, each 99% available, serve the same data. What is the availability if either one is enough?

1 − (0.01 × 0.01) = 1 − 0.0001 = 99.99% (assuming independent failures and automatic failover).

Q4. A service handles 500 requests/s with an average latency of 40 ms. How many requests are in flight on average?

By Little's law, L = λW = 500 × 0.040 = 20 requests.

Q5. Why is p99 latency more useful than average latency for a user-facing API?

The average is pulled around by outliers and can describe no real request. p99 tells you the experience of the slowest 1% of requests, which on a busy service is still thousands of users per minute. With fan-out, these tail delays affect a much larger share of page loads.

Q6. What is the difference between an SLO and an SLA?

An SLO is an internal target for a measured indicator (for example, 99.9% of requests succeed within 300 ms over 30 days). An SLA is an external agreement with customers that attaches consequences, such as service credits, to missing a promised level. SLAs are usually looser than SLOs.


Summary

  • Latency is the time for one request; throughput is requests completed per second; availability is the fraction of time the system works.

  • Know orders of magnitude: memory ≪ SSD ≪ network round trip ≪ cross-continent round trip.

  • Use percentiles (p50, p95, p99), not averages; tail latency grows with fan-out.

  • Little's law: in-flight requests = arrival rate × latency. Latency rises sharply as utilisation approaches 100%.

  • Availability in series multiplies (gets worse); in parallel it improves. Remove single points of failure.

  • Availability = MTBF / (MTBF + MTTR); faster recovery is often the easiest win.

  • SLI measures, SLO targets, SLA promises with consequences; the error budget is the failure you are allowed.

Measure the tail, keep headroom, and never let one box decide whether your whole system is up.