Chapter 08 of 33

Load Balancing

In Scalability: Vertical vs Horizontal we decided that the way to handle more traffic is to add more servers. But that raises an obvious question: if you have ten identical servers, who decides which server handles each request? Users type one domain name; they cannot pick a server themselves. Something must sit in front of the servers and spread the work.

That something is a load balancer. It is one of the most common boxes in any system design diagram, and interviewers love to ask how it chooses servers, how it detects failures, and how it avoids becoming a single point of failure itself.

A load balancer is a component that receives incoming requests or connections and distributes them across a pool of servers, so that no single server is overloaded and failed servers stop receiving traffic.

                         ┌──────────→ Server A
                         │
  Clients ──→ Load Balancer ─────────→ Server B
                         │
                         └──────────→ Server C   (marked unhealthy: no traffic)

Think of a big railway reservation office with many counters. A staff member at the entrance directs each person to a free counter and stops sending people to a counter whose clerk has gone on break. That staff member is the load balancer.


What a Load Balancer Gives You

  • Scalability: add servers behind the load balancer to handle more traffic; clients never notice.

  • Availability: through health checks, a crashed server is removed from rotation automatically, so users see no errors (or very few).

  • Zero-downtime deployments: take servers out one at a time, upgrade them, put them back.

  • A single entry point: one address for clients, plus a natural place for TLS termination, logging and request routing.


Where Load Balancers Sit

A large system has load balancers at several layers, not just at the front door.

  Users
    │
    ▼
  DNS / Global load balancing  (pick a region)
    │
    ▼
  Load balancer  ──→  Web / API servers         (layer 1)
                          │
                          ▼
                  Internal load balancer ──→  Service instances   (layer 2)
                          │
                          ▼
                  Database proxy / read-replica balancing        (layer 3)

Between internal services, load balancing is often done on the client side (each caller keeps a list of healthy instances and picks one itself) or by a service mesh sidecar. We look at how callers find those instances in API Gateway & Service Discovery.


Layer 4 vs Layer 7 Load Balancing

From the networks course you know the OSI layers. Load balancers are classified by how deep into the packet they look.

  • Layer 4 (transport layer): decides using only IP addresses, ports and protocol (TCP/UDP). It does not read the HTTP request. It forwards the connection to a server and then shovels bytes back and forth.

  • Layer 7 (application layer): terminates the connection, reads the full HTTP request (URL path, headers, cookies) and then decides. It can route /api/* to one pool and /images/* to another.

Feature

Layer 4

Layer 7

Looks at

IP, port, protocol

URL, headers, cookies, method, body

Protocols

Any TCP/UDP (databases, games, MQTT...)

HTTP/1.1, HTTP/2, HTTP/3, gRPC, WebSocket

Speed and overhead

Very fast, low CPU

More work per request (parsing, often TLS)

Content-based routing

No

Yes (path, host, header rules)

TLS

Usually passed through

Usually terminated at the load balancer

Can modify requests

No

Yes (add headers, rewrite URLs, compress)

Examples

AWS Network Load Balancer, Linux IPVS, HAProxy in TCP mode

AWS Application Load Balancer, Nginx, Envoy, HAProxy in HTTP mode

A common real-world pattern combines both: a fast L4 tier spreads connections across a fleet of L7 proxies, which then do the smart HTTP routing.


Load Balancing Algorithms

The algorithm is how the load balancer picks a server for each request. Algorithms fall into two families:

  • Static: decide without looking at the servers' current state (round robin, weighted round robin, hashing).

  • Dynamic: use live information such as open connections or response time (least connections, least response time, power of two choices).

1. Round Robin

Send requests to servers in order: A, B, C, A, B, C... Simple and fair when all servers are identical and all requests cost about the same. It is the default in Nginx and in AWS Application Load Balancers.

Weakness: it ignores load. If one request takes 5 seconds and others take 5 ms, a server can be swamped while round robin keeps sending it more.

2. Weighted Round Robin

Servers get weights proportional to their capacity. With weights A = 5, B = 1, C = 1, server A gets 5 of every 7 requests. A naive version sends A, A, A, A, A, B, C (bursty). Nginx uses a smooth version that interleaves them:

Smooth weighted round robin (each step):
  1. Add each server's weight to its "current" score.
  2. Pick the server with the highest current score.
  3. Subtract the total weight (7) from the chosen server's score.

Step  current before pick (A,B,C)   pick   after subtracting 7
 1      5,  1,  1                    A      -2,  1,  1
 2      3,  2,  2                    A      -4,  2,  2
 3      1,  3,  3                    B       1, -4,  3
 4      6, -3,  4                    A      -1, -3,  4
 5      4, -2,  5                    C       4, -2, -2
 6      9, -1, -1                    A       2, -1, -1
 7      7,  0,  0                    A       0,  0,  0   ← back to start

Sequence: A A B A C A A

3. Least Connections

Send the request to the server with the fewest active connections. Great when request durations vary a lot, or for long-lived connections like WebSockets. A weighted version divides connections by weight. AWS ALB offers a similar "least outstanding requests" algorithm.

4. Least Response Time

Pick the server with the lowest recent average response time (often combined with fewest connections). It adapts to slow servers automatically but needs the load balancer to keep timing statistics.

5. IP Hash / Hash-Based

Compute hash(client IP) mod N (or hash of a URL, user ID or header). The same client always lands on the same server, giving a simple form of stickiness, and hashing by URL improves cache hits on the servers.

Weakness: when N changes (a server is added or removed), mod N changes the target for most keys. Consistent hashing fixes this so only about 1/N of keys move; we study it in its own chapter, Consistent Hashing. Also, many users behind one office or mobile carrier NAT share an IP, which can make the load uneven.

6. Random and Power of Two Choices

Picking a random server works surprisingly well at large scale, but random luck still creates imbalance. The power of two choices trick is: pick two servers at random and send the request to the less loaded of the two. This tiny change makes the load dramatically more even, and it needs no global state, which is why it is popular in distributed load balancers (Nginx's random two and Envoy's least-request balancer use this idea).

Algorithm

Type

Best when

Weakness

Round robin

Static

Identical servers, similar requests

Ignores actual load

Weighted round robin

Static

Servers of different sizes

Weights must be maintained by hand

Least connections

Dynamic

Variable request time, long-lived connections

Connection count is not always the true load

Least response time

Dynamic

Servers with varying performance

More tracking; can oscillate

IP / key hash

Static

Stickiness, cache affinity

Uneven with NAT; reshuffles when pool changes (unless consistent hashing)

Power of two choices

Dynamic

Large fleets, many load balancers

Slightly random; needs load info for two servers

Seeing the Algorithms in Code

The Python program below implements these algorithms on a small pool and prints which server each one picks.

import hashlib
import random


class Server:
    def __init__(self, name, weight=1):
        self.name = name
        self.weight = weight
        self.active = 0          # open connections
        self.healthy = True
        self.current = 0         # used by smooth weighted round robin


class RoundRobin:
    def __init__(self, servers):
        self.servers = servers
        self.i = 0

    def pick(self, client=None):
        healthy = [s for s in self.servers if s.healthy]
        s = healthy[self.i % len(healthy)]
        self.i += 1
        return s


class SmoothWeightedRoundRobin:
    """The interleaving weighted round robin used by Nginx."""
    def __init__(self, servers):
        self.servers = servers

    def pick(self, client=None):
        healthy = [s for s in self.servers if s.healthy]
        total = sum(s.weight for s in healthy)
        for s in healthy:
            s.current += s.weight
        best = max(healthy, key=lambda s: s.current)
        best.current -= total
        return best


class LeastConnections:
    def __init__(self, servers):
        self.servers = servers

    def pick(self, client=None):
        healthy = [s for s in self.servers if s.healthy]
        return min(healthy, key=lambda s: s.active)


class IPHash:
    def __init__(self, servers):
        self.servers = servers

    def pick(self, client):
        h = int(hashlib.md5(client.encode()).hexdigest(), 16)
        return self.servers[h % len(self.servers)]


def names(balancer, n, client=None):
    return " ".join(balancer.pick(client).name for _ in range(n))


print("Round robin        :", names(RoundRobin([Server("A"), Server("B"), Server("C")]), 7))

swrr = SmoothWeightedRoundRobin([Server("A", 5), Server("B", 1), Server("C", 1)])
print("Smooth weighted RR :", names(swrr, 7), " (weights A=5, B=1, C=1)")

# least connections: A already has 10 long-lived connections
pool = [Server("A"), Server("B"), Server("C")]
pool[0].active, pool[1].active, pool[2].active = 10, 3, 4
lc = LeastConnections(pool)
picks = []
for _ in range(5):
    s = lc.pick()
    s.active += 1
    picks.append(s.name)
print("Least connections  :", " ".join(picks),
      " -> active now", {s.name: s.active for s in pool})

ih = IPHash([Server("A"), Server("B"), Server("C")])
for ip in ["49.36.10.7", "103.21.5.9", "49.36.10.7", "157.48.2.1"]:
    print(f"IP hash {ip:<11}:", ih.pick(ip).name)

# health check: B goes down
rr_pool = [Server("A"), Server("B"), Server("C")]
rr = RoundRobin(rr_pool)
rr_pool[1].healthy = False
print("RR with B unhealthy:", names(rr, 6))

# random vs power of two choices: 10,000 requests on 10 servers
def simulate(choose, n_servers=10, n_req=10_000, seed=1):
    rng = random.Random(seed)
    load = [0] * n_servers
    for _ in range(n_req):
        load[choose(rng, load)] += 1
    return max(load), min(load)

def one_random(rng, load):
    return rng.randrange(len(load))

def two_choices(rng, load):
    a, b = rng.sample(range(len(load)), 2)
    return a if load[a] <= load[b] else b

print("Random choice   max/min load:", simulate(one_random))
print("Two choices     max/min load:", simulate(two_choices))

Output:

Round robin        : A B C A B C A
Smooth weighted RR : A A B A C A A  (weights A=5, B=1, C=1)
Least connections  : B B C B C  -> active now {'A': 10, 'B': 6, 'C': 6}
IP hash 49.36.10.7 : A
IP hash 103.21.5.9 : A
IP hash 49.36.10.7 : A
IP hash 157.48.2.1 : B
RR with B unhealthy: A C A C A C
Random choice   max/min load: (1037, 952)
Two choices     max/min load: (1001, 999)

Three things to notice: least connections avoided the busy server A completely; the same IP (49.36.10.7) always went to the same server; and with only one extra random sample, the two-choices method reduced the spread between the busiest and quietest server from 85 requests to just 2.


Health Checks

A load balancer is only useful if it knows which servers are alive. It finds out in two ways.

  • Active health checks: the load balancer periodically sends a probe (for example GET /health every 10 seconds). After a configured number of consecutive failures the server is marked unhealthy; after a number of successes it is marked healthy again.

  • Passive health checks: the load balancer watches real traffic. If requests to a server fail or time out repeatedly, it stops using that server for a while. In open-source Nginx, this is controlled by max_fails (default 1) and fail_timeout (default 10 seconds).

Time   Probe /health on Server B     State
 0s    200 OK                        healthy
10s    timeout                       healthy (1 failure)
20s    timeout                       healthy (2 failures)
30s    timeout                       UNHEALTHY (threshold 3 reached) → no traffic
40s    200 OK                        unhealthy (1 success)
50s    200 OK                        HEALTHY again (threshold 2 reached) → traffic resumes

What should /health check? A shallow check ("the process responds") is safe. A deep check ("the process can reach the database") catches more problems but is dangerous: if the shared database blips, every server fails its check at once and the load balancer removes the whole fleet. Many teams keep the main health check shallow and alert on dependencies separately.

Connection Draining

When you remove a server (for a deployment or scale-in), the load balancer stops sending it new requests but lets in-flight requests finish for a grace period. AWS calls this the deregistration delay (300 seconds by default). Without draining, users would see errors during every deployment.


Sticky Sessions (Session Affinity)

Some applications keep user session data (like a login session or shopping cart) in the memory of one server. If the next request goes to a different server, that data is missing. Sticky sessions make the load balancer send a user to the same server every time, usually by setting a cookie (or by IP hash).

Sticky sessions

Stateless servers + shared session store

Easy for old applications

Needs a session store (for example Redis) or signed tokens

Uneven load: a "heavy" user is stuck on one server

Any server can handle any request: even load

Server crash loses its users' sessions

Crash loses nothing

Scaling out does not help existing users

New servers help immediately

The modern recommendation: make servers stateless and keep session data in a shared store or in a token (see Authentication & Authorization: OAuth & JWT). Use stickiness only when you must, for example for WebSocket connections, which are naturally tied to one server for their lifetime.


Avoiding a Single Point of Failure

If all traffic passes through one load balancer, that load balancer is now the single point of failure. Common solutions:

  • Active-passive pair: two load balancers share a virtual (floating) IP. The active one holds the IP; if it dies, the passive one takes over the IP within seconds. On Linux this is often done with VRRP (for example using keepalived).

  • Active-active: several load balancers all serve traffic, reached through multiple DNS records or anycast. If one fails, the others carry its share.

  • Managed cloud load balancers: services like AWS ELB are themselves distributed across multiple nodes and availability zones, so redundancy is built in.

               Virtual IP 203.0.113.10
                 │
       ┌─────────┴─────────┐
   LB-1 (active)       LB-2 (standby)   ← heartbeats between them
       │                    (takes over the IP if LB-1 stops responding)
   Server pool

DNS Load Balancing and Global Load Balancing

The simplest load balancing of all is DNS round robin: return several IP addresses for one name and let clients pick. It is cheap but crude: DNS answers are cached (so failover is slow), and DNS knows nothing about server load.

Global Server Load Balancing (GSLB) uses smarter DNS (or anycast) to send users to the best region: the nearest one, or a healthy one if a whole region fails. Inside each region, normal load balancers then spread traffic across servers. This is the same idea CDNs use to choose a PoP, as we saw in Content Delivery Networks (CDN).


Load Balancers in Practice

Nginx Configuration Example

Nginx is widely used as an L7 load balancer (and reverse proxy, covered in Proxies: Forward & Reverse). Here is a typical configuration:

upstream app_servers {
    least_conn;                                   # algorithm (default is weighted round robin)
    server 10.0.0.11:8080 weight=3;               # bigger machine, gets more traffic
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:8080;
    server 10.0.0.20:8080 backup;                 # used only if all others are down
}

server {
    listen 80;
    location / {
        proxy_pass http://app_servers;
    }
}

Other algorithm directives in open-source Nginx include ip_hash (which hashes the first three octets of an IPv4 address, or the whole IPv6 address), hash $request_uri consistent (consistent hashing on any key) and random two least_conn (power of two choices). Active health checks (health_check) are a feature of the commercial NGINX Plus; open-source Nginx relies on passive checks.

Cloud Load Balancers (AWS Example)

AWS service

Layer

Typical use

Application Load Balancer (ALB)

L7

HTTP/HTTPS/gRPC apps; path- and host-based routing. Algorithms: round robin (default), least outstanding requests, weighted random.

Network Load Balancer (NLB)

L4

TCP/UDP/TLS traffic, very high throughput, static IPs per zone

Gateway Load Balancer (GWLB)

L3

Inserting fleets of firewalls or inspection appliances into traffic

Other common tools: HAProxy (L4 and L7), Envoy (L7, popular in service meshes), and the load balancers offered by Google Cloud and Azure.


Worked Example: Sizing a Load-Balanced Tier

From Back-of-the-Envelope Estimation, our photo app peaks at about 52,400 QPS, and we assumed one app server handles about 1,000 QPS.

Servers needed at peak              = 52,400 / 1,000 ≈ 53
Run servers at ≤ 70% so spikes fit  = 53 / 0.7       ≈ 76
Spread across 3 availability zones  = 26 per zone (78 total)
If one whole zone fails             = 52 servers left → 52,400 / 52 ≈ 1,008 QPS each

With 26 servers per zone, losing a full zone leaves the remaining servers at about their 1,000 QPS limit. That is just enough; a cautious team would add a few more. This kind of reasoning ("what happens when a zone dies?") is exactly what interviewers want to hear.


Common Mistakes

  • Drawing one load balancer and calling it done. Always explain how the load balancer itself is made highly available.

  • Using round robin for very uneven requests. Prefer least connections or least outstanding requests when some requests are much heavier.

  • Relying on sticky sessions for important state. A server crash loses the sessions. Keep servers stateless.

  • Using plain hash mod N when servers come and go. Most keys move on every change; use consistent hashing.

  • Deep health checks on a shared dependency. One database hiccup can mark every server unhealthy at once.

  • Confusing L4 and L7. An L4 load balancer cannot route by URL path or read cookies.

  • Forgetting connection draining. Deployments then cut off in-flight requests.


Interview Questions

1. What is the difference between L4 and L7 load balancing?

An L4 load balancer routes using IP, port and protocol only, without reading the application data; it is fast and works for any TCP/UDP protocol. An L7 load balancer understands the application protocol (usually HTTP), so it can route by path, host, headers or cookies, terminate TLS and modify requests, at the cost of more processing per request.

2. When would you choose least connections over round robin?

When requests vary a lot in duration or connections are long-lived (file uploads, WebSockets, slow reports). Round robin would keep giving work to a server already busy with long requests; least connections sends new work to the least busy server.

3. How does a load balancer know a server is down?

Through health checks. Active checks probe an endpoint such as /health at a fixed interval and mark the server unhealthy after a threshold of consecutive failures. Passive checks watch real traffic and back off from servers whose requests fail or time out.

4. What are sticky sessions and why are they often discouraged?

Sticky sessions send the same client to the same server (via a cookie or IP hash) so in-memory session data is available. They cause uneven load, lose sessions when a server dies, and make scaling less effective. Stateless servers with a shared session store or tokens are preferred.

5. The load balancer is a single point of failure. How do you fix that?

Run at least two: an active-passive pair sharing a virtual IP (failover with VRRP), or active-active instances behind DNS or anycast. Managed cloud load balancers already run across multiple nodes and zones.

6. Why does "power of two choices" balance load so much better than pure random?

Pure random choice lets some servers get unlucky streaks. Sampling two servers and choosing the less loaded one almost always avoids the busiest servers, which sharply reduces the maximum load, while still requiring no global coordination between load balancers.

7. With IP hash over 4 servers, what happens when a 5th server is added?

The target is hash mod N; changing N from 4 to 5 changes the result for most keys (about 80% here), so most clients move to a different server, losing any per-server state or cache warmth. Consistent hashing limits the movement to about 1/N of keys.


Summary

  • A load balancer spreads requests across servers and removes failed ones, enabling horizontal scaling and high availability.

  • L4 balances connections using IP/port; L7 reads HTTP and can route by content.

  • Algorithms: round robin, weighted round robin, least connections, least response time, hashing, and power of two choices.

  • Health checks (active and passive) and connection draining make failures and deployments invisible to users.

  • Prefer stateless servers over sticky sessions.

  • Make the load balancer itself redundant (active-passive with a virtual IP, active-active, or a managed service), and use DNS/GSLB to balance across regions.

Add servers to scale, add a load balancer to use them, and add a second load balancer so the first one is not your new weak point.