Your college fest website works perfectly in testing. Then registrations open, 40,000 students hit it in the same ten minutes, and the page stops loading. Nothing was "wrong" with the code. The system simply could not handle that much load.
There are two basic ways to give a system more capacity: make the machine bigger, or use more machines. These are called vertical and horizontal scaling. This chapter explains both, compares them, shows (with running code) why horizontal scaling needs stateless servers, and explains why adding servers does not give unlimited speed-up.
Vertical scaling (scale up) Horizontal scaling (scale out)
┌─────────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐
│ │ │ S1 │ │ S2 │ │ S3 │ │ S4 │
│ BIG │ ← more CPU, RAM, └────┘ └────┘ └────┘ └────┘
│ SERVER │ faster disks ↑ more machines of the same size
└─────────┘
What Is Scalability?
Scalability is the ability of a system to handle increased load by adding resources, while keeping performance (such as latency) within acceptable limits.
Two parts of this definition matter:
"Increased load" must be described with numbers. These are called load parameters: requests per second, number of concurrent users, read/write ratio, data size, or something specific like "messages per chat group". Different systems break under different load parameters.
"By adding resources": a scalable system grows by adding hardware, not by rewriting the code every time traffic doubles.
Scalability is not the same as speed. A system can be fast for 10 users and collapse at 10,000 (fast but not scalable), or be a little slower per request but keep that speed at any size (scalable).
Vertical Scaling (Scale Up)
Vertical scaling means increasing the capacity of a single machine: more CPU cores, more RAM, faster disks or a faster network card.
Example: your database runs on a server with 8 cores and 32 GB RAM and is running out of memory. You move it to a server with 32 cores and 256 GB RAM. In the cloud this can be as simple as changing the instance type and restarting.
Advantages
No code changes. The application does not even know it moved to a bigger machine.
Simple to operate. One machine to monitor, back up and debug.
No distributed-systems problems. Data is in one place, so transactions, joins and consistency work as usual.
Fast local communication. Everything talks through memory, not the network.
Disadvantages
Hard ceiling. There is a largest machine you can buy or rent. Beyond that, vertical scaling stops.
Cost grows faster than capacity. High-end servers cost disproportionately more per unit of power than commodity ones.
Single point of failure. If the one big machine dies, everything is down.
Downtime to upgrade. Resizing usually needs a restart, unless you have a standby to fail over to.
Vertical scaling is still a very good first step, especially for databases, because scaling a database horizontally is much harder than scaling stateless app servers.
Horizontal Scaling (Scale Out)
Horizontal scaling means adding more machines and spreading the load across them, usually with a load balancer in front.
┌──→ [ App server 1 ]
Users ──→ [ Load ] ┼──→ [ App server 2 ] ──→ [ Shared data: DB, cache ]
[ balancer ] ┼──→ [ App server 3 ]
└──→ [ App server 4 ] ← add more when traffic grows
Advantages
Near-unlimited growth for work that can be split: keep adding machines.
Fault tolerance. If one server dies, the load balancer sends traffic to the others.
Commodity hardware. Many ordinary machines are cheaper per unit of capacity than one huge one.
Elasticity. In the cloud you can add servers for the evening peak and remove them at night (auto-scaling).
Rolling deployments. Update servers one by one with zero downtime.
Disadvantages
Needs a load balancer, which itself must be redundant (see the Load Balancing chapter).
Servers should be stateless, or you must handle session state carefully (shown below).
Distributed-systems problems appear for data: copies can disagree, networks partition, transactions across machines are hard.
More moving parts to deploy, monitor and debug.
Vertical vs Horizontal: Comparison
Feature | Vertical (scale up) | Horizontal (scale out) |
|---|---|---|
How | Bigger machine | More machines |
Upper limit | Largest available machine | Very high, if the work can be split |
Fault tolerance | Single point of failure | Survives individual machine failures |
Code changes | Usually none | Often needed (statelessness, partitioning) |
Data consistency | Simple: one copy | Harder: many machines, many copies |
Cost curve | Rises steeply at the high end | Roughly linear with commodity machines |
Upgrade downtime | Often needs a restart | Add/remove machines with no downtime |
Best for | Databases at moderate scale, quick fixes, early-stage products | Stateless web/app tiers, very large scale, high availability |
In practice, real systems use both. A common pattern: scale stateless app servers horizontally, scale the database vertically as long as possible, then add read replicas, caching and finally sharding. Some people call the mix of both "diagonal scaling".
The Key Requirement: Stateless Servers
In the Client-Server Architecture & APIs chapter we saw that a stateful server keeps per-user data (like a login session or a shopping cart) in its own memory. Let us see what goes wrong when we scale such servers horizontally.
The script below simulates three app servers behind a round-robin load balancer: request 1 goes to S1, request 2 to S2, request 3 to S3. User Asha logs in and then adds two items to her cart.
import itertools
class AppServer:
def __init__(self, name, session_store=None):
self.name = name
self.local_sessions = {} # used only when no shared store
self.shared = session_store
def sessions(self):
return self.shared if self.shared is not None else self.local_sessions
def handle(self, user, action):
s = self.sessions()
if action == "login":
s[user] = {"cart": []}
return f"{self.name}: {user} logged in"
if user not in s:
return f"{self.name}: 401 - who is {user}? please log in"
s[user]["cart"].append(action)
return f"{self.name}: {user} cart = {s[user]['cart']}"
def run(title, servers):
print(title)
rr = itertools.cycle(servers) # simple round-robin
for action in ["login", "add phone", "add cover"]:
print(" ", next(rr).handle("asha", action))
run("Stateful servers (session kept in server memory):",
[AppServer("S1"), AppServer("S2"), AppServer("S3")])
shared = {} # stands in for Redis / a DB
run("Stateless servers (session in a shared store):",
[AppServer("S1", shared), AppServer("S2", shared), AppServer("S3", shared)])
Output:
Stateful servers (session kept in server memory):
S1: asha logged in
S2: 401 - who is asha? please log in
S3: 401 - who is asha? please log in
Stateless servers (session in a shared store):
S1: asha logged in
S2: asha cart = ['add phone']
S3: asha cart = ['add phone', 'add cover']
With stateful servers, Asha logged in on S1, but S2 and S3 have never heard of her. With a shared session store, every server can answer every request. The app servers themselves hold no user state, so they are interchangeable: we can add, remove or restart any of them.
Three ways to handle sessions
Approach | How it works | Trade-off |
|---|---|---|
Sticky sessions | Load balancer always sends a user to the same server (by cookie or IP hash) | Quick fix, but uneven load, and sessions are lost when that server dies |
Shared session store | Sessions live in Redis or a database that every server reads | Servers become stateless; the store must be fast and highly available |
Client-side token | Signed token (for example a JWT) carries the user identity with each request | No lookup needed; but tokens are hard to revoke before they expire |
Make the application tier stateless and push state down into data stores built for it. This single rule is what makes horizontal scaling of app servers easy.
Worked Example: How Many Servers?
Problem: a load test shows one app server handles 2,000 requests/s while staying at a safe 60% CPU. Peak traffic is expected to be 30,000 requests/s. How many servers do we need? What if we spread them over 3 availability zones (separate data-centre buildings) and must survive losing a whole zone?
import math
peak_rps = 30_000 # expected peak requests per second
rps_per_server = 2_000 # measured in a load test at 60% CPU (safe zone)
needed = math.ceil(peak_rps / rps_per_server)
print(f"servers needed for peak : {needed}")
print(f"with N+1 (survive 1 failure) : {needed + 1}")
# Spread across 3 availability zones; must survive losing a whole zone
zones = 3
per_zone = math.ceil(needed / (zones - 1)) # the 2 remaining zones carry everything
print(f"per zone (survive 1 zone loss): {per_zone} x {zones} zones = {per_zone * zones}")
Output:
servers needed for peak : 15
with N+1 (survive 1 failure) : 16
per zone (survive 1 zone loss): 8 x 3 zones = 24
Step by step:
30,000 / 2,000 = 15 servers just to carry the peak.
Add one spare (N+1) so a single server failure does not overload the rest: 16.
To survive losing a whole zone, the remaining 2 zones must carry all 15 servers' worth of load: ⌈15 / 2⌉ = 8 per zone, so 8 × 3 = 24 servers in total.
Notice how availability requirements, not just traffic, drive the server count. We measure servers at 60% CPU, not 100%, because latency climbs sharply as a machine gets close to full (we will see why in Latency, Throughput & Availability).
Why More Servers Is Not Infinite Speed: Amdahl's Law
Horizontal scaling works best when requests are independent: 1,000 users loading their own profile pages can be served by 1,000 different machines. But many workloads have a part that cannot be split, such as a single shared database row everyone updates, a global lock, or a step that must run in order.
Amdahl's law captures this for a fixed amount of work. If a fraction p of the work can run in parallel and the rest (1 − p) cannot, the best possible speed-up with n machines is:
Speed-up(n) = 1 / ((1 − p) + p / n)
As n → ∞, Speed-up → 1 / (1 − p)
def amdahl(p, n):
"""Speed-up with n machines when fraction p of the work can be parallelised."""
return 1 / ((1 - p) + p / n)
print("servers p=0.50 p=0.90 p=0.99")
for n in [1, 2, 4, 8, 16, 64, 1024]:
print(f"{n:7d} {amdahl(0.50, n):7.2f} {amdahl(0.90, n):7.2f} {amdahl(0.99, n):7.2f}")
Output:
servers p=0.50 p=0.90 p=0.99
1 1.00 1.00 1.00
2 1.33 1.82 1.98
4 1.60 3.08 3.88
8 1.78 4.71 7.48
16 1.88 6.40 13.91
64 1.97 8.77 39.26
1024 2.00 9.91 91.18
Even with 1,024 machines, a job that is 90% parallel runs at most about 10× faster. The serial 10% dominates. For system design the lesson is: find the part everyone has to go through (often the database or a single coordinator) because that is what limits scaling. Techniques in later chapters, such as Caching, Database Replication, Database Sharding & Partitioning and Message Queues & Pub/Sub, are mostly ways of shrinking that shared part.
Scaling Each Layer
Layer | Usual first step | Next steps |
|---|---|---|
Static content (images, JS, CSS) | Serve from a CDN | Object storage behind the CDN |
App / web servers | Make stateless, scale horizontally behind a load balancer | Auto-scaling; multiple zones and regions |
Reads from the database | Vertical scaling, indexes, caching | Read replicas |
Writes to the database | Vertical scaling, batching writes | Sharding (partitioning) across many databases |
Slow background work | Move to a queue with worker processes | Add more workers horizontally |
Stateless tiers scale out easily; stateful tiers (databases, caches, queues) are where the hard work and the later chapters of this tutorial are.
Auto-Scaling and Elasticity
Elasticity means capacity follows demand automatically. A cloud auto-scaling group watches a metric (CPU, requests per server, queue length) and adds or removes servers to keep it near a target.
Target: average CPU ≈ 60%
CPU 85% for 3 minutes → add 2 servers
CPU 30% for 15 minutes → remove 1 server
Never go below 4 servers (minimum for redundancy across zones)
Practical points: new servers take time to boot, so scale out early and scale in slowly; for known events (exam results at 10 AM, a sale at midnight) pre-scale before the spike; and make sure the database can handle the extra app servers, since each one opens its own connections.
Common Mistakes
Adding app servers when the database is the bottleneck. More app servers just send more queries to an already-overloaded database.
Keeping sessions or uploaded files on local disk of an app server. They vanish or become invisible to other servers when you scale out.
Treating sticky sessions as a full solution. They hide the state problem until a server dies.
Running servers at 100% capacity in the plan. Leave headroom for spikes and failures.
Forgetting the load balancer can be a single point of failure. Run it redundantly.
Sharding too early. It adds huge complexity; try vertical scaling, indexing, caching and replicas first.
Interview Questions
Q1. When would you prefer vertical scaling over horizontal scaling?
When the load is moderate and the component is hard to distribute, typically a relational database early in a product's life. Vertical scaling needs no code changes and keeps transactions simple. You switch to horizontal techniques when you approach the largest affordable machine or need fault tolerance beyond one box.
Q2. Why must app servers be stateless to scale horizontally?
A load balancer may send each request to a different server. If session data lives in one server's memory, other servers cannot see it, and it is lost when that server restarts. Stateless servers keep state in a shared store or a client token, so any server can handle any request and servers can be added or removed freely.
Q3. A job is 95% parallelisable. What is the maximum speed-up with unlimited machines?
By Amdahl's law the limit is 1 / (1 − 0.95) = 1 / 0.05 = 20×, no matter how many machines you add.
Q4. One server handles 1,500 req/s at safe load. Peak is 20,000 req/s, and you need N+2 redundancy. How many servers?
⌈20,000 / 1,500⌉ = ⌈13.33⌉ = 14 servers for the peak, plus 2 spares = 16 servers.
Q5. What are the downsides of sticky sessions?
Load becomes uneven (some servers get heavy users), sessions are lost if the server fails, scaling in is harder because servers hold user state, and IP-based stickiness breaks when many users share one IP (for example behind a college NAT).
Summary
Scalability is handling more load by adding resources while keeping performance acceptable; describe load with concrete load parameters.
Vertical scaling (bigger machine) is simple but has a ceiling, steep cost and a single point of failure.
Horizontal scaling (more machines) gives fault tolerance and large growth but needs load balancing and stateless servers.
Move session state to a shared store or a client token; sticky sessions are only a partial fix.
Plan capacity with headroom, N+1 spares and zone failures in mind.
Amdahl's law: the part of the work that cannot be split limits the speed-up; find and shrink the shared bottleneck.
Scale up while it is cheap and simple; scale out when you need more room or more safety. Keep the servers stateless either way.