Once a system is split into many services (see Monolith vs Microservices), two practical problems appear immediately.
From the outside: a mobile app should not need to know that there are 30 services, at which addresses, each with its own authentication and rate limits. It needs one front door.
On the inside: service instances come and go all the time. Autoscaling adds 10 order-service containers at 7 PM and removes them at midnight; a crashed container restarts on another machine with a new IP address. How does one service find the current, healthy addresses of another?
The front door is the API gateway. The address book is service discovery. This chapter explains both, how they work together, and the tools used in practice.
┌────────────── API Gateway ──────────────┐
Mobile / Web ──→ │ TLS · auth · rate limit · route · logs │
└───────┬───────────────┬─────────────┬───┘
│ "where is │ │
│ orders?" ▼ ▼
│ users ×2 payments ×3
▼
┌─ Service registry ─┐ orders ×4 (IPs change
│ orders: 10.0.1.5 │ ◄─── register + heartbeat ──
│ 10.0.1.7 … │ as instances start/stop)
└────────────────────┘
The API Gateway
An API gateway is a server that acts as the single entry point for clients into a set of backend services. It receives every external request, applies cross-cutting policies such as authentication and rate limiting, and routes the request to the right service.
Why not let clients call services directly?
Coupling: clients would hard-code service addresses and the internal split. Splitting or merging services would break every app version in users' phones.
Duplicated logic: every service would need its own TLS setup, token validation, rate limiting, CORS rules and logging.
Chattiness: one screen might need data from five services: five round trips over a slow mobile network.
Security: every internal service would be exposed to the internet.
What a gateway does
Responsibility | Meaning |
|---|---|
Routing | Map public paths to services: |
Authentication | Validate the token (for example a JWT) once at the edge; pass the verified user ID to services in a header. Bad requests never reach the backend. (Token formats such as JWT get their own chapter later in this tutorial; here you only need the idea that the token can be checked once, at the edge.) |
Rate limiting / quotas | Per user, per API key or per IP; return HTTP 429 Too Many Requests when exceeded. The algorithms behind this are the topic of the very next chapter, Rate Limiting. |
TLS termination | Handle HTTPS at the edge so services can focus on business logic (internal traffic may be re-encrypted, for example with mTLS). |
Request aggregation | Call several services in parallel and combine the results into one response for the client. |
Protocol translation | Accept REST/JSON from browsers and call internal gRPC services (see API Design: REST, GraphQL & gRPC). |
Transformation | Rewrite paths and headers, strip internal fields from responses. |
Observability | Access logs, metrics per route, and a trace ID attached to each request so it can be followed through all services. |
Traffic control | Canary releases (send 5% of traffic to the new version), blue-green switching, maintenance responses, response caching. |
Backend for Frontend (BFF)
A web dashboard, an Android app and a smartwatch need very different data shapes. One general gateway tends to fill with client-specific special cases. The Backend for Frontend pattern gives each client type its own thin gateway layer, owned by that client's team:
Web app ──→ Web BFF ──┐
Mobile app ──→ Mobile BFF ──┼──→ users · orders · catalog · payments
Partner API ──→ Public gateway┘
The risks of a gateway
Single point of failure: if the gateway is down, everything is down. Run several instances behind a load balancer, across availability zones.
Extra hop: adds a little latency (usually a millisecond or two inside a region).
Becoming a monolith again: putting business logic in the gateway re-creates a central bottleneck that every team must change. Keep it to cross-cutting concerns and simple routing.
Gateway vs load balancer vs reverse proxy vs service mesh
Component | Main job | Traffic it handles |
|---|---|---|
Load balancer | Spread requests across identical instances of one service | Any (L4 or L7) |
Reverse proxy | Sit in front of servers: TLS, caching, compression, forwarding (explored fully in a later chapter on proxies) | Incoming HTTP |
API gateway | A reverse proxy specialised for APIs: routing across many services, auth, quotas, API keys, transformation | North-south (clients → system) |
Service mesh (Istio, Linkerd) | A proxy beside each service (sidecar) or per node handles service-to-service retries, timeouts, mTLS and metrics | East-west (service → service) |
The lines blur: many products can play several roles. Remember the core difference: a load balancer cares about instances, a gateway cares about APIs and clients, and a mesh cares about internal calls.
Popular options
Managed cloud gateways: Amazon API Gateway, Azure API Management, Google Apigee.
Self-hosted: Kong, Envoy (and Envoy Gateway), NGINX, Traefik, Spring Cloud Gateway.
Kubernetes: the Gateway API is the newer, role-oriented standard for configuring L7 routing into a cluster (resources such as
GatewayandHTTPRoute). The widely used community Ingress NGINX controller was retired in March 2026 (no further releases or security fixes), and the Kubernetes project recommends moving to Gateway API or another controller.
Service Discovery
In the old world, a config file said "the database is at 192.168.1.20" and that was true for years. In a cloud or container world, instances are ephemeral: their IP addresses change on every deploy, scale event, or crash. Hard-coding addresses is impossible.
Service discovery is the mechanism by which a client finds the network locations (IP and port) of the currently available, healthy instances of a service, using a logical name such as "orders".
The service registry
At the centre is a service registry: a highly available database of service name → list of healthy instances.
Register: when an instance starts, it is added with its address (and often metadata such as version and zone).
Health: the instance sends periodic heartbeats, or the registry actively runs health checks (call
/health, open a TCP port).Expire: an instance that misses heartbeats beyond a TTL, or fails health checks, is removed. On a clean shutdown, it deregisters itself.
Lookup: clients ask for healthy instances by name (through an API or DNS), often caching the answer and watching for changes.
Because every call depends on it, the registry itself is replicated. Consul, etcd and ZooKeeper use consensus (Raft, or ZAB for ZooKeeper) and favour consistency; Netflix Eureka deliberately favours availability and may serve slightly stale lists during a partition. This is the CAP trade-off from the CAP Theorem & PACELC chapter in action. For discovery, a slightly stale list is usually acceptable as long as clients retry on another instance.
Who registers the instance?
Pattern | How | Trade-off |
|---|---|---|
Self-registration | The service code (or its library) registers itself and sends heartbeats. Common with Eureka and Spring Cloud. | Simple, but every service in every language needs the registration logic. |
Third-party registration | The platform registers instances it starts. Kubernetes does this automatically for pods. | Services stay unaware of discovery; requires a capable platform. |
Client-side vs server-side discovery
Client-side discovery Server-side discovery
client ──1. lookup "orders"──→ registry client ──→ load balancer / router ──→ instance
client ←──2. [10.0.1.5, 10.0.1.7]── (the router queries the registry
client ──3. picks one, calls it directly──→ and picks the instance)
Aspect | Client-side | Server-side |
|---|---|---|
Who chooses the instance? | The calling service (a library with a load-balancing policy) | A router or load balancer in between |
Network hops | One (direct) | Two (via the router) |
Client complexity | Higher: discovery library per language | Low: clients call one stable address |
Examples | Eureka with a client-side load balancer; gRPC client-side load balancing | Kubernetes Services, AWS load balancers, Consul with a proxy |
DNS-based discovery
DNS is the oldest discovery system: resolve a name, get addresses. SRV records (RFC 2782) can even return ports and priorities. Consul offers a DNS interface (for example orders.service.consul), and Kubernetes gives every Service a DNS name such as orders.shop.svc.cluster.local.
The weakness of plain DNS is caching: clients and operating systems may keep a resolved IP longer than intended, so they keep calling an instance that is gone. Use short TTLs, or put a stable virtual address in front of the changing instances, which is what Kubernetes does.
How Kubernetes does it
A Service gets a stable name and a stable virtual IP (ClusterIP).
The control plane tracks the pods behind it in EndpointSlices, marking each endpoint ready only while its pod's readiness probe passes; only ready endpoints receive normal traffic.
On every node, kube-proxy (or a replacement such as an eBPF-based networking plugin) programs rules so traffic sent to the ClusterIP reaches one of the ready pods.
Cluster DNS (CoreDNS) maps the Service name to the ClusterIP.
So in Kubernetes, discovery is server-side and third-party registered: your code just calls http://orders, and the platform does the rest.
Worked Example: Gateway + Registry Together
The simulation below has a registry with a 30-second heartbeat TTL and a gateway that checks the token, routes by longest matching prefix, looks up live instances, and balances round robin (client-side discovery done by the gateway). Unmatched /api/* paths still go to the old monolith, which is the strangler fig migration from the Monolith vs Microservices chapter.
import itertools
TTL = 30 # seconds without a heartbeat -> considered dead
class Registry:
"""A tiny service registry (the job Consul, etcd or Eureka do)."""
def __init__(self):
self.instances = {} # (service, address) -> last heartbeat time
def register(self, service, address, now):
self.instances[(service, address)] = now
heartbeat = register # a heartbeat just refreshes the timestamp
def deregister(self, service, address):
self.instances.pop((service, address), None)
def lookup(self, service, now):
return sorted(addr for (svc, addr), seen in self.instances.items()
if svc == service and now - seen <= TTL)
class Gateway:
"""Single entry point: auth check, prefix routing, client-side round robin."""
def __init__(self, registry, routes):
self.registry = registry
self.routes = sorted(routes.items(), key=lambda r: -len(r[0])) # longest prefix first
self.rr = {} # service -> counter
def handle(self, path, token, now):
if token != "valid-jwt":
return "401 Unauthorized (stopped at the gateway)"
for prefix, service in self.routes:
if path.startswith(prefix):
live = self.registry.lookup(service, now)
if not live:
return f"503 no healthy instance of {service}"
n = next(self.rr.setdefault(service, itertools.count()))
target = live[n % len(live)]
return f"{path} -> {service} @ {target}{path.removeprefix('/api')}" # strip public prefix
return "404 no route"
reg = Registry()
reg.register("orders", "10.0.1.5:8080", now=0)
reg.register("orders", "10.0.1.6:8080", now=0)
reg.register("users", "10.0.2.9:9000", now=0)
reg.register("monolith", "10.0.9.1:8000", now=0) # old app still serves the rest
gw = Gateway(reg, {"/api/orders/": "orders", "/api/users/": "users", "/api/": "monolith"})
print("t=10")
for path in ("/api/orders/101", "/api/orders/102", "/api/users/me", "/api/reports/x"):
print(" ", gw.handle(path, "valid-jwt", now=10))
print(" ", gw.handle("/api/orders/103", "no-token", now=10))
print("t=40: 10.0.1.6 crashed (no heartbeat since t=0); others sent heartbeats at t=25")
reg.heartbeat("orders", "10.0.1.5:8080", now=25)
reg.heartbeat("users", "10.0.2.9:9000", now=25)
reg.heartbeat("monolith", "10.0.9.1:8000", now=25)
print(" registry lookup:", reg.lookup("orders", now=40))
for path in ("/api/orders/104", "/api/orders/105"):
print(" ", gw.handle(path, "valid-jwt", now=40))
print("t=41: autoscaler adds 10.0.1.7")
reg.register("orders", "10.0.1.7:8080", now=41)
for path in ("/api/orders/106", "/api/orders/107"):
print(" ", gw.handle(path, "valid-jwt", now=41))
Output:
t=10
/api/orders/101 -> orders @ 10.0.1.5:8080/orders/101
/api/orders/102 -> orders @ 10.0.1.6:8080/orders/102
/api/users/me -> users @ 10.0.2.9:9000/users/me
/api/reports/x -> monolith @ 10.0.9.1:8000/reports/x
401 Unauthorized (stopped at the gateway)
t=40: 10.0.1.6 crashed (no heartbeat since t=0); others sent heartbeats at t=25
registry lookup: ['10.0.1.5:8080']
/api/orders/104 -> orders @ 10.0.1.5:8080/orders/104
/api/orders/105 -> orders @ 10.0.1.5:8080/orders/105
t=41: autoscaler adds 10.0.1.7
/api/orders/106 -> orders @ 10.0.1.5:8080/orders/106
/api/orders/107 -> orders @ 10.0.1.7:8080/orders/107
Walk through it:
t = 10: two order instances alternate (round robin).
/api/orders/wins over the shorter/api/prefix, while/api/reports/xfalls through to the monolith. A request without a valid token is rejected at the gateway and never touches a service.t = 40: instance 10.0.1.6 stopped sending heartbeats. Its last one was at t = 0, more than 30 s ago, so the registry stops returning it and all traffic goes to 10.0.1.5. Nobody edited any configuration.
t = 41: a new instance registers and immediately starts receiving traffic.
Note the gap: between the crash and the TTL expiry (up to 30 s here), the registry may still return the dead instance. Real systems shrink this gap with active health checks, and callers protect themselves with timeouts and a retry on a different instance.
Health Checks: Liveness vs Readiness
Check | Question | If it fails |
|---|---|---|
Liveness | Is the process alive or stuck (deadlocked)? | Restart it |
Readiness | Can it serve traffic right now (warmed up, connected to its database)? | Remove it from discovery / load balancing, but do not restart |
A common mistake is making the liveness check depend on the database: when the database blips, every instance gets restarted at once, turning a small incident into an outage.
Common Mistakes
Running a single gateway instance. It becomes the single point of failure for the entire product.
Putting business logic in the gateway. It turns into a shared monolith that every team must change.
Hard-coding IP addresses in configuration in a dynamic environment.
Long DNS TTLs or clients that cache lookups forever, so traffic keeps flowing to dead instances.
Trusting the registry blindly. Lists can be stale; always use timeouts and retry on another instance.
Confusing a gateway with a service mesh. The gateway handles client-to-system (north-south) traffic; the mesh handles service-to-service (east-west) traffic.
Interview Questions
Q1. What does an API gateway do, and why use one in a microservices system?
It is the single entry point for external clients. It routes requests to the right service and centralises cross-cutting concerns: authentication, rate limiting, TLS termination, request aggregation, protocol translation, logging, metrics and tracing. Clients stay decoupled from the internal service layout, services avoid duplicating edge logic, and internal services are not exposed to the internet.
Q2. Explain client-side vs server-side service discovery.
In client-side discovery the calling service queries the registry and chooses an instance itself with a load-balancing policy (for example Eureka with a client-side load balancer). It saves a hop but needs a discovery library in each language. In server-side discovery the client calls a stable address of a router or load balancer that looks up the registry and forwards the request (for example a Kubernetes Service or a cloud load balancer). Clients stay simple at the cost of an extra hop and another component to run.
Q3. An instance crashes without deregistering. How does the system stop sending it traffic?
The registry notices that heartbeats stopped (TTL expiry) or that health checks fail, and removes it from lookup results. Until then, callers protect themselves with timeouts, retries on a different instance, and circuit breakers. Load balancers that run their own health checks also take it out of rotation.
Q4. Is the API gateway a single point of failure? How do you avoid it?
Logically it is on every request path, so it must not be a single machine. Run multiple stateless gateway instances behind a load balancer (or DNS / anycast), spread across availability zones, keep their configuration in a replicated store, and autoscale them. Keep the gateway thin so it is fast and rarely changes.
Q5. What is the Backend for Frontend pattern?
Instead of one general gateway serving all client types, each client type (web, mobile, partners) gets its own thin backend layer that aggregates and shapes data exactly for that client. It avoids one gateway full of client-specific special cases and lets each frontend team evolve its API independently.
Summary
An API gateway is the single front door: routing, auth, rate limiting, TLS, aggregation, translation and observability for external traffic.
Keep gateways thin, replicated and free of business logic; use BFFs when client needs differ widely.
Service discovery maps a service name to its current healthy instances, using a replicated registry fed by registration, heartbeats and health checks.
Client-side discovery: the caller picks an instance. Server-side: a router picks it. Kubernetes uses Services, EndpointSlices, readiness probes and DNS.
Registries can be stale for a moment, so callers always need timeouts and retries.
Clients should know one address, services should know only names, and the platform should keep the mapping from names to healthy machines up to date.