Almost every app you use stores "files": profile photos on Instagram, PDFs of notes, resumes uploaded on job portals, videos on YouTube, backups of your WhatsApp chats. A relational database is a poor home for these large binary blobs. They bloat the database, make backups slow, and waste expensive database memory and I/O on data that is simply read and written as a whole.
So where do these files go? Large systems use one of three kinds of storage: block storage, file storage and object storage. In this chapter we learn how they differ, why object storage (like Amazon S3) has become the default for user-uploaded content, and the design patterns used to upload and serve files at scale. We will build on SQL vs NoSQL Databases, Database Replication and Content Delivery Networks (CDN).
Object storage is a storage architecture that keeps data as objects in a flat namespace. Each object consists of the data itself, a unique key, and metadata, and is accessed as a whole through an HTTP API (PUT, GET, DELETE) rather than through a file system.
The Three Kinds of Storage
Block Storage
Block storage gives a machine a raw disk: a long list of fixed-size blocks, each with an address. It knows nothing about files. The operating system puts a file system (ext4, NTFS, XFS) on top of it, or a database manages the blocks directly.
Examples: your laptop's SSD, a SAN volume, Amazon EBS, Google Persistent Disk.
Very low latency and supports small random reads and in-place updates.
Usually attached to one server at a time (some cloud volumes allow multi-attach, but the servers must then coordinate writes themselves, for example with a cluster file system).
Best for: database data files, a virtual machine's boot disk, anything needing fast random I/O.
File Storage
File storage provides the familiar hierarchy of folders and files, shared over a network so that many machines can mount the same file system at the same time.
Examples: NFS and SMB file servers, NAS devices, Amazon EFS, Azure Files.
Supports normal file operations: open, seek, append, rename, lock, permissions.
Best for: shared folders, legacy applications that expect a file path, content management systems, home directories.
Harder to scale to billions of files, because maintaining a consistent directory tree with locks across many machines is expensive.
Object Storage
Object storage throws away the directory tree. Data is stored as objects inside buckets, and each object is found by its key, a string such as users/42/avatar.png. The slashes are just characters in the key; there are no real folders. You talk to it over HTTP: PUT to upload, GET to download, DELETE to remove.
Examples: Amazon S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, and self-hosted S3-compatible systems such as MinIO and Ceph.
Scales to practically unlimited objects and petabytes of data.
Objects are written and read as a whole; you cannot edit byte 500 of an object in place. To change it, you upload a new version.
Very cheap per GB, extremely durable, and accessible from anywhere via a URL.
Block: [blk 0][blk 1][blk 2][blk 3] ... ← OS/database decides what they mean
File: /home
└── ravi
├── notes.pdf ← real folders, paths, locks
└── photos/trip.jpg
Object: bucket "cslp-uploads"
key "users/42/avatar.png" → data + metadata
key "users/42/resume.pdf" → data + metadata ← flat map: key → object
key "readme.txt" → data + metadata
Block vs File vs Object
Feature | Block | File | Object |
|---|---|---|---|
Unit of data | Fixed-size block | File in a folder tree | Object (data + metadata + key) |
Access method | Disk protocols (OS sees a disk) | NFS / SMB, normal file paths | HTTP API (PUT/GET/DELETE) |
In-place updates | Yes, any block | Yes, any byte | No, replace the whole object |
Shared by many servers | Usually no | Yes | Yes, by anyone with access |
Scale | Size of a volume | Large, but metadata becomes a bottleneck | Practically unlimited |
Latency | Lowest (sub-millisecond) | Low | Higher (tens of milliseconds per request is common) |
Cost per GB | Highest | Medium | Lowest |
Typical use | Databases, VM disks | Shared folders, legacy apps | Images, videos, backups, logs, data lakes |
Core Concepts of Object Storage
We will use Amazon S3 terms, since most other services copy them.
Bucket: a top-level container for objects, with its own access rules and settings. Bucket names must be unique within their namespace: by default, S3 general purpose bucket names are unique across all AWS accounts, and since March 2026 you can also opt into a namespace reserved for your own account and region.
Key: the full name of an object inside a bucket, e.g.
notes/dbms/unit1.pdf.Object: the bytes plus metadata. System metadata includes size, content type and last-modified time; you can add your own user metadata.
ETag: an identifier for a particular version of the object's content, used to check whether it changed (useful for caching).
Versioning: if enabled, overwriting or deleting an object keeps the older versions, which protects against accidents.
Lifecycle rules: automatically move objects to cheaper storage classes after some days, or delete them (e.g. "delete logs older than 90 days").
"Folders" Are Just Prefixes
When the S3 console shows you folders, it is really doing a LIST with a prefix and a delimiter (/). The toy store below shows how a flat key-value map can still look like a folder tree:
import hashlib
class ToyObjectStore:
"""A flat key -> (bytes, metadata) map. No real folders exist."""
def __init__(self):
self.buckets = {}
def create_bucket(self, name):
self.buckets[name] = {}
def put(self, bucket, key, data, content_type):
etag = hashlib.md5(data).hexdigest() # fingerprint of the content
self.buckets[bucket][key] = {
"data": data,
"meta": {"Content-Type": content_type, "Size": len(data), "ETag": etag},
}
return etag
def get(self, bucket, key):
obj = self.buckets[bucket][key]
return obj["data"], obj["meta"]
def list(self, bucket, prefix="", delimiter=None):
"""Like a LIST call: keys with a prefix; delimiter groups 'folders'."""
keys, folders = [], set()
for key in sorted(self.buckets[bucket]):
if not key.startswith(prefix):
continue
rest = key[len(prefix):]
if delimiter and delimiter in rest:
folders.add(prefix + rest.split(delimiter)[0] + delimiter)
else:
keys.append(key)
return keys, sorted(folders)
s3 = ToyObjectStore()
s3.create_bucket("cslp-uploads")
s3.put("cslp-uploads", "users/42/avatar.png", b"\x89PNG...", "image/png")
s3.put("cslp-uploads", "users/42/resume.pdf", b"%PDF-1.7...", "application/pdf")
s3.put("cslp-uploads", "users/77/avatar.png", b"\x89PNG..!", "image/png")
s3.put("cslp-uploads", "readme.txt", b"hello", "text/plain")
data, meta = s3.get("cslp-uploads", "users/42/resume.pdf")
print("GET users/42/resume.pdf ->", meta)
print("LIST prefix='' delimiter='/' ->", s3.list("cslp-uploads", "", "/"))
print("LIST prefix='users/' delimiter='/' ->", s3.list("cslp-uploads", "users/", "/"))
print("LIST prefix='users/42/' ->", s3.list("cslp-uploads", "users/42/"))
Output:
GET users/42/resume.pdf -> {'Content-Type': 'application/pdf', 'Size': 11, 'ETag': '07fed6503f3ee63fe54dc7dad4efedac'}
LIST prefix='' delimiter='/' -> (['readme.txt'], ['users/'])
LIST prefix='users/' delimiter='/' -> ([], ['users/42/', 'users/77/'])
LIST prefix='users/42/' -> (['users/42/avatar.png', 'users/42/resume.pdf'], [])
There is no "users" directory anywhere in the store, yet the listing shows one. This is why "renaming a folder" in object storage is expensive: it means copying and deleting every object with that prefix. (The MD5-based ETag here is a simplification; real services compute ETags in their own ways, for example differently for multipart uploads.)
How Object Storage Achieves Durability
Amazon S3 Standard is designed for 99.999999999% (eleven nines) durability of objects over a year, and stores data across multiple Availability Zones (separate data centres) in a region. How can a system promise that, when individual disks fail every day?
Replication
The simple way: keep 3 full copies on 3 different machines, ideally in different racks or data centres (the idea from Database Replication). Easy to understand and fast to read, but it costs 3× the raw storage.
Erasure Coding
The cheaper way: split each object into k data fragments and compute m extra parity fragments, then store all k + m fragments on different machines. Any k of them are enough to rebuild the object, so the system survives the loss of any m fragments.
Object ──split──→ D1 D2 D3 D4 D5 D6 D7 D8 (k = 8 data fragments)
──encode─→ P1 P2 P3 P4 (m = 4 parity fragments)
12 fragments on 12 different disks/nodes.
Any 8 of the 12 can rebuild the object → survives 4 failures.
Storage overhead = (k + m) / k = 12 / 8 = 1.5×
Method | Storage overhead | Failures tolerated | Trade-off |
|---|---|---|---|
3-way replication | 3× | 2 copies | Simple, fast reads and repairs |
Erasure coding 8 + 4 | 1.5× | Any 4 fragments | Half the storage, but more CPU and network to encode and rebuild |
Large object stores commonly use erasure coding for most data because, at petabyte scale, saving half the disks saves a fortune. (The exact schemes used inside commercial clouds are not all public; 8 + 4 is an illustration.)
Consistency
Since December 2020, Amazon S3 provides strong read-after-write consistency for PUT and DELETE operations on objects, at no extra cost: once a write succeeds, any following read or list sees it. Older articles that say "S3 is eventually consistent for overwrites" are out of date. Other object stores differ, so always check the documentation of the one you use (see Consistency Models).
Uploading Large Files: Multipart Upload
Uploading a 20 GB video as one HTTP request is fragile. If the connection drops at 19 GB, you start again. Multipart upload splits the file into parts that are uploaded independently (and in parallel), then combined:
Initiate the upload and get an upload ID.
Upload parts (numbered 1, 2, 3...) in any order, in parallel. Retry only the parts that fail.
Complete the upload with the list of parts; the store assembles the final object.
Current Amazon S3 limits (worth checking in the S3 documentation, since AWS raises them over time):
Item | Limit |
|---|---|
Single PUT upload | Up to 5 GB |
Part size (multipart) | 5 MiB to 5 GiB (the last part may be smaller) |
Number of parts | Up to 10,000 |
Maximum object size | 50 TB (raised from 5 TB in December 2025; precisely 10,000 parts × 5 GiB = 50,000 GiB ≈ 48.8 TiB) |
Worked example: what is the smallest part size you can use for a 1 TB file?
1 TB = 10^12 bytes, max 10,000 parts
minimum part size = 10^12 / 10,000 = 10^8 bytes ≈ 95.4 MiB
→ choose 100 MiB parts (10^12 / (100 × 2^20) ≈ 9,537 parts, under the limit)
AWS recommends considering multipart upload once objects reach about 100 MB.
Design Pattern: Metadata in the Database, Bytes in Object Storage
Do not put files in your database. Store the bytes in object storage and a small metadata row in the database:
files table (SQL)
┌─────────┬─────────┬──────────────────────────────┬──────────┬────────────┐
│ file_id │ user_id │ object_key │ size │ status │
├─────────┼─────────┼──────────────────────────────┼──────────┼────────────┤
│ 9001 │ 42 │ users/42/9001-resume.pdf │ 182340 │ UPLOADED │
└─────────┴─────────┴──────────────────────────────┴──────────┴────────────┘
Queries such as "show all files of user 42" hit the small, indexed table; the heavy bytes stay in cheap storage.
Direct Uploads with Pre-Signed URLs
If every upload passed through your application servers, they would spend all their bandwidth and memory shuffling bytes. Instead, the app hands the client a pre-signed URL: a URL with a signature that grants permission to upload (or download) one specific object for a limited time. The client then talks to object storage directly.
1. Client ──"I want to upload resume.pdf"──→ App server
2. App server: checks auth, creates DB row (status = PENDING),
signs a URL for key users/42/9001-resume.pdf, valid 15 min
3. App server ──pre-signed URL──→ Client
4. Client ──PUT file bytes──────────────────→ Object storage (app not involved)
5. Object storage ──"object created" event──→ App / queue
6. App marks row UPLOADED, maybe triggers virus scan or thumbnail job
Step 5 often uses an event notification into a queue (see Message Queues & Pub/Sub), so slow work like image resizing or video transcoding happens asynchronously.
Serving Files: Put a CDN in Front
For downloads, especially public images and videos, place a CDN in front of the bucket. Users in Chennai and Delhi get the file from a nearby edge server, and the bucket only serves cache misses (see Content Delivery Networks (CDN)). Private files are served with short-lived pre-signed download URLs or signed CDN URLs.
Storage Classes: Paying for How Often You Read
Object stores offer cheaper classes for data that is rarely accessed, in exchange for retrieval fees or slower access. In S3 these include S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA (infrequent access), S3 One Zone-IA, and the Glacier classes (Instant Retrieval, Flexible Retrieval and Deep Archive). There is also S3 Express One Zone for very low latency in a single zone.
Data | Access pattern | Sensible class |
|---|---|---|
New profile photos, current course videos | Read often | Standard (+ CDN) |
Last year's uploads | Read now and then | Infrequent access |
Compliance archives, old backups | Almost never; hours of wait is acceptable | Archive (Glacier Deep Archive) |
Lifecycle rules move data between classes automatically.
Back-of-the-Envelope: Storage for a Photo App
Using the method from Back-of-the-Envelope Estimation: 2 million photos uploaded per day, average 500 KB each.
Per day: 2,000,000 × 500 KB = 10^12 bytes = 1 TB/day
Per year: 1 TB × 365 = 365 TB/year (logical data)
Raw disk needed per year:
3-way replication: 365 × 3 ≈ 1.1 PB
Erasure code 8 + 4: 365 × 1.5 ≈ 0.55 PB
About a third of a petabyte of new logical data every year is routine for object storage and impossible to keep sensibly inside a relational database.
Distributed File Systems
Between NFS and object storage sit distributed file systems built for big data. Google File System (GFS, 2003 paper) and its open-source cousin HDFS split huge files into large chunks (64 MB in GFS; 128 MB by default in modern HDFS), replicate each chunk (3 copies by default), and keep all file and chunk metadata on a master node (GFS master / HDFS NameNode). They are optimised for very large files read sequentially by batch jobs. Today many data platforms store their data directly in object storage instead (a "data lake"), and run processing engines on top of it.
Common Mistakes
Storing images or PDFs as BLOBs in the main database. It makes the database huge, slow to back up and expensive to scale.
Routing every upload through the app server instead of using pre-signed URLs.
Treating object storage like a file system: appending to objects, renaming "folders", or doing many tiny random reads. Each of these is either impossible or slow.
Public buckets by accident. Misconfigured bucket permissions have leaked huge amounts of private data. Keep buckets private by default and share through signed URLs or a CDN.
Quoting old facts: "S3 is eventually consistent" (strong since 2020) or "max object size is 5 TB" (50 TB since December 2025).
Confusing durability with availability. Eleven nines of durability means data is very unlikely to be lost; it does not mean the service never has downtime. S3 Standard's availability design target is 99.99%.
Interview Questions
Q1. What is the difference between block, file and object storage?
Block storage exposes a raw disk of addressable blocks to one server (good for databases). File storage exposes a shared folder hierarchy over NFS/SMB (good for shared files and legacy apps). Object storage stores whole objects with keys and metadata in a flat namespace accessed over HTTP (good for massive amounts of unstructured data like images, videos and backups).
Q2. Why are uploads usually done with pre-signed URLs?
The client uploads directly to object storage, so the application servers do not spend bandwidth, memory and connection time relaying large files. The signature limits the permission to one object and a short time window.
Q3. With erasure coding 10 + 4, what is the storage overhead and how many failures are tolerated?
Overhead = (10 + 4) / 10 = 1.4×. Any 10 of the 14 fragments rebuild the object, so up to 4 fragment losses are tolerated.
Q4. Why is renaming a "folder" slow in object storage?
Folders do not exist; they are key prefixes. Renaming means copying every object to a new key and deleting the old one.
Q5. How would you design storage for user-uploaded videos?
Metadata in a database; original video uploaded via multipart upload with pre-signed URLs into object storage; an event triggers transcoding workers through a queue; transcoded outputs stored back in object storage; served to viewers through a CDN; lifecycle rules move old originals to cheaper storage classes.
Summary
Block storage: raw disks for one server; fastest; for databases and VMs.
File storage: shared folder trees over the network; for shared and legacy workloads.
Object storage: flat buckets of key → object, HTTP API, cheap, huge scale, very durable; the default for user content.
Durability comes from replication or erasure coding; erasure coding saves storage at the cost of computation.
Amazon S3 now offers strong read-after-write consistency and objects up to 50 TB, uploaded in parts via multipart upload.
Standard pattern: metadata in the DB, bytes in object storage, uploads via pre-signed URLs, downloads via a CDN.
Rows go in the database, bytes go in the bucket, and the CDN carries them to the user.