Chapter 23 of 33

Object Storage & File Storage

Almost every app you use stores "files": profile photos on Instagram, PDFs of notes, resumes uploaded on job portals, videos on YouTube, backups of your WhatsApp chats. A relational database is a poor home for these large binary blobs. They bloat the database, make backups slow, and waste expensive database memory and I/O on data that is simply read and written as a whole.

So where do these files go? Large systems use one of three kinds of storage: block storage, file storage and object storage. In this chapter we learn how they differ, why object storage (like Amazon S3) has become the default for user-uploaded content, and the design patterns used to upload and serve files at scale. We will build on SQL vs NoSQL Databases, Database Replication and Content Delivery Networks (CDN).

Object storage is a storage architecture that keeps data as objects in a flat namespace. Each object consists of the data itself, a unique key, and metadata, and is accessed as a whole through an HTTP API (PUT, GET, DELETE) rather than through a file system.


The Three Kinds of Storage

Block Storage

Block storage gives a machine a raw disk: a long list of fixed-size blocks, each with an address. It knows nothing about files. The operating system puts a file system (ext4, NTFS, XFS) on top of it, or a database manages the blocks directly.

  • Examples: your laptop's SSD, a SAN volume, Amazon EBS, Google Persistent Disk.

  • Very low latency and supports small random reads and in-place updates.

  • Usually attached to one server at a time (some cloud volumes allow multi-attach, but the servers must then coordinate writes themselves, for example with a cluster file system).

  • Best for: database data files, a virtual machine's boot disk, anything needing fast random I/O.

File Storage

File storage provides the familiar hierarchy of folders and files, shared over a network so that many machines can mount the same file system at the same time.

  • Examples: NFS and SMB file servers, NAS devices, Amazon EFS, Azure Files.

  • Supports normal file operations: open, seek, append, rename, lock, permissions.

  • Best for: shared folders, legacy applications that expect a file path, content management systems, home directories.

  • Harder to scale to billions of files, because maintaining a consistent directory tree with locks across many machines is expensive.

Object Storage

Object storage throws away the directory tree. Data is stored as objects inside buckets, and each object is found by its key, a string such as users/42/avatar.png. The slashes are just characters in the key; there are no real folders. You talk to it over HTTP: PUT to upload, GET to download, DELETE to remove.

  • Examples: Amazon S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, and self-hosted S3-compatible systems such as MinIO and Ceph.

  • Scales to practically unlimited objects and petabytes of data.

  • Objects are written and read as a whole; you cannot edit byte 500 of an object in place. To change it, you upload a new version.

  • Very cheap per GB, extremely durable, and accessible from anywhere via a URL.

Block:   [blk 0][blk 1][blk 2][blk 3] ...   ← OS/database decides what they mean

File:    /home
           └── ravi
                ├── notes.pdf               ← real folders, paths, locks
                └── photos/trip.jpg

Object:  bucket "cslp-uploads"
           key "users/42/avatar.png"   → data + metadata
           key "users/42/resume.pdf"   → data + metadata   ← flat map: key → object
           key "readme.txt"            → data + metadata

Block vs File vs Object

Feature

Block

File

Object

Unit of data

Fixed-size block

File in a folder tree

Object (data + metadata + key)

Access method

Disk protocols (OS sees a disk)

NFS / SMB, normal file paths

HTTP API (PUT/GET/DELETE)

In-place updates

Yes, any block

Yes, any byte

No, replace the whole object

Shared by many servers

Usually no

Yes

Yes, by anyone with access

Scale

Size of a volume

Large, but metadata becomes a bottleneck

Practically unlimited

Latency

Lowest (sub-millisecond)

Low

Higher (tens of milliseconds per request is common)

Cost per GB

Highest

Medium

Lowest

Typical use

Databases, VM disks

Shared folders, legacy apps

Images, videos, backups, logs, data lakes


Core Concepts of Object Storage

We will use Amazon S3 terms, since most other services copy them.

  • Bucket: a top-level container for objects, with its own access rules and settings. Bucket names must be unique within their namespace: by default, S3 general purpose bucket names are unique across all AWS accounts, and since March 2026 you can also opt into a namespace reserved for your own account and region.

  • Key: the full name of an object inside a bucket, e.g. notes/dbms/unit1.pdf.

  • Object: the bytes plus metadata. System metadata includes size, content type and last-modified time; you can add your own user metadata.

  • ETag: an identifier for a particular version of the object's content, used to check whether it changed (useful for caching).

  • Versioning: if enabled, overwriting or deleting an object keeps the older versions, which protects against accidents.

  • Lifecycle rules: automatically move objects to cheaper storage classes after some days, or delete them (e.g. "delete logs older than 90 days").

"Folders" Are Just Prefixes

When the S3 console shows you folders, it is really doing a LIST with a prefix and a delimiter (/). The toy store below shows how a flat key-value map can still look like a folder tree:

import hashlib


class ToyObjectStore:
    """A flat key -> (bytes, metadata) map. No real folders exist."""

    def __init__(self):
        self.buckets = {}

    def create_bucket(self, name):
        self.buckets[name] = {}

    def put(self, bucket, key, data, content_type):
        etag = hashlib.md5(data).hexdigest()       # fingerprint of the content
        self.buckets[bucket][key] = {
            "data": data,
            "meta": {"Content-Type": content_type, "Size": len(data), "ETag": etag},
        }
        return etag

    def get(self, bucket, key):
        obj = self.buckets[bucket][key]
        return obj["data"], obj["meta"]

    def list(self, bucket, prefix="", delimiter=None):
        """Like a LIST call: keys with a prefix; delimiter groups 'folders'."""
        keys, folders = [], set()
        for key in sorted(self.buckets[bucket]):
            if not key.startswith(prefix):
                continue
            rest = key[len(prefix):]
            if delimiter and delimiter in rest:
                folders.add(prefix + rest.split(delimiter)[0] + delimiter)
            else:
                keys.append(key)
        return keys, sorted(folders)


s3 = ToyObjectStore()
s3.create_bucket("cslp-uploads")
s3.put("cslp-uploads", "users/42/avatar.png", b"\x89PNG...", "image/png")
s3.put("cslp-uploads", "users/42/resume.pdf", b"%PDF-1.7...", "application/pdf")
s3.put("cslp-uploads", "users/77/avatar.png", b"\x89PNG..!", "image/png")
s3.put("cslp-uploads", "readme.txt", b"hello", "text/plain")

data, meta = s3.get("cslp-uploads", "users/42/resume.pdf")
print("GET users/42/resume.pdf ->", meta)

print("LIST prefix='' delimiter='/'        ->", s3.list("cslp-uploads", "", "/"))
print("LIST prefix='users/' delimiter='/'  ->", s3.list("cslp-uploads", "users/", "/"))
print("LIST prefix='users/42/'             ->", s3.list("cslp-uploads", "users/42/"))

Output:

GET users/42/resume.pdf -> {'Content-Type': 'application/pdf', 'Size': 11, 'ETag': '07fed6503f3ee63fe54dc7dad4efedac'}
LIST prefix='' delimiter='/'        -> (['readme.txt'], ['users/'])
LIST prefix='users/' delimiter='/'  -> ([], ['users/42/', 'users/77/'])
LIST prefix='users/42/'             -> (['users/42/avatar.png', 'users/42/resume.pdf'], [])

There is no "users" directory anywhere in the store, yet the listing shows one. This is why "renaming a folder" in object storage is expensive: it means copying and deleting every object with that prefix. (The MD5-based ETag here is a simplification; real services compute ETags in their own ways, for example differently for multipart uploads.)


How Object Storage Achieves Durability

Amazon S3 Standard is designed for 99.999999999% (eleven nines) durability of objects over a year, and stores data across multiple Availability Zones (separate data centres) in a region. How can a system promise that, when individual disks fail every day?

Replication

The simple way: keep 3 full copies on 3 different machines, ideally in different racks or data centres (the idea from Database Replication). Easy to understand and fast to read, but it costs 3× the raw storage.

Erasure Coding

The cheaper way: split each object into k data fragments and compute m extra parity fragments, then store all k + m fragments on different machines. Any k of them are enough to rebuild the object, so the system survives the loss of any m fragments.

Object ──split──→ D1 D2 D3 D4 D5 D6 D7 D8   (k = 8 data fragments)
        ──encode─→ P1 P2 P3 P4               (m = 4 parity fragments)

12 fragments on 12 different disks/nodes.
Any 8 of the 12 can rebuild the object → survives 4 failures.
Storage overhead = (k + m) / k = 12 / 8 = 1.5×

Method

Storage overhead

Failures tolerated

Trade-off

3-way replication

3×

2 copies

Simple, fast reads and repairs

Erasure coding 8 + 4

1.5×

Any 4 fragments

Half the storage, but more CPU and network to encode and rebuild

Large object stores commonly use erasure coding for most data because, at petabyte scale, saving half the disks saves a fortune. (The exact schemes used inside commercial clouds are not all public; 8 + 4 is an illustration.)

Consistency

Since December 2020, Amazon S3 provides strong read-after-write consistency for PUT and DELETE operations on objects, at no extra cost: once a write succeeds, any following read or list sees it. Older articles that say "S3 is eventually consistent for overwrites" are out of date. Other object stores differ, so always check the documentation of the one you use (see Consistency Models).


Uploading Large Files: Multipart Upload

Uploading a 20 GB video as one HTTP request is fragile. If the connection drops at 19 GB, you start again. Multipart upload splits the file into parts that are uploaded independently (and in parallel), then combined:

  1. Initiate the upload and get an upload ID.

  2. Upload parts (numbered 1, 2, 3...) in any order, in parallel. Retry only the parts that fail.

  3. Complete the upload with the list of parts; the store assembles the final object.

Current Amazon S3 limits (worth checking in the S3 documentation, since AWS raises them over time):

Item

Limit

Single PUT upload

Up to 5 GB

Part size (multipart)

5 MiB to 5 GiB (the last part may be smaller)

Number of parts

Up to 10,000

Maximum object size

50 TB (raised from 5 TB in December 2025; precisely 10,000 parts × 5 GiB = 50,000 GiB ≈ 48.8 TiB)

Worked example: what is the smallest part size you can use for a 1 TB file?

1 TB = 10^12 bytes, max 10,000 parts
minimum part size = 10^12 / 10,000 = 10^8 bytes ≈ 95.4 MiB
→ choose 100 MiB parts (10^12 / (100 × 2^20) ≈ 9,537 parts, under the limit)

AWS recommends considering multipart upload once objects reach about 100 MB.


Design Pattern: Metadata in the Database, Bytes in Object Storage

Do not put files in your database. Store the bytes in object storage and a small metadata row in the database:

files table (SQL)
┌─────────┬─────────┬──────────────────────────────┬──────────┬────────────┐
│ file_id │ user_id │ object_key                   │ size     │ status     │
├─────────┼─────────┼──────────────────────────────┼──────────┼────────────┤
│ 9001    │ 42      │ users/42/9001-resume.pdf     │ 182340   │ UPLOADED   │
└─────────┴─────────┴──────────────────────────────┴──────────┴────────────┘

Queries such as "show all files of user 42" hit the small, indexed table; the heavy bytes stay in cheap storage.

Direct Uploads with Pre-Signed URLs

If every upload passed through your application servers, they would spend all their bandwidth and memory shuffling bytes. Instead, the app hands the client a pre-signed URL: a URL with a signature that grants permission to upload (or download) one specific object for a limited time. The client then talks to object storage directly.

1. Client ──"I want to upload resume.pdf"──→ App server
2. App server: checks auth, creates DB row (status = PENDING),
               signs a URL for key users/42/9001-resume.pdf, valid 15 min
3. App server ──pre-signed URL──→ Client
4. Client ──PUT file bytes──────────────────→ Object storage   (app not involved)
5. Object storage ──"object created" event──→ App / queue
6. App marks row UPLOADED, maybe triggers virus scan or thumbnail job

Step 5 often uses an event notification into a queue (see Message Queues & Pub/Sub), so slow work like image resizing or video transcoding happens asynchronously.

Serving Files: Put a CDN in Front

For downloads, especially public images and videos, place a CDN in front of the bucket. Users in Chennai and Delhi get the file from a nearby edge server, and the bucket only serves cache misses (see Content Delivery Networks (CDN)). Private files are served with short-lived pre-signed download URLs or signed CDN URLs.


Storage Classes: Paying for How Often You Read

Object stores offer cheaper classes for data that is rarely accessed, in exchange for retrieval fees or slower access. In S3 these include S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA (infrequent access), S3 One Zone-IA, and the Glacier classes (Instant Retrieval, Flexible Retrieval and Deep Archive). There is also S3 Express One Zone for very low latency in a single zone.

Data

Access pattern

Sensible class

New profile photos, current course videos

Read often

Standard (+ CDN)

Last year's uploads

Read now and then

Infrequent access

Compliance archives, old backups

Almost never; hours of wait is acceptable

Archive (Glacier Deep Archive)

Lifecycle rules move data between classes automatically.


Back-of-the-Envelope: Storage for a Photo App

Using the method from Back-of-the-Envelope Estimation: 2 million photos uploaded per day, average 500 KB each.

Per day:   2,000,000 × 500 KB = 10^12 bytes = 1 TB/day
Per year:  1 TB × 365 = 365 TB/year (logical data)

Raw disk needed per year:
  3-way replication:   365 × 3   ≈ 1.1 PB
  Erasure code 8 + 4:  365 × 1.5 ≈ 0.55 PB

About a third of a petabyte of new logical data every year is routine for object storage and impossible to keep sensibly inside a relational database.


Distributed File Systems

Between NFS and object storage sit distributed file systems built for big data. Google File System (GFS, 2003 paper) and its open-source cousin HDFS split huge files into large chunks (64 MB in GFS; 128 MB by default in modern HDFS), replicate each chunk (3 copies by default), and keep all file and chunk metadata on a master node (GFS master / HDFS NameNode). They are optimised for very large files read sequentially by batch jobs. Today many data platforms store their data directly in object storage instead (a "data lake"), and run processing engines on top of it.


Common Mistakes

  • Storing images or PDFs as BLOBs in the main database. It makes the database huge, slow to back up and expensive to scale.

  • Routing every upload through the app server instead of using pre-signed URLs.

  • Treating object storage like a file system: appending to objects, renaming "folders", or doing many tiny random reads. Each of these is either impossible or slow.

  • Public buckets by accident. Misconfigured bucket permissions have leaked huge amounts of private data. Keep buckets private by default and share through signed URLs or a CDN.

  • Quoting old facts: "S3 is eventually consistent" (strong since 2020) or "max object size is 5 TB" (50 TB since December 2025).

  • Confusing durability with availability. Eleven nines of durability means data is very unlikely to be lost; it does not mean the service never has downtime. S3 Standard's availability design target is 99.99%.


Interview Questions

Q1. What is the difference between block, file and object storage?

Block storage exposes a raw disk of addressable blocks to one server (good for databases). File storage exposes a shared folder hierarchy over NFS/SMB (good for shared files and legacy apps). Object storage stores whole objects with keys and metadata in a flat namespace accessed over HTTP (good for massive amounts of unstructured data like images, videos and backups).

Q2. Why are uploads usually done with pre-signed URLs?

The client uploads directly to object storage, so the application servers do not spend bandwidth, memory and connection time relaying large files. The signature limits the permission to one object and a short time window.

Q3. With erasure coding 10 + 4, what is the storage overhead and how many failures are tolerated?

Overhead = (10 + 4) / 10 = 1.4×. Any 10 of the 14 fragments rebuild the object, so up to 4 fragment losses are tolerated.

Q4. Why is renaming a "folder" slow in object storage?

Folders do not exist; they are key prefixes. Renaming means copying every object to a new key and deleting the old one.

Q5. How would you design storage for user-uploaded videos?

Metadata in a database; original video uploaded via multipart upload with pre-signed URLs into object storage; an event triggers transcoding workers through a queue; transcoded outputs stored back in object storage; served to viewers through a CDN; lifecycle rules move old originals to cheaper storage classes.


Summary

  • Block storage: raw disks for one server; fastest; for databases and VMs.

  • File storage: shared folder trees over the network; for shared and legacy workloads.

  • Object storage: flat buckets of key → object, HTTP API, cheap, huge scale, very durable; the default for user content.

  • Durability comes from replication or erasure coding; erasure coding saves storage at the cost of computation.

  • Amazon S3 now offers strong read-after-write consistency and objects up to 50 TB, uploaded in parts via multipart upload.

  • Standard pattern: metadata in the DB, bytes in object storage, uploads via pre-signed URLs, downloads via a CDN.

Rows go in the database, bytes go in the bucket, and the CDN carries them to the user.