Lesson 1 of 6
Vertical vs Horizontal Scaling
"The cheapest, fastest, and most reliable components of a computer system are those that aren't there." — Gordon Bell
For four parts of this Volume you have learned to design a system — scope it, estimate it, weigh its trade-offs, wire up its data and its APIs. Now the traffic arrives. One server that answered in 40ms at launch is answering in 4 seconds at 10pm, and the graph only points one way. You have exactly two levers to pull, and every scaling story in the history of the web is some blend of the two: make the machine bigger, or add more machines. That is it. The whole discipline of scaling — caching, load balancing, replication, sharding, the lot — hangs off which of these two you reach for, when, and why. This is the first chapter of Scaling & Performance and the free preview, because it is the fork in the road everything after it depends on.
Your app is out of headroom. Do you buy a bigger box or buy more boxes — and what does each choice quietly cost you in money, complexity, and the ability to survive a machine dying at 3am?
Core Concepts
Scaling means adding capacity to handle more load. There are exactly two directions to add it. Scaling up — vertical scaling — replaces your machine with a more powerful one: more CPU cores, more RAM, faster disks. Scaling out — horizontal scaling — leaves the machine alone and adds more of them, splitting the load across the fleet. The names are worth fixing in your head with the picture:
Vertical: simplest thing that works
Vertical scaling is beautifully boring. You resize the instance (on a cloud, often a reboot and a bigger bill), and your exact same code runs faster. No distribution, no coordination, no new failure modes to reason about — the mental model you already have still holds. That simplicity is a real engineering asset, and it is why almost every system should start here.
But scaling up runs into three walls. First, a hard ceiling: there is a biggest machine you can buy, and once you're on it there is nowhere left to go. Second, the cost curve bends the wrong way — the top-end instance is not twice the price of the mid-range one for twice the power; it's often several times the price for the last increment of performance, because you're paying for exotic hardware few people need. Third, and most damning, one machine is a single point of failure: when it reboots, patches, or dies, your entire service is down. You have put all your eggs in one very expensive basket.
Horizontal: near-limitless, but you pay in complexity
Horizontal scaling answers all three. Need more capacity? Add another commodity box — there's no ceiling, just your budget. Those boxes are cheap, mass-produced hardware rather than exotic top-of-range parts, so the cost scales roughly linearly instead of bending upward. And because there are many of them behind a load balancer (Part II), one dying just means the balancer stops routing to it — the other machines carry on. Redundancy and high availability come almost for free once you're already running more than one.
The bill for all this arrives as complexity. The moment you have more than one app server, you are running a distributed system, with everything Part III warned you about: requests for the same user may land on different machines, machines fail independently, and "just read it from memory" stops being safe. Which brings us to the single idea that makes horizontal scaling possible at all.
A restaurant with one overworked chef has two ways to serve more diners. Scale up: hire one superhuman chef with more hands and a bigger stove — simple, but there's a limit to how fast one human can cook, and if they call in sick the kitchen is dark. Scale out: hire ten ordinary cooks and add a head waiter (the load balancer) to hand each new order to whichever cook is free. Far more meals per hour, and one cook going home sick barely dents service — but now the cooks must share one recipe book and one pantry instead of each keeping ingredients in their own pockets. That shared pantry is the hard part.
The comparison, side by side
| Dimension | Vertical (scale up) | Horizontal (scale out) |
|---|---|---|
| Cost curve | Bends upward — top-end hardware costs a premium per unit of power | Roughly linear — commodity boxes, pay as you grow |
| Scale ceiling | Hard limit: the biggest machine that exists | Near-limitless: keep adding nodes |
| Complexity | Low — same code, no distribution | High — distributed system, load balancing, coordination |
| Availability | One machine = single point of failure | Redundant by design; survives node loss |
| Failure impact | Whole service down when the box goes | One node down = lost fraction of capacity |
| Best when | Early stage; stateful tiers; simplicity matters most | You've outgrown one box; need HA and elastic scale |
Stateless is the key enabler
Here is the pivot the whole chapter turns on. To scale horizontally, your app servers must be stateless — they must hold no per-user data in their own memory between requests. Why? Because the load balancer is free to send a user's next request to any server in the fleet. If server A remembered you were logged in and server B has never heard of you, you get logged out at random. That is exactly what happens with naive in-memory sessions:
The fix is to push state out of the app server: into a shared store like Redis that every server can read, or onto the client as a signed token (a JWT the browser sends with each request). Now the app servers are interchangeable cattle, not pets — the load balancer can route to any of them, add new ones, or kill sick ones, and no user notices. "Sticky sessions" (pinning a user to one server) is the workaround people reach for to avoid this refactor, and it partly defeats the point: it re-introduces a mini single-point-of-failure per user and unbalances the fleet.
What actually happens when a team scales out for the first time:
• The session bug appears first. Add a second server and users start getting logged out at random — that's in-memory sessions meeting a load balancer. The fix is Redis-backed sessions or tokens, and it's non-negotiable.
• Local file uploads break next. A file written to server A's disk isn't on server B. Anything written to local disk is per-machine state in disguise — push it to object storage (S3).
• The database becomes the bottleneck. You add app servers happily until they all hammer one database, and now it is the ceiling. Scaling the app tier is the easy 20%; scaling the data tier is the other 80%.
The asymmetry: easy tier vs hard tier
Notice what stateless buys you. Scaling the stateless tier (your app/web servers) is genuinely easy: add boxes, put them behind a load balancer, done — and auto-scaling can even do it for you. Scaling the stateful tier (your databases) is the hard part, because data is state — you cannot just clone a database and hope, or two copies will disagree the instant someone writes. This is the asymmetry that defines real systems: the app tier scales out almost trivially, while the data tier needs the heavy machinery you met in Part III — replication (read replicas to spread reads) and sharding (splitting the data itself across machines to spread writes). When people say "scaling is hard," they almost always mean scaling the database.
The Academy is a Next.js app with a Drizzle/MySQL database on a single host — deliberately, honestly, vertically scaled. For its traffic that is the correct architecture: one well-sized box and a managed database serve every student, certificate, and quiz attempt with room to spare, and nobody has to reason about distributed sessions or shard keys. If it ever went viral, the first move would be modest — a bigger box (vertical), then move sessions to Redis and put two app servers behind a load balancer (horizontal, stateless), and only then add a read replica for the database. That order — cheap and simple first, distributed only when forced — is the whole pragmatic lesson, lived at small scale.
The pragmatic reality
Put together, the industry playbook is unglamorous and nearly universal: start vertical, go horizontal when you must. A single beefy box plus a managed database serves a startling amount of traffic — tens of thousands of users is not exotic. When one box genuinely isn't enough, you make the app tier stateless and scale it out behind a load balancer (with auto-scaling adding nodes as load rises and removing them as it falls, so you pay for what you use), and you scale the data tier with replicas and shards. Most large systems are exactly this shape: a horizontally-scaled stateless app tier in front of a carefully-scaled stateful data tier.
For decades, "scaling" meant scaling up — buy a bigger mainframe or SPARC server, the way banks and airlines did. Then in the early 2000s Google made a radical bet: instead of a few expensive, reliable machines, use thousands of cheap, unreliable commodity PCs and make the software tolerate their failures. Their 2003–2004 papers on GFS and MapReduce showed the industry you could scale out on hardware that was expected to break, and design around it. That inversion — reliability as a property of the system, not the machine — is the intellectual root of every cloud you deploy to today.
Common Mistakes
- Scaling out before you need to. Standing up Kubernetes, service meshes, and five microservices for an app three people use. You've bought web-scale complexity to serve local-shop traffic — and slowed the team down doing it.
- Assuming vertical has no limit. Riding the instance-size slider up until you're on the biggest (and priciest) box, with no plan for the day it isn't enough — and no redundancy the whole time.
- Scaling out a stateful app tier. Adding servers while sessions still live in local memory, then debugging "random logouts" for a week. Statelessness is the prerequisite, not an afterthought.
- Forgetting the database is the real ceiling. Ten app servers all pointed at one database just means you found the bottleneck faster. Scaling the stateless tier is easy; the data tier is the work.
- Treating one big box as a solution to availability. Vertical scaling gives you zero redundancy — a bigger single point of failure is still a single point of failure.
- Hand-waving the cost curve. "We'll just get a bigger instance" ignores that the top of the range costs a premium; past a point, commodity boxes are cheaper per unit of work.
Your app runs on a machine that costs $200/month and handles 5,000 requests/second. Traffic is set to hit 50,000 req/s — 10× — by year's end. Option A: one machine 10× as powerful. But the biggest instance your cloud offers tops out at ~7× your current one, and costs not 7× but 14× the price. Option B: ten of your current boxes behind a load balancer — $2,000/month in compute, plus the engineering cost of making the app stateless, plus a load balancer. Which is cheaper? Which survives a machine dying? And which one hits a wall the following year when you need 100×? Sit with the fact that Option A can be both more expensive and a dead end — that asymmetry is why "scale out" won the web.
Interview Corner
- Sketch the diagram. On paper, draw the same web app twice: once vertically scaled (one big box + database), once horizontally scaled (load balancer → three stateless app servers → shared Redis for sessions → database). Mark the single point of failure in each.
- Say the trade-off out loud. In one breath, explain why you'd start vertical and what specific signal would make you switch to horizontal — and name the one refactor that switch requires.
- Estimate the numbers. A box handles 5,000 req/s and costs $200/mo. You need 40,000 req/s. Compare the cost and the failure impact of one 8× box versus eight commodity boxes behind a load balancer. Which hits a ceiling sooner?
- Find the hidden state. List three things an app server might keep locally (sessions, uploaded files, an in-memory cache) and, for each, say where it must move so the server becomes stateless.
Chapter Summary
- Two ways to add capacity: vertical (a bigger machine) and horizontal (more machines). Every scaling strategy is a blend of these.
- Vertical is simplest — same code, no distribution — but has a hard ceiling, a cost curve that bends upward at the top, and is a single point of failure.
- Horizontal scales near-limitlessly on commodity hardware with built-in redundancy, but makes you run a distributed system with all its complexity.
- Stateless app servers are the key enabler of horizontal scaling: push session/state to a shared store (Redis) or the client (tokens) so any server can serve any request.
- The asymmetry that defines real systems: the stateless app tier scales out trivially; the stateful data tier is the hard part, needing replication and sharding.
- The pragmatic playbook: start vertical, go horizontal when you must — a single beefy box plus a managed database serves surprisingly large loads. Auto-scaling adds/removes nodes with demand.
- The anti-pattern is scaling prematurely — distributing before you need to buys web-scale complexity to serve small-scale traffic.
Adding machines is one way to serve more requests. But there's a cheaper, faster lever we've mentioned three times already and kept deferring: the fastest request is the one you never make, and the fastest query is the one you never run. Before we throw more hardware at load, we should stop doing redundant work — by remembering answers instead of recomputing them. That is the next chapter: Caching Strategies.
Editorial reflection — the instinct that separates engineers who scale systems from those who merely admire diagrams is knowing that the second machine is never free. Vertical scaling asks nothing of your architecture and everything of your wallet; horizontal scaling asks little of your wallet and everything of your architecture — chiefly, that you give up the comfortable fiction that a server can remember things. Most of scaling is learning to defer that second machine as long as honestly possible, and then, when the day comes, to have built something stateless enough that adding it is boring. Boring, at scale, is the highest praise there is.
Q&A · 0
Enrol to ask questions and join the discussion.
No questions yet — be the first to ask.