Rate Limiting & Throttling

"Every system has a breaking point. A rate limiter is how you decide where it is — instead of finding out at 3 a.m." — a lesson usually learned the expensive way

Your service is a shared resource with a finite budget: so many database connections, so much CPU, so many dollars per million calls to the payment provider behind it. Most of the time that budget is plenty, because most callers are polite. Then a script with a while (true) loop, a misconfigured mobile app retrying in a tight loop, a scraper, or an outright attacker points a firehose at one endpoint — and a service sized for a thousand requests a second is asked for a hundred thousand. Without a governor, the outcome is not "a bit slower for that one caller." It is the database saturating, every request queuing, timeouts cascading, and the service going down for everyone — including the well-behaved users who did nothing wrong.

A rate limiter is that governor. It caps how many requests a given caller may make in a given window, and when they exceed it, it refuses the excess — cheaply, deliberately, and early — so the expensive machinery behind it stays healthy. It is the first and most important line of defence in Part VI, because everything else we'll build (idempotency, pagination, observability, horizontal scale) assumes the service is still up to defend.

Why rate limit at all

Four distinct pressures push almost every production API toward a rate limiter, and it's worth separating them because they call for different limits:

• Prevent abuse and denial-of-service. A brute-force login attack, credential stuffing, a scraper harvesting your catalogue, or a plain flood all look the same to your server: far more requests than any legitimate user would make. A rate limit turns "unlimited attempts" into "five per minute," which is the difference between a password that can be guessed and one that can't.
• Ensure fair use. On a shared or multi-tenant API, one heavy customer must not starve everyone else. A per-tenant limit guarantees each caller a slice of capacity, so one team's runaway batch job doesn't degrade every other team's latency.
• Protect what's downstream. Your handler is rarely the bottleneck — the database, a third-party API, or a queue behind it is. A limiter at the front door bounds the load that can ever reach those slower, more fragile, often rate-limited-themselves dependencies. It's a shield for the whole chain, not just the edge.
• Control cost. When each request costs real money — an LLM token bill, an SMS, a per-call SaaS charge — an unbounded caller is an unbounded invoice. A rate limit is a spending cap with teeth.

The unifying idea: a request you refuse in a microsecond is one that never touches your database, your wallet, or your downstream partners. Refusing early is not a failure of hospitality; it's what keeps the service available for the requests that deserve to succeed.

Three words that get confused: rate limiting, throttling, load shedding

Interviewers and postmortems use these precisely, so it's worth pinning them down early:

• Rate limiting is a policy per caller: "you get N requests per window." It's about fairness and abuse — a contract with each client about how much they may ask for. Over the limit, you reject (or delay) that caller's excess.
• Throttling is the enforcement mechanism — how you slow or reject requests once a limit is hit. Rejecting outright with a 429 is one form; deliberately delaying a response to smooth the rate is another. People often use "throttling" loosely to mean "rate limiting," but strictly it's the act of applying the brakes, not the counting policy behind it.
• Load shedding is a global, health-driven decision: "the system as a whole is overloaded right now, so drop some requests regardless of who sent them." It's not about any one caller's fairness — it's triage to keep the service alive, often dropping the lowest-priority work first.

There's a fourth cousin, backpressure: instead of dropping work, the system signals upstream to slow down — a full queue that refuses new items, a stream that pauses its producer. Rate limiting says "no" at the door; backpressure says "not yet" up the pipe. In practice a robust service uses all of them: per-caller rate limits for fairness, load shedding as a last-resort survival valve when overall health degrades, and backpressure between its own internal stages. This chapter is mostly about the first.

The algorithms, and their trade-offs

At its heart every rate limiter answers one question — "has this caller had too many requests recently?" — and the algorithms differ in how they define "recently" and how much they cost to track. Here are the five you must know:

Two of these dominate real systems. Token bucket is the default for public APIs because real traffic is bursty — a user who's been idle for a minute and then loads a page firing eight requests at once is normal, not abusive, and the bucket happily accommodates that while still enforcing the long-run average. Leaky bucket is what you reach for when the thing you're protecting demands a constant feed and cannot tolerate spikes at all. The two sliding-window variants are refinements of the naive fixed window; the sliding window counter in particular is the sweet spot that most limiter libraries actually implement.

Building a token bucket, runnable

The best way to understand the token bucket is to run one. The version below takes a capacity (the burst size), a refill rate (tokens added per second), and a tryRemove(now) method that returns true (allow) or false (deny). Crucially, we pass the current time in as a fake now argument instead of reading the real clock — that makes the limiter a pure function of time and lets us simulate a burst deterministically. Run it:

Read the output carefully. At t=0 the bucket starts full with 5 tokens, so the first five requests are allowed and drain it to zero; the next three in the same instant are denied — that's the burst allowance doing its job. Then two seconds pass, the refill adds 2 × 1 = 2 tokens, and both later requests are allowed again. The bucket permitted a spike up to its capacity, then held the caller to the sustained rate. That single behaviour — forgiving bursts while enforcing the average — is why the token bucket is the workhorse of API rate limiting.

Contrast: the fixed window and its boundary flaw

The simplest limiter is a fixed-window counter: one integer per caller, reset when the clock window rolls over. It's cheap and obvious — and it has a famous flaw. Run this to see it:

The limit is "3 per second," yet six requests sail through between t=900ms and t=1050ms — roughly a 150-millisecond span — because three landed at the tail of one window and three at the head of the next. A determined caller can reliably push double your intended rate right at each boundary. That's exactly the gap the sliding-window variants close, and why the token bucket (which has no fixed boundary to game) is usually the better default.

Per what? The dimension you limit on

A limit is meaningless until you say per what. The same endpoint typically carries several limiters stacked on different keys:

• Per IP address. The default for unauthenticated traffic — before you know who someone is, their IP is all you have. Good against crude floods and anonymous scraping. Weak on its own: many users can share one IP (a corporate NAT, a university, a mobile carrier), and attackers can spread across many IPs (a botnet), so an IP limit that's tight enough to stop abuse may punish legitimate shared-IP users.
• Per user / per API key. The gold standard for authenticated traffic. It ties the limit to an identity you actually control and bill, survives IP changes, and lets you offer different tiers (free vs. paid). This is where fairness and monetization live.
• Per endpoint. Not every route deserves the same budget. POST /login and POST /password-reset want tight limits (abuse targets); a cheap GET /health wants a loose one. Limit expensive or sensitive endpoints harder than cheap read-only ones.
• Global. A ceiling on total traffic to the whole service (or one endpoint across all callers), independent of who's sending it — the last-resort valve that shades into load shedding: "no matter who you are, the service as a whole will accept no more than X right now."

In practice these compose: an anonymous request might face a per-IP limit and a per-endpoint limit and a global cap; an authenticated one swaps the per-IP for a per-key limit. The first bucket to run dry rejects the request.

Saying "no" correctly: 429, Retry-After, and rate-limit headers

When you reject a request for exceeding a limit, HTTP has an exact answer: 429 Too Many Requests. Using it correctly is what separates an API clients can cooperate with from one they'll hammer blindly. (Status codes in general were covered in Part II — Error Handling & Status Codes — this is the reliability-specific one.)

A good 429 tells the caller not just no, but when to come back and how much budget they had:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1706616000
Content-Type: application/json

{ "error": "rate_limit_exceeded", "message": "Too many requests. Retry in 30s." }

• Retry-After — the single most important header on a 429. It tells the client how long to wait (seconds, or an HTTP date) before retrying. A well-behaved client reads and obeys it instead of retrying immediately and making the overload worse.
• X-RateLimit-Limit — the ceiling for the current window (here, 100).
• X-RateLimit-Remaining — how many requests are left in this window. Send this on every response, not just 429s, so a good client can self-throttle before it ever hits the wall.
• X-RateLimit-Reset — when the window resets (commonly a Unix timestamp), so the client knows when its budget refreshes.

The X-RateLimit-* headers are a de-facto convention rather than a formal standard (an IETF RateLimit-* standard is emerging, and providers vary in exact names), but the intent is universal: publish the limit, what's left, and when it resets, so clients can behave. Retry-After, by contrast, is a standard HTTP header and works with both 429 and 503.

Distributed rate limiting: one limit across many servers

Everything so far assumed a single process with the counter in its own memory. Real backends run many instances behind a load balancer (the whole point of Part VI's later chapters). Now the naive approach breaks: if each of five instances keeps its own in-memory token bucket for a "100/min" limit, a caller spread across all five effectively gets 500/min. The limit you advertised is a fifth of the limit you actually enforce.

The fix is a shared counter that all instances read and update — and the standard home for it is Redis, the same in-memory store we leaned on for distributed caching in the Caching in the Backend chapter (Part V), and which you first met as a shared cache back in System Design (Vol 5) and Databases (Vol 7). A rate limiter is, after all, just a counter with a TTL, which is precisely what an in-memory key/value store excels at:

• The crudest version: INCR ratelimit:{key}:{window} and set an expiry on first touch; if the returned count exceeds the limit, reject. That's a distributed fixed-window limiter in two commands.
• A subtlety: the "check the count, then increment" sequence must be atomic, or two instances can both read "99," both increment, and both allow — overshooting the limit. Redis handles this with atomic operations and, for token/sliding-window logic, a small Lua script that runs the whole check-and-decrement on the server as one indivisible step. (This is exactly the serialize the decision discipline from concurrency and transactions — Part V's Transactions in Request Handlers.)

The trade-off is a network hop per request (sub-millisecond on a local network, but not free) and a new dependency: if Redis is down, you must decide fail-open (allow requests, prioritising availability) or fail-closed (reject, prioritising protection). Most APIs fail open on the limiter — a rate limiter that takes your service down when it breaks has defeated its own purpose — while keeping a coarse in-process cap as a backstop.

Where the limiter lives

A rate limiter should run as early in the request path as possible — every layer a rejected request has to traverse is wasted work. Two common homes, both introduced earlier in this volume:

• At the gateway / edge. An API gateway, reverse proxy, or CDN (Part III — API Gateways & BFF) can enforce limits before the request ever reaches your application code, protecting all services behind it uniformly and absorbing floods at the cheapest possible layer. This is the ideal place for coarse per-IP and global limits.
• As application middleware. A limiter implemented as middleware (Part II — Middleware) runs inside your app, before your handler. It's less efficient than the edge (the request already reached your process) but far more aware: it knows the authenticated user, the specific endpoint, the cost of the operation, and can apply nuanced per-user, per-endpoint limits the gateway can't easily express.

Mature systems use both: crude, cheap limits at the edge to shrug off volumetric attacks, and precise, identity-aware limits in middleware for fairness and tiering. The principle is the same as validation and auth — reject the clearly-bad as early as you can, and reserve the expensive, context-rich checks for what makes it through.

What clients should do: back off, don't hammer

A rate limiter only works if clients cooperate, and the correct client behaviour on a 429 is the mirror image of everything above. When a caller receives a 429, it should:

1. Honour Retry-After if present — wait exactly that long, no less.
2. Otherwise, back off exponentially with jitter — wait 1s, then 2s, then 4s… with a random fraction added so a thundering herd of clients doesn't all retry in perfect lockstep and re-create the spike.
3. Never retry in a tight loop. Immediately re-sending a rejected request just burns your own budget and deepens the overload — the exact opposite of what a 429 asks for.

There's a critical partnership here with the next chapter, Idempotency & Retries: retrying is only safe if the operation is idempotent — if re-sending the same request can't accidentally charge a card or create a duplicate order. Rate limiting tells clients when to retry; idempotency is what makes retrying safe. The two together are how a distributed system stays both available and correct under stress. We'll build the idempotency half next.

Common Mistakes

• No Retry-After on a 429. You reject the request but don't say when to come back, so the client retries immediately — turning your defensive 429 into a retry storm that makes the overload worse.
• Rate limiting on the wrong key. Limiting per IP when users sit behind a shared NAT punishes a whole office for one bad actor; limiting per user when the attacker is unauthenticated does nothing. Match the key to who you're actually trying to constrain.
• Per-instance counters in a multi-server deployment. Each instance enforces its own slice, so the real limit is N × what you advertised (or a fraction of it, depending on routing). Use a shared store for anything that must be enforced globally.
• Non-atomic check-then-increment. "Read the count, decide, then write it back" lets two concurrent requests both read the old value and both get allowed, overshooting the limit. The decision must be a single atomic operation.
• A limiter that fails closed silently. If Redis hiccups and your limiter rejects everything, you've built an outage generator. Decide fail-open vs. fail-closed deliberately, and usually fail open with a backstop.
• Fixed window where bursts matter. Using the naive fixed-window counter and being surprised when callers reliably push 2× your rate at the window boundary. Prefer token bucket or a sliding-window counter.
• Limiting reads and writes the same. A cheap GET and an expensive POST /reports shouldn't share a budget. Weight limits by cost (token buckets make this easy — charge more tokens for pricier operations).

Summary

• A rate limiter caps how many requests a caller may make in a window and refuses the excess early and cheaply, so abuse, accidents, and floods can't take the service down for everyone.
• Rate limit to prevent abuse/DoS, ensure fair use, protect downstream dependencies, and control cost — a request refused at the door never touches your database, wallet, or partners.
• Know the five algorithms and their trade-offs: fixed window (cheap, boundary-doubling flaw), sliding window log (accurate, memory-heavy), sliding window counter (the pragmatic favourite), token bucket (allows bursts — the API default), leaky bucket (smooths to a constant rate).
• Choose the key deliberately: per-IP for anonymous traffic, per-user/API-key for authenticated fairness and tiering, per-endpoint to guard expensive/sensitive routes, and a global cap as a survival valve. They compose.
• Reject with 429 Too Many Requests plus Retry-After, and publish X-RateLimit-Limit/Remaining/Reset on every response so clients can self-throttle before hitting the wall.
• Across many instances, keep the counter in a shared store (Redis) with an atomic check-and-decrement; decide fail-open vs. fail-closed deliberately. Enforce coarse limits at the gateway/edge and precise, identity-aware limits in middleware.
• Distinguish rate limiting (per-caller policy) from throttling (the enforcement) and load shedding / backpressure (global, health-driven survival). Good clients respond to a 429 by backing off with jitter, never retry-looping.

Next we'll take up the other half of surviving retries: idempotency — designing operations so that a client (or a network) safely re-sending the same request can't double-charge a card or create a duplicate order, which is exactly what makes the backoff-and-retry behaviour a limiter encourages safe to do.