Rate Limiting: Protecting APIs from Abuse

10:00 AM, and the Website Falls Over

Every Indian student who has tried to book a Tatkal train ticket knows this feeling. At exactly 10:00 AM, the IRCTC booking window opens for AC classes, and for a few seconds the page freezes, spins, or throws an error before anything happens. It is not because the IRCTC servers are badly built. It is because in the same second, hundreds of thousands of people across the country click "Book Now" simultaneously, each click firing off a request to the same set of servers. No computer, however powerful, can process an unlimited number of requests per second. Somewhere, a decision has to be made about how many requests get handled right away, which ones wait, and which ones get turned away entirely so that the system does not collapse for everyone.

This is the exact problem that rate limiting solves. It is not a workaround or a symptom of bad engineering — it is a deliberate, carefully designed piece of every serious backend system, from IRCTC to GitHub to any app that talks to another app over the internet. This chapter is about how it actually works: the algorithms, the numbers, the trade-offs, and the code you would write to build one yourself.

What an API Request Actually Costs a Server

Recall that an API (Application Programming Interface) is how one piece of software asks another for something — your browser asking IRCTC's server for seat availability, or a weather app asking a government server for today's forecast. Each of these asks is called a request, and the server's answer is a response.

A request is never free. When a server receives one, it typically has to authenticate who is asking, read or write data in a database, possibly do some computation, and then package a response. Each of these steps consumes CPU time, memory, and database connections — all of which exist in limited supply on any real machine. A server that can comfortably handle 2,000 requests per second might start timing out, returning errors, or crashing outright at 10,000 requests per second, even if every single one of those requests is from an honest, non-malicious user.

This is the insight that surprises most beginners: a server does not need to be under attack to be overwhelmed. A flash sale, a cricket score going viral, or a Tatkal window opening can generate more legitimate traffic in one second than the server was ever designed to handle. If nothing intervenes, the server slows down for every user, including the ones being perfectly reasonable — and past a certain point, it stops responding entirely. Rate limiting exists to prevent exactly this: it decides, in advance and by rule, how many requests any one source is allowed to make in a given period of time, and it rejects the rest cleanly rather than letting the whole system degrade for everyone.

Defining a Rate Limit Precisely

A rate limit is usually written as a pair of numbers: a maximum number of requests, R, allowed within a time window, T. For example, "100 requests per minute" means R = 100 and T = 60 seconds. This single idea — cap the count of events within a period of time — is the seed from which every rate-limiting algorithm grows. The interesting engineering problem is not the idea itself; it is how you implement "count events within a time period" efficiently, fairly, and without leaving loopholes. There are a few standard algorithms, each making a different trade-off. We will build the two most important ones from scratch.

Algorithm 1: The Fixed Window Counter

The simplest possible approach: pick a window size, say 60 seconds, and keep a counter for each user that resets to zero every time a new 60-second window begins. Every incoming request checks the counter for the current window; if it is below the limit, the request is allowed and the counter increments, otherwise the request is rejected.

requests_this_window = {}   # maps (user_id, window_number) to a count
WINDOW_SECONDS = 60
LIMIT = 100

def current_window(timestamp):
    return int(timestamp // WINDOW_SECONDS)

def allow_request(user_id, timestamp):
    window = current_window(timestamp)
    key = (user_id, window)
    count = requests_this_window.get(key, 0)
    if count >= LIMIT:
        return False          # limit reached, reject
    requests_this_window[key] = count + 1
    return True

This works, and it is cheap: one dictionary lookup and one increment per request. But it has a serious flaw right at the window boundary, worth tracing through with real numbers to see exactly why.

Suppose the limit is 100 requests per minute, and windows are anchored to the clock, so one window is 11:59:00–11:59:59 and the next is 12:00:00–12:00:59. A client sends 100 requests between 11:59:58 and 11:59:59 — all counted in the first window, all allowed, since the count never exceeds 100. The clock ticks over to 12:00:00, the counter for this user resets to zero, and the client immediately sends 100 more requests between 12:00:00 and 12:00:01 — all allowed again, since it is a fresh window with a fresh counter.

Add it up: 200 requests were let through in roughly three seconds, even though the rule was "100 per minute," implying an intended average of under 2 requests per second. The server just absorbed a burst more than 30 times its intended average rate, purely because the two 100-request bursts straddled a window boundary. The fixed window counter never lies about its own window — it genuinely never exceeds 100 within any single window — but it says nothing about what can happen across two adjacent windows, and that gap is exactly what a burst can exploit.

Algorithm 2: The Token Bucket

Think about how your school canteen might handle a limited resource fairly. Suppose every student is issued a card that can hold at most 5 lunch coupons. Once every minute, the canteen's system automatically adds 1 new coupon to every card — but a card can never hold more than 5 at once; extra coupons are simply not added once it is full. Each time a student buys a snack, one coupon is spent. If a student has saved up all 5 coupons, they can buy 5 snacks back-to-back in the same minute. If their card is empty, they must wait for the next coupon before buying anything.

This is precisely how a token bucket rate limiter works, with "coupons" renamed to tokens. A bucket has a maximum capacity (how many tokens it can hold at once — this sets the size of the burst a client is allowed) and a refill rate (tokens added per second — this sets the long-run sustainable rate). Every request tries to remove one token; if a token is available, the request proceeds and the token is spent; if the bucket is empty, the request is rejected. Unlike the fixed window counter, there is no sharp reset moment where the whole counter snaps back to zero — tokens trickle in continuously, so there is no boundary to exploit.

Here is a complete implementation:

import time

class TokenBucket:
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity          # max tokens the bucket can hold
        self.tokens = capacity            # bucket starts full
        self.refill_rate = refill_rate    # tokens added per second
        self.last_check = time.time()

    def allow_request(self):
        now = time.time()
        elapsed = now - self.last_check
        self.last_check = now

        # Add tokens earned during the elapsed time, but never
        # let the bucket hold more than its capacity
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)

        if self.tokens >= 1:
            self.tokens -= 1
            return True      # allowed, one token spent
        return False         # bucket empty -- reject with HTTP 429

Let us trace this exactly, by hand, with capacity = 5 and refill_rate = 1 token per second, starting with a full bucket (tokens = 5):

• Requests 1 through 5 all arrive at t = 0 seconds, one after another. Elapsed time between each is effectively 0, so no refill happens between them. Tokens go 5 → 4 → 3 → 2 → 1 → 0. All five are allowed — this is the burst capacity in action.
• Request 6 also arrives at t = 0 seconds. Tokens = 0, and 0 is not ≥ 1, so this request is rejected.
• Request 7 arrives at t = 3 seconds. Elapsed since the last check is 3 seconds, so tokens gained = 3 × 1 = 3. New token count = min(5, 0 + 3) = 3, which is ≥ 1, so the request is allowed, spending one token and leaving tokens = 2.
• Request 8 arrives at t = 3.5 seconds. Elapsed = 0.5 seconds, tokens gained = 0.5. New token count = min(5, 2 + 0.5) = 2.5, which is ≥ 1, so it is allowed, leaving tokens = 1.5.

Notice what this achieves: a client that has been idle can burst up to 5 requests instantly (useful for, say, loading five images on a page load), but it can never sustain more than 1 request per second on average over a long period, because that is exactly how fast the bucket refills. The two parameters — capacity and refill_rate — let a system designer independently tune "how big a burst do I tolerate" versus "what is the long-run rate I can sustain," and this makes the token bucket far more resistant to boundary tricks than the fixed window counter.

Two More Approaches, Briefly

Real systems sometimes use two other designs, and it is worth knowing the trade-offs:

• Leaky bucket is the mirror image of the token bucket. Requests arrive and queue up in a bucket; the server processes (drains) them at a fixed, constant rate no matter how bursty the arrivals were. This smooths traffic perfectly but adds waiting time for anything above the drain rate, and if the queue fills up, new requests are dropped.
• Sliding window log keeps a timestamp for every request a client has made recently, and counts how many timestamps fall within the last T seconds before allowing a new one. This is the most accurate algorithm — it has no boundary flaw at all — but it costs more memory, since it stores a growing list of timestamps per client instead of a single number.

The token bucket is popular in practice because it strikes a good balance: it is cheap to store (just two numbers per client), it allows controlled bursts, and it has none of the fixed window's boundary problem.

What the Client Sees: HTTP 429

When a server rejects a request because of a rate limit, it does not simply go silent. The HTTP protocol defines a specific status code for this exact situation: 429 Too Many Requests. A well-built API also sends a Retry-After header telling the client how many seconds to wait before trying again:

HTTP/1.1 429 Too Many Requests
Retry-After: 3
Content-Type: application/json

{"error": "rate_limit_exceeded", "message": "Try again in 3 seconds"}

A concrete example: GitHub's REST API allows an authenticated request about 5,000 requests per hour. If you wrote a Python script to fetch information about every repository in a large organisation and looped without checking your remaining quota, you would eventually receive a 429. The correct fix is not to remove the limit — you cannot — but to slow the script down, batch requests more efficiently, or wait for the reset time the API reports back to you.

Who Gets Limited: Per-User, Per-IP, or Global?

An important design decision is what counts as "one client" for the purpose of counting requests. Three common choices:

• Per-IP-address: simple to implement, but unfair in a very common Indian scenario — a school, cybercafé, or apartment complex where dozens of devices share one public IP address through NAT (Network Address Translation). If the limit is applied per IP, one heavy user, or one bug in one student's code, can exhaust the quota for every other device on that same network, even though they had nothing to do with it.
• Per-user or per-API-key: the server issues each registered client a unique key, and counts requests against that key rather than the network address. This is fairer — it matches a limit to an actual account, no matter how many people share a Wi-Fi connection — and it is why almost every serious API requires signing up for a key before making requests, partly for exactly this reason.
• Global: a single limit shared by the entire system, used to protect a downstream resource (like a database or a third-party service the whole application depends on) regardless of who is asking. This is common as an extra safety net layered on top of per-user limits, not a replacement for them.

Two Misconceptions to Unlearn

Misconception: "Rate limiting exists only to stop hackers." Attackers are one reason, but a smaller one than most people assume. The IRCTC Tatkal crowd is not attacking anything — every one of those requests comes from an honest passenger trying to book a real seat. Similarly, a bug in your own code — a loop that forgot to add a delay, or that retries immediately after every failure — can hammer a server far harder than most deliberate attackers bother to. Rate limiting's everyday job is protecting a system from ordinary, well-intentioned overload; stopping abuse is a secondary benefit on top of that.

Misconception: "Getting a 429 response means my code is broken." A 429 is not an error in the sense of a bug — it is an expected, documented part of the API's contract, exactly like how "sold out" is an expected answer from a ticket counter, not a malfunction. The mistake would be reacting to a 429 by immediately retrying at full speed, which only makes things worse for you and the server. The correct client behaviour is to respect the Retry-After value, or if none is given, to use exponential backoff — wait 1 second, then if still rejected wait 2 seconds, then 4, then 8, doubling each time — so a client under pressure backs off gracefully instead of hammering the server harder the more it gets rejected.

How a Token Bucket Decides: A Diagram

Check Your Understanding

1. A fixed window counter allows 60 requests per minute. A client sends 60 requests at 10:59:59 and 60 more at 11:00:01. How many total requests were let through in that roughly 2-second span, and what does this reveal about the algorithm?
   Answer: 120 requests in about 2 seconds. It reveals the fixed window's boundary flaw: each window individually obeys the 60-request limit, but two adjacent windows can be exploited together to push through nearly double the intended rate right at the reset moment.
2. A token bucket has capacity = 10 and refill_rate = 2 tokens/second, and starts full. Ten requests arrive instantly at t = 0. Then, 0.5 seconds later, three more requests arrive back-to-back. How many of these three are allowed?
   Answer: After the first ten, tokens = 0. At t = 0.5s, tokens gained = 0.5 × 2 = 1, so tokens = min(10, 0+1) = 1. The first of the three new requests is allowed (tokens drop to 0); the next two arrive with no further elapsed time, so tokens stay at 0 and both are rejected with a 429.
3. Why is limiting requests strictly by IP address unfair to students on a shared school or hostel Wi-Fi connection?
   Answer: Many devices behind a NAT router share one public IP address, so an IP-based limit is really a limit on the whole building, not on any one device. One student's heavy or buggy usage can exhaust the quota and lock out everyone else sharing that IP, which is why per-user or per-API-key limiting is fairer.
4. In the token bucket, what does raising the capacity control, and what does raising the refill_rate control? Are they the same thing?
   Answer: No. Capacity controls how large a single burst of requests can be handled instantly; refill_rate controls the long-run sustainable average rate once any burst has been used up. A system can allow big bursts with a small sustained rate (high capacity, low refill_rate) or the opposite, independently.

Summary

• A server has finite capacity; without limits, a burst of even entirely honest traffic (a Tatkal window opening, a flash sale) can overwhelm it for everyone. Rate limiting deliberately rejects some requests to keep the system usable for the rest.
• A rate limit is defined as R requests per T seconds for a given client.
• The fixed window counter resets a count to zero at fixed intervals; it is simple but allows up to roughly 2× the intended rate right at a window boundary.
• The token bucket holds up to capacity tokens, refills continuously at refill_rate tokens/second, and spends one token per allowed request; it supports controlled bursts without the boundary flaw.
• Leaky bucket smooths traffic to a constant output rate by queuing; sliding window log is the most accurate but the most memory-hungry, since it stores every recent request timestamp.
• A rejected request returns HTTP 429 Too Many Requests, ideally with a Retry-After header; well-behaved clients back off (often exponentially) rather than retrying immediately.
• Limits can be applied per IP, per user/API key, or globally; per-user limiting is generally fairer than per-IP, since many real devices can share one IP address.
• Rate limiting protects against ordinary overload from real users at least as often as it stops deliberate abuse, and a 429 is a normal, expected response, not evidence of a bug.