Load Balancing: Distributing Request Traffic

One Counter, One Thousand Passengers

Picture the IRCTC Tatkal booking window opening at 10:00 AM sharp. In the first thirty seconds, tens of thousands of people across the country hit "Book Now" for the same handful of trains. If IRCTC ran on a single computer answering one request after another, that computer would fall hopelessly behind within the first second — new requests would pile up faster than it could ever finish the old ones, and everyone's screen would freeze or time out. This is not a hypothetical. It is exactly why ticket-booking systems, UPI payment apps, and exam result portals occasionally slow to a crawl when everyone logs in at once: too many requests arriving for the machinery to keep up.

The obvious fix is to not use one computer. Set up ten identical computers, each capable of processing booking requests independently, and split the incoming crowd across them. But this raises a genuinely tricky question that this chapter is about: when a request arrives, which of the ten computers should handle it? Send too many to one machine and it becomes the new bottleneck, even while its neighbours sit idle. Get this decision right, and ten machines can serve almost ten times the traffic. Get it wrong, and you might barely do better than one. The component that makes this decision, request by request, is called a load balancer, and the rules it uses are load balancing algorithms.

Servers, Requests, and Why One Isn't Enough

Before going further, let's be precise about the vocabulary, since the rest of the chapter depends on it. A server here means a running program (often on its own machine or virtual machine) that can accept a request — for example, "check seat availability on train 12951" — do the necessary work, and send back a response. A single server has a real, physical limit: its processor can only execute so many instructions per second, its memory can hold only so much data at once, and its network connection can only carry so many bytes per second. When requests arrive faster than the server can finish them, they queue up, and every person waiting in that queue experiences it as "the website is slow."

The standard engineering answer is horizontal scaling: instead of buying one enormously powerful (and enormously expensive) machine, run the same application on several ordinary machines side by side. If a school library has one librarian who can help 20 students an hour, and 100 students show up in the same hour, the fix isn't to somehow make one librarian work five times faster — it's to open five help desks, each with its own librarian, and direct students to whichever desk is free. Load balancing is precisely this direction-giving function, applied to computers instead of librarians.

The Load Balancer: A Traffic Controller, Not a Copier

A load balancer sits between the clients (the students' laptops and phones sending requests) and the pool of servers (the librarians). Every request from the outside world arrives at the load balancer first. The load balancer does not process the request itself — it picks exactly one server from the pool and forwards the request there, then relays that server's response back to the client. The client never even needs to know how many servers exist behind the load balancer; as far as it is concerned, there is one system to talk to.

It is worth being careful here about what a load balancer is not. It is not a router that simply moves packets toward a destination address written on them — a router doesn't choose between several equally-valid destinations based on their current workload; a load balancer does. And it does not duplicate every request to every server (that would multiply the total work by the number of servers, achieving nothing). It picks one server per request, using a specific rule. The rest of this chapter is about what those rules can be, and why some are much better than others.

Algorithm 1: Round Robin

The simplest possible rule is round robin: keep a numbered list of servers, and cycle through them in order, wrapping back to the start after the last one. If you have three servers S1, S2, S3, then request 1 goes to S1, request 2 to S2, request 3 to S3, request 4 back to S1, and so on. This is exactly like a teacher distributing notebooks to three rows of a classroom one at a time, row 1, row 2, row 3, row 1, row 2, row 3 — everyone ends up with (roughly) the same number of notebooks.

In code, the "next server to use" is just an index that increases by one each time, wrapping around using the remainder (modulo) operator:

class RoundRobinBalancer:
    def __init__(self, servers):
        self.servers = servers
        self.next_index = 0

    def get_server(self):
        server = self.servers[self.next_index]
        self.next_index = (self.next_index + 1) % len(self.servers)
        return server

lb = RoundRobinBalancer(["S1", "S2", "S3"])
for i in range(1, 7):
    print(f"Request {i} -> {lb.get_server()}")

Let's trace this line by line, because tracing is how you actually confirm code is correct rather than just hoping it is. self.servers is ["S1", "S2", "S3"] and self.next_index starts at 0. Call 1: server = self.servers[0] which is "S1"; then next_index = (0 + 1) % 3 = 1. Call 2: server = self.servers[1] = "S2"; next_index = (1+1) % 3 = 2. Call 3: server = self.servers[2] = "S3"; next_index = (2+1) % 3 = 0 — it has wrapped around. Call 4 repeats call 1's logic exactly, giving "S1" again. Continuing this by hand for all six requests gives:

Request 1 -> S1
Request 2 -> S2
Request 3 -> S3
Request 4 -> S1
Request 5 -> S2
Request 6 -> S3

Each server receives exactly two of the six requests. The modulo operator is what makes the "wraparound" work — it is the same idea as clock arithmetic, where 13 o'clock wraps back to 1 o'clock on a 12-hour clock.

Why Round Robin Alone Isn't Enough

Round robin guarantees that every server receives an equal number of requests. It does not guarantee that every server receives an equal amount of work — and that distinction matters enormously in practice, because not all requests take the same time to finish. A request to "search available train seats between two stations on a given date" involves scanning a large table and can take several seconds. A request to "confirm that a seat number exists" is nearly instant.

Consider three servers S1, S2, S3, and five requests arriving one per second, where R1 (assigned to S1) happens to be a slow 5-second search, while R2, R3, R4, R5 are all quick 1-second lookups. Round robin assigns purely by arrival order, with no idea how long each request will take:

R4 is forced onto S1 purely because it is "S1's turn" in the cycle, even though S1 is still grinding through R1 and S2 and S3 have long since gone idle. R4 has to wait in a queue behind R1 until t=5, even though two other servers were sitting completely free. This is the core weakness of round robin: it distributes request count evenly, but has no way to notice that one server is currently overloaded while others are idle.

Algorithm 2: Least Connections

A smarter rule is least connections: track how many requests each server is currently handling (its active connection count), and always send the new request to whichever server has the fewest. When a server finishes a request, its count goes back down. This directly fixes the blind spot above — a server stuck on a long-running request will have a high count and simply won't be chosen again until it catches up.

class LeastConnectionsBalancer:
    def __init__(self, servers):
        self.active = {s: 0 for s in servers}

    def assign(self):
        server = min(self.active, key=self.active.get)
        self.active[server] += 1
        return server

    def finish(self, server):
        self.active[server] -= 1

lb = LeastConnectionsBalancer(["S1", "S2", "S3"])
r1 = lb.assign()   # goes to whichever server has fewest active requests
r2 = lb.assign()
lb.finish(r2)      # r2's server is now free again
r3 = lb.assign()
print(r1, r2, r3)

Tracing this: self.active starts as {"S1": 0, "S2": 0, "S3": 0}. In Python, dictionaries remember the order keys were inserted, and min(dict, key=dict.get) returns the first key that achieves the minimum value when there's a tie. So r1 = lb.assign(): all three are tied at 0, "S1" is first, so r1 = "S1" and active["S1"] becomes 1. r2 = lb.assign(): now active is {"S1":1, "S2":0, "S3":0}; the minimum is 0, and "S2" is the first key with that value, so r2 = "S2", active["S2"] becomes 1. lb.finish(r2) decrements active["S2"] back to 0, giving {"S1":1, "S2":0, "S3":0}. r3 = lb.assign(): minimum is again 0, and "S2" is still the first key with value 0 (S3 is also 0, but S2 comes first in the dictionary), so r3 = "S2". The printed output is S1 S2 S2. Notice S3 was never used in this trace — least connections has no memory of "whose turn it is," it only ever looks at current load, so a server can legitimately be skipped for a while if others keep freeing up faster. Real load-balancer implementations typically break exact ties by rotating through the tied servers rather than always favouring the same one, to avoid this kind of skew, but the core rule — always pick the least-loaded — is unchanged.

Applying least connections to the earlier five-request scenario: when R4 arrives at t=3, the active counts are S1=1 (still busy with R1), S2=0 (finished R2 at t=2), S3=0 (finished R3 at t=3). Least connections sends R4 to S2 or S3 instead of S1, so it starts immediately instead of queuing. This is a strictly better outcome than round robin produced for the exact same traffic.

Algorithm 3: Weighted Round Robin

Least connections assumes all servers are equally powerful. That's often false — a data centre might have one newer server that can comfortably handle twice the traffic of two older ones sitting alongside it. Weighted round robin handles this by giving each server a weight proportional to its capacity, and cycling through servers so that each gets a share of requests matching its weight. If S1 has weight 2 and S2, S3 each have weight 1, the assignment pattern repeats as S1, S1, S2, S3 — S1 appears twice as often as S2 or S3 in every cycle of four:

Out of 8 requests, S1 receives 4 (half), while S2 and S3 receive 2 each (a quarter apiece) — exactly matching the 2:1:1 weight ratio. Weighted round robin and least connections are often combined in real systems into "weighted least connections," where the server chosen is the one with the lowest ratio of active connections to its weight, but that combination is beyond what we need to build here — the important idea is that the weight parameter lets an operator tell the load balancer "this machine can genuinely take on more."

Two Practical Details: Health Checks and Sticky Sessions

Two more pieces complete the picture of how real load balancers behave. First, a health check: the load balancer periodically pings each server (for instance, every few seconds) to confirm it is still responding correctly. If a server crashes or starts returning errors, the load balancer removes it from the pool immediately, so no new requests are sent to a dead machine — round robin or least connections both simply skip it until it's confirmed healthy again. Without health checks, a crashed server would still take its "turn" under round robin and every request routed to it would fail.

Second, session persistence (also called "sticky sessions"): some applications temporarily store information about a specific user in the memory of whichever server first handled them — for example, the contents of a shopping cart mid-checkout. If the very next request from that same user gets load-balanced to a different server that never saw their cart, the data appears to vanish. Load balancers can be configured to remember, for a given client, which server handled them last, and keep routing that client's requests to the same server. This is a deliberate exception carved out of pure round robin or least connections, made for a specific correctness reason, not a performance one.

Misconception: "Equal Requests Means Equal Load"

The single most common misunderstanding is treating round robin's guarantee — equal number of requests per server — as if it were a guarantee of equal work per server. The worked timeline earlier in this chapter is a direct counter-example: S1 received exactly the same count of requests as S2 and S3, yet ended up as a bottleneck because one of its requests happened to take five times longer than the others. This is worth internalizing precisely because round robin's simplicity makes it tempting to assume it's always "fair." It is fair only along one axis (request count), and real workloads — database queries of varying complexity, file uploads of varying size, video calls of varying length — routinely vary enormously along the axis that actually matters, which is processing time. That's exactly why least-connections-style algorithms, which react to real, current load rather than a fixed rotation, are preferred for workloads where requests are not roughly uniform in cost.

A second, smaller misconception worth naming: adding servers does not automatically make every part of a system faster in proportion. If a hundred web servers all read from the same single database, that database can become the new bottleneck no matter how many web servers you add in front of it — the load balancer only distributes the traffic reaching the layer it sits in front of; it does nothing for a shared resource like a single database that every one of those servers still depends on.

Visualizing the Flow

The diagram below shows nine numbered requests arriving at a load balancer running round robin, and exactly how they fan out across three servers — requests 1, 4, 7 always land on S1; 2, 5, 8 on S2; 3, 6, 9 on S3.

Where This Fits in the Networking Picture

Load balancers are usually described by which layer of network communication they operate at. A Layer 4 load balancer looks only at network-level information — source and destination IP address and port number — and routes based on that, without reading the actual content of the request; it is fast but cannot make decisions based on, say, the specific URL being requested. A Layer 7 load balancer reads the actual application-level request (for example, an HTTP request's path, such as /search versus /payment) and can route different types of requests to different specialised server pools — sending all payment requests to servers with stricter security, for instance. You are not expected to memorise the OSI model in depth for this chapter; the point is only that "which server should handle this?" and "how much information am I allowed to look at to decide?" are two separate design questions.

Worked Practice

Try these using the exact reasoning from this chapter before checking the worked answers.

1. Four servers S1–S4 are handled by a round robin balancer whose next_index currently sits at 2 (0-indexed, so the next request will go to S3). Requests 1 through 7 arrive. Which server handles request 5?
Worked answer: Starting index 2 means the assignment order is S3, S4, S1, S2, S3, S4, S1 for requests 1–7 (index cycles 2,3,0,1,2,3,0 — using (2+i-1) % 4 for request i). Request 5 corresponds to index (2+4) % 4 = 2, which is S3.

2. Under least connections, three servers currently have active counts S1=3, S2=1, S3=2. A new request arrives, then two requests on S2 finish, then another new request arrives. Where do the two new requests go?
Worked answer: First new request: minimum is S2 (1), so it goes to S2, making S2=2. Then two S2 requests finish: S2 = 2 − 2 = 0. Now counts are S1=3, S2=0, S3=2. Second new request: minimum is S2 (0), so it also goes to S2.

3. A weighted round robin setup has S1 with weight 3 and S2 with weight 1. Out of the next 12 requests, how many should S1 receive, and why can't you answer this using ordinary (unweighted) round robin logic?
Worked answer: The weight ratio is 3:1, so out of every 4 requests, S1 should get 3 and S2 should get 1 — over 12 requests (three full cycles of 4), S1 receives 9 and S2 receives 3. Ordinary round robin has no concept of weight; it would give both servers 6 each regardless of their actual capacity, overloading the weaker one or under-using the stronger one.

Summary

• A single server has a hard ceiling on how many requests it can process per second; horizontal scaling adds more servers to raise that ceiling, but only works if incoming requests are correctly distributed among them.
• A load balancer sits in front of a server pool, choosing exactly one server per incoming request — it is not a router (which doesn't weigh current load) and not a duplicator (which would waste work by processing every request everywhere).
• Round robin cycles through servers in fixed order using index = (index + 1) % number_of_servers. It guarantees an equal request count per server but not equal work, since request durations can vary widely.
• Least connections always routes to the server currently handling the fewest active requests, adapting to real load rather than following a fixed rotation.
• Weighted round robin lets more powerful servers receive a proportionally larger share of requests.
• Health checks remove unresponsive servers from rotation automatically; sticky sessions deliberately override normal balancing to keep one client's requests on one server when that server holds state the client needs.
• The core misconception to avoid: "equal number of requests" is not the same guarantee as "equal load" — and load balancing only helps at the layer it controls, not at shared resources like a single database sitting behind every server.

Think About It

Think about this: How would you explain load balancing: distributing request traffic to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.