Load Balancing: Distributing Traffic

Every year on CBSE results day, lakhs of Class 10 and Class 12 students open the results portal within minutes of the announcement. Imagine the portal runs on a single server that can comfortably handle 500 requests per second. At 11:00 AM the announcement goes out on the news, and "Check My Result" starts receiving 4,500 new requests every second, steadily, second after second. Multiply that out and by the fourth second, 4,500 × 4 = 18,000 requests have arrived in total — while the server, working flat out at its fixed 500-per-second ceiling, has only managed to clear 500 × 4 = 2,000 of them. That leaves roughly 16,000 requests still waiting in an ever-growing queue, and the gap widens every second that follows, because the arrival rate never slows down while the server's processing speed stays fixed. The server does not crash from any single request. It drowns because requests keep arriving faster than one machine can ever clear them, no matter how efficient its code is.

The fix an engineer would reach for is not "buy a server that is 40 times faster" — no single machine scales that cleanly, and even if it did, it would sit mostly idle for 364 days a year, waiting for the next results day. The real fix is to run several ordinary servers side by side — say, 10 servers, each still capable of only 500 requests per second — and put one component in front of them whose only job is to decide, for every incoming request, which of the 10 servers should handle it. Ten servers at 500 requests/second each give a combined capacity of 5,000 requests/second, comfortably above the 4,500/second surge. That decision-making component is called a load balancer, and the rules it uses to make that decision are what this chapter is about.

What exactly does a load balancer do?

Picture the ticket counters at a large railway reservation office. If there is only one counter, the queue behind it grows without bound the moment more people arrive per minute than the clerk can serve. Open five counters, and a queue of the same size drains five times faster — but only if newcomers are actually spread across all five counters. If everyone keeps walking to counter 1 out of habit while counters 2 to 5 stand empty, opening those extra counters achieved nothing. Somebody — or something — has to actively direct each new arrival to a counter, ideally one that is not already backed up.

A load balancer is exactly that directing mechanism, sitting in front of a group of identical backend servers. Every request from a student's browser reaches the load balancer first. The load balancer never processes the actual request itself — it does not check anyone's result — it only decides which backend server will do that work, then forwards the request there. From the outside, students only ever see one address (like results.cbse.gov.in); internally, that one address is backed by many machines whose existence is invisible to the student. The set of rules the load balancer uses to pick a server each time is called a load balancing algorithm, and different algorithms make this choice in genuinely different ways, each with different strengths. We will build up four of them, each one fixing a specific weakness in the one before it.

Algorithm 1: Round Robin — the simplest possible rule

The most direct rule a load balancer can follow is: keep a numbered list of servers, and hand out requests to them in strict rotation — server 1, then server 2, then server 3, then back to server 1, forever. This is called round robin, and it needs almost no memory: just one number, the index of whichever server goes next.

servers = ["Server-A", "Server-B", "Server-C"]
next_index = 0

def get_server():
    global next_index
    chosen = servers[next_index % len(servers)]
    next_index += 1
    return chosen

for request_number in range(1, 8):
    print(request_number, get_server())

Trace this by hand, since that is the only way to trust code instead of guessing at it. next_index starts at 0. Request 1 computes 0 % 3 = 0, so it picks servers[0], which is Server-A, and next_index becomes 1. Request 2 computes 1 % 3 = 1 → Server-B, index becomes 2. Request 3: 2 % 3 = 2 → Server-C, index becomes 3. Request 4: 3 % 3 = 0 → Server-A again, index becomes 4. Continuing this pattern for all seven requests gives the sequence Server-A, Server-B, Server-C, Server-A, Server-B, Server-C, Server-A. Notice the % (modulo) operator is doing all the real work here — it wraps the ever-increasing counter back into the range 0, 1, 2 every time it would otherwise run off the end of the list, which is precisely the "back to counter 1" behaviour we wanted.

Round robin's hidden assumption is that every server is equally powerful and every request takes roughly the same amount of work to answer. For a results-checking system, where every request is "look up one roll number and return one PDF," that assumption is close enough to true, and round robin performs very well. It starts to break down the moment either assumption fails — which is exactly the problem the next algorithm solves.

Algorithm 2: Weighted Round Robin — when servers are not equal

Suppose the results-portal team upgrades their infrastructure gradually rather than all at once. They now have three servers: Server-A is a brand-new machine rated at three times the request capacity of the old ones, Server-B is a mid-range machine at twice the old capacity, and Server-C is the original modest machine. Plain round robin would send Server-C the exact same one-third share of traffic as the far more powerful Server-A — wasting Server-A's extra capacity while overloading Server-C. The fix is to give each server a weight proportional to its capacity: Server-A gets weight 3, Server-B gets weight 2, Server-C gets weight 1, and over any block of six requests, Server-A should receive three, Server-B should receive two, and Server-C should receive one.

The first idea most people reach for is to write out each server as many times as its weight to build an expanded list, then cycle through that expanded list exactly like plain round robin:

weights = {"Server-A": 3, "Server-B": 2, "Server-C": 1}

expanded = []
for server, weight in weights.items():
    expanded.extend([server] * weight)
# expanded is ["Server-A", "Server-A", "Server-A", "Server-B", "Server-B", "Server-C"]

for i in range(6):
    print(expanded[i % len(expanded)])

Run this and the six requests go, in order: Server-A, Server-A, Server-A, Server-B, Server-B, Server-C. The long-run ratio really is 3:2:1 as required — but look at when each server is picked, not just how often. Server-A is chosen three times in an unbroken row before anyone else gets a turn, then Server-B twice in a row, then Server-C once. This is called clustering: correct on average, but bursty in the moment. If each request takes, say, 20 milliseconds to answer, Server-A is hit with three requests back to back and must process all three before Server-B or Server-C see anything at all — a small traffic jam lands on Server-A every single cycle, even though its long-term share of the work is exactly right.

Production load balancers such as NGINX solve this clustering problem with a different bookkeeping trick, often called smooth weighted round robin. Instead of pre-expanding a list, every server keeps a running score called its current weight, starting at 0. On every single request, the load balancer adds each server's fixed weight to its current weight, picks whichever server now has the highest current weight, and then subtracts the total of all weights from the winner's score before moving to the next request. A server that was just picked takes a big penalty and has to "earn back" its score over several rounds before it can win again, which is what breaks up the clustering.

weights = {"Server-A": 3, "Server-B": 2, "Server-C": 1}
total_weight = sum(weights.values())          # 6
current_weight = {server: 0 for server in weights}

def get_server_smooth():
    for server, weight in weights.items():
        current_weight[server] += weight
    best = max(current_weight, key=current_weight.get)
    current_weight[best] -= total_weight
    return best

for i in range(6):
    print(get_server_smooth())

Trace the first two picks by hand. Before request 1, every current weight is 0. Adding each server's fixed weight gives Server-A = 3, Server-B = 2, Server-C = 1; the highest is Server-A, so it is chosen, and then 6 (the total weight) is subtracted from it, leaving Server-A at 3 − 6 = −3. Before request 2, add the weights again: Server-A = −3 + 3 = 0, Server-B = 2 + 2 = 4, Server-C = 1 + 1 = 2. Now Server-B has the highest score and is chosen, dropping to 4 − 6 = −2. Running this all the way through six requests produces the sequence Server-A, Server-B, Server-A, Server-C, Server-B, Server-A — the exact output of the code above, checked by actually executing it. Compare the two sequences side by side:

Both sequences send Server-A exactly three requests, Server-B exactly two, and Server-C exactly one out of every six — the weighted ratio 3:2:1 is honoured perfectly by both methods, and that part was never in question. What changes is the spacing: the naive method piles all of Server-A's three requests together in one burst, while the smooth method spreads them out as A, _, A, _, _, A, never sending the same server two requests in a row. For a server whose job is CPU-heavy (like re-rendering a results PDF), spreading the load evenly over time — rather than in synchronized bursts — keeps response times steadier for every student, even though the total work handed to each server across a full cycle is identical either way.

Algorithm 3: Least Connections — when requests are not equal

Weighted round robin fixes unequal servers, but it still assumes every request takes about the same time to finish. That assumption fails for a results portal that also serves a "Download detailed marksheet with subject-wise analysis" feature alongside the simple "Check pass/fail status" lookup. The detailed marksheet takes far longer to generate. If round robin keeps sending new requests to a server that is still busy grinding through a slow marksheet request from two turns ago, that server's queue backs up while a server that got lucky with three quick lookups sits idle — even though both received the same number of requests overall.

The least connections algorithm sidesteps this by tracking, for every server, how many requests are currently being handled but not yet finished, and always routing the next request to whichever server has the fewest active connections right now — not whichever server is "next in line."

servers = ["Server-A", "Server-B", "Server-C"]
conn_count = {s: 0 for s in servers}

def get_server_least_conn():
    chosen = min(conn_count, key=conn_count.get)
    conn_count[chosen] += 1
    return chosen

for request_number in range(1, 6):
    print(request_number, get_server_least_conn())

Trace it for five arriving requests, assuming none of them has finished yet (a fair simplification for a short burst at the very start of results day, when everything is still queuing up). Before request 1, every count is 0, so min() returns the first server it encounters among the tied minimums — Server-A, since Python dictionaries preserve the order keys were inserted in, and Server-A was inserted first. Server-A's count becomes 1. Before request 2, counts are A=1, B=0, C=0; the minimum is 0, and the first server with that value is Server-B, so it is chosen and its count becomes 1. Before request 3, counts are A=1, B=1, C=0; Server-C is the unique minimum and is chosen, becoming 1. Before request 4, all three counts are tied at 1, so the first one in insertion order — Server-A — wins again, becoming 2. Before request 5, counts are A=2, B=1, C=1; the minimum is 1, and Server-B is the first server with that value, so it is chosen, becoming 2. The full sequence is Server-A, Server-B, Server-C, Server-A, Server-B, and the final connection counts are A=2, B=2, C=1 — exactly what running this code produces.

Least connections needs more bookkeeping than round robin (a live count per server, updated both when a request starts and when it finishes, which the simplified trace above deliberately ignores for clarity). In exchange, it reacts to real, current load rather than a fixed schedule, which is why it is the standard choice for backend services where request duration varies a lot — file uploads, database-heavy queries, or anything where "next in the rotation" and "actually free right now" can disagree.

Algorithm 4: Sticky Sessions — when the server needs to remember you

All three algorithms above silently assume that any server can answer any request equally well, with no memory of past requests needed. That is true for a stateless lookup like "fetch roll number 4471203's result." It breaks for anything that involves a multi-step process — for example, a school admin portal where a teacher logs in, and the server keeps her logged-in session data (her identity, what she's allowed to edit) in that server's own memory rather than in a shared database. If her first request lands on Server-A and her very next click gets routed to Server-B by round robin, Server-B has never heard of her session and will ask her to log in again.

The fix is sticky sessions (also called session affinity): once a particular user's first request is assigned to a server, every subsequent request from that same user is deliberately sent to that same server, bypassing whatever the normal load-balancing algorithm would have picked. A simple way to implement this is to compute a fixed number from something that identifies the user — their session ID or login token — and use that number to pick a server consistently:

def assign_server(student_roll_number, num_servers=3):
    return student_roll_number % num_servers

servers = ["Server-A", "Server-B", "Server-C"]
roll_number = 14
server_index = assign_server(roll_number, len(servers))
print(servers[server_index])

For roll number 14 with 3 servers, 14 % 3 = 2, so server_index is 2, and servers[2] is Server-C. Every future request carrying roll number 14 recomputes the same 14 % 3 = 2 and lands on Server-C again, every time, without the load balancer needing to remember anything about roll number 14 specifically — the formula itself is the memory.

A common misconception is that sticky sessions and even load distribution are opposites — that "sticky" automatically means "unbalanced." That is not quite right. Stickiness only decides where a returning user's requests go; it says nothing about how new users get assigned in the first place, which is still handled by round robin, weighted round robin, or least connections underneath. The real risk is narrower: if a small number of users generate unusually long-lived or unusually heavy sessions — say, a school administrator uploading thousands of student records while pinned to one server for two hours — that one server can end up doing more work than the others for the duration of those sessions, even though new users keep arriving at all three servers in a perfectly even ratio. Sticky sessions trade a small, session-shaped risk of imbalance for the ability to keep server-side memory at all — a trade that is unavoidable unless the session data is moved somewhere all servers can share, such as a common database, which is a design choice with its own costs.

Detecting a dead server: health checks

Every algorithm above assumes all the servers in the list are actually working. In practice, servers crash, run out of memory, or lose network connectivity — and continuing to send requests to a server that is silently dead just turns every request routed there into an error the student sees. Load balancers guard against this with health checks: at a fixed interval (commonly every few seconds), the load balancer sends a small test request — often nothing more than "are you alive?" — to every server on its list. A server that fails to answer correctly within a short timeout, or answers with an error, is marked unhealthy and is temporarily removed from the rotation used by whichever algorithm is active. None of the four algorithms above send it any traffic until a later health check finds it responding normally again, at which point it quietly rejoins the pool.

Health checks are what make every algorithm in this chapter safe to run unattended. Round robin, weighted round robin, least connections, and sticky sessions all describe how to choose among the servers currently considered healthy — health checking is the separate, ongoing process that decides which servers are allowed to be in that list at any given moment. A server that goes down at 11:03 AM on results day is simply skipped by the next health check cycle, its share of traffic silently absorbed by the remaining healthy servers, with no human needing to notice and intervene in the middle of the surge.

Choosing an algorithm

None of these four algorithms is universally "best" — each is the right tool for a specific mismatch. Use round robin when servers are identical and requests are roughly uniform in cost, because it is the simplest to reason about and needs no live state. Use weighted round robin when servers differ in capacity but requests are still roughly uniform, so that a more powerful machine earns a proportionally larger share without any request-level tracking. Use least connections when request cost itself varies unpredictably, since it is the only one of the four that reacts to how busy a server actually is right now rather than following a fixed schedule. Layer sticky sessions on top of any of the other three specifically when a server must remember something about a user between requests — and pair it with health checks in every real deployment, since a load balancer that keeps trusting a dead server defeats the entire purpose of having more than one server in the first place.

Check your understanding

1. A site has 4 identical servers under plain round robin. Requests 1 through 10 arrive one after another. Which server (numbered 1–4, with server 1 taking request 1) handles request 10? Work it out using the modulo operator, the way the traced code above does.
2. Three servers have weights 5, 3, and 2 (total 10). Under the naive "expand the list, then cycle" method, how many requests in a row would the weight-5 server receive before any other server gets a turn? Why does the smooth current-weight method avoid this?
3. A least-connections load balancer shows current counts Server-A=3, Server-B=1, Server-C=1 just before a new request arrives. Which server receives it, and why might this be a better decision than round robin would make at this exact moment?
4. Explain, in your own words, why sticky sessions do not automatically cause unbalanced load — and describe one specific situation where they genuinely would.
5. A health check fails for Server-B at 2:00:00 PM, and the load balancer's health-check interval is every 5 seconds. In the worst case, how long could Server-B keep receiving misdirected requests after it actually goes down, and why does that number depend on the interval you choose?

Summary

A load balancer sits in front of a group of backend servers and decides, for each incoming request, which server should handle it — this is essential once traffic exceeds what any single server can process, as on CBSE results day. Round robin rotates through servers in a fixed cycle using the modulo operator and works well when servers and requests are both roughly uniform. Weighted round robin assigns unequal servers a proportional share of traffic; the naive way to generate that share (expand each server by its weight, then cycle) produces the correct long-run ratio but clusters a powerful server's requests into an unbroken burst, while the smooth current-weight method — add each weight every round, pick the highest running score, then subtract the total from the winner — produces the same ratio spread out over time instead of bunched together. Least connections tracks live, in-progress request counts per server and routes to whichever server is least busy right now, which matters whenever requests take unpredictable amounts of time to finish. Sticky sessions pin a specific user's repeat requests to whichever server first handled them, so that server-side memory (like a login session) keeps working — at the cost of a narrow, session-shaped risk of imbalance, not a guarantee of one. Health checks run underneath all four algorithms, periodically testing whether each server is still alive and silently removing a failed one from rotation until it recovers, which is what lets a multi-server system survive a single server's failure without anyone needing to intervene by hand.