Federated Learning: Collaborative ML Without Sharing Data

Picture a video-streaming app that wants to warn you before you exhaust your monthly data pack. To do that well, it needs a model that predicts "how many GB will this user consume this month, given how many hours per day they stream video?" A model like this gets better when it learns from many users, not just one. The obvious approach is to collect everyone's streaming logs on a central server and train on all of it together. But a streaming log is personal — it can reveal what shows someone watches, at what time of night, on what kind of connection. Most people, and increasingly the law in most countries including India's Digital Personal Data Protection Act, 2023, do not want that kind of behavioural log leaving their phone. So the app is stuck: it needs data from thousands of devices to train a good model, but it cannot ethically or legally pull that data into one place.

Federated learning is the technique that resolves exactly this conflict. Instead of moving data to a central model, it moves the model to the data, trains a little bit on each device, and only ever sends back the small set of numbers the model learned — never the raw examples that produced them. This chapter builds the idea from a first-principles worked example, gives you a working Python simulation you can trace by hand, and then digs into where the "no data ever leaves the device" story is more subtle than it first sounds.

The Core Problem: Good Models Need Lots of Data, But Data Has an Owner

In every machine learning setup you have studied so far — linear regression, classification, whatever the model — the standard recipe has three steps: gather a training set in one place, run a training algorithm (like gradient descent) over it, and ship the trained model. That recipe silently assumes something: that it is fine to copy everyone's raw data into one location, usually a company's server, before training even starts.

That assumption breaks down in a lot of real situations a programmer will actually run into:

• Regulation. Hospitals in different cities may each hold patient records, but health data is legally sensitive; moving it across hospital boundaries (sometimes even across state or national borders) can be restricted or require consent that is impractical to collect at scale.
• Scale and bandwidth. A predictive keyboard app runs on hundreds of millions of phones. Uploading every keystroke from every phone to a server, continuously, is a genuinely enormous, expensive data-transfer problem — even if privacy were not a concern.
• Ownership and trust. Three competing banks might each want a better fraud-detection model, and pooling transaction patterns across all three would make every one of their models stronger. But no bank wants to physically hand its customer transaction data to a rival or even to a neutral third party.

Federated learning was introduced by researchers at Google in 2016 to solve the second problem above — improving the next-word predictions in the Gboard keyboard app without uploading what people type. The idea generalises cleanly to the other two problems as well, which is why it is now used well beyond keyboards.

The Core Idea, Before Any Formulas

Here is the entire idea in one sentence, which we will spend the rest of this chapter unpacking carefully: a central server keeps a shared model and repeatedly (1) sends the current model out to a group of devices, (2) lets each device improve it a little using only its own local data, and (3) collects back and averages just the improved model numbers — not the data.

Think of it like a group of students each solving practice questions from their own private notebook, then reporting back only their final answer approach (say, a formula they refined) to a class WhatsApp group — not photographs of their notebook pages. The teacher (the server) combines everyone's refined formula into one improved class formula, sends that back out, and the cycle repeats. Nobody's notebook ever leaves their hands, yet the class formula keeps improving because it absorbs the lessons each student learned, even though it never sees the raw work that produced those lessons.

Setting Up a Model We Can Compute By Hand

To make this precise without needing calculus, we'll use the simplest possible learnable model: a single-weight linear predictor predicted_y = w × x. In our streaming-data scenario, x is hours streamed per day and y is GB used that month; w is the one number the model needs to learn, and it is shared by the whole system as the "global model."

To keep the arithmetic traceable by hand, imagine three phones, each holding exactly one private usage record it has logged so far (in reality a phone would hold thousands of records — we use one per phone purely so every step of the arithmetic can be checked):

• Phone 1 (Mumbai): 2 hours/day streaming → 3 GB used — (x=2, y=3)
• Phone 2 (Chennai): 3 hours/day streaming → 6 GB used — (x=3, y=6)
• Phone 3 (Guwahati): 1 hour/day streaming → 1.5 GB used — (x=1, y=1.5)

Notice these three points don't lie on one perfect line (3/2 = 1.5, but 6/3 = 2, and 1.5/1 = 1.5) — so there is no single weight w that fits everyone exactly. That is realistic and important: federated learning has to find a reasonable compromise weight, exactly like ordinary machine learning does when data is noisy.

The Local Update Rule

Each device improves the model using a rule you may already recognise as a gentle form of gradient descent, sometimes called the delta rule: nudge the weight in the direction that would have reduced today's prediction error, scaled by a small step size called the learning rate (call it lr):

new_w = old_w + lr × x × (actual_y - predicted_y)

Read this the way you would read any assignment statement: if the model under-predicted (actual_y bigger than predicted_y), the adjustment is positive, so the weight grows a bit — the model learns to predict a larger value next time for that kind of input. If it over-predicted, the weight shrinks. The size of x also matters: an input that was larger gets more "blame" (or credit) for the error, because it contributed more to the prediction in the first place.

Round 1: Every Device Trains Locally

The server initialises the global weight arbitrarily, say w = 1.0, and broadcasts it to all three phones. Using a learning rate of lr = 0.1, each phone runs the update rule once on its own single data point:

Phone 1: predicted = 1.0 × 2 = 2
         error = 3 - 2 = 1
         new_w = 1.0 + 0.1 × 2 × 1 = 1.0 + 0.2 = 1.2000

Phone 2: predicted = 1.0 × 3 = 3
         error = 6 - 3 = 3
         new_w = 1.0 + 0.1 × 3 × 3 = 1.0 + 0.9 = 1.9000

Phone 3: predicted = 1.0 × 1 = 1
         error = 1.5 - 1 = 0.5
         new_w = 1.0 + 0.1 × 1 × 0.5 = 1.0 + 0.05 = 1.0500

Notice something instructive: Phone 2's local weight moved the most (1.0 → 1.9), because it had both a bigger input x and a bigger error. Each phone, left on its own, would drift toward a weight that fits only its single household — a real weakness of learning purely from one device's tiny, biased slice of the world. This is exactly why the numbers now get combined.

The server never sees (2, 3), (3, 6), or (1, 1.5). It only receives three numbers: 1.2000, 1.9000, 1.0500. It combines them with a simple average — this averaging step is called FedAvg (Federated Averaging), and it is the algorithm that gives federated learning its name:

new_global_w = (1.2000 + 1.9000 + 1.0500) / 3
             = 4.1500 / 3
             = 1.3833

One round of federated learning is now complete. The global weight moved from 1.0000 to 1.3833 — and it moved in a way that reflects a compromise across all three phones' data, even though the server only ever handled three small floating-point numbers, never a single usage record.

Round 2: The Loop Repeats

The server broadcasts the new global weight, 1.3833, back out. Each phone repeats the same local update, now starting from this improved value:

Phone 1: predicted = 1.3833 × 2 = 2.7667
         error = 3 - 2.7667 = 0.2333
         new_w = 1.3833 + 0.1 × 2 × 0.2333 = 1.3833 + 0.0467 = 1.4300

Phone 2: predicted = 1.3833 × 3 = 4.1500
         error = 6 - 4.1500 = 1.8500
         new_w = 1.3833 + 0.1 × 3 × 1.8500 = 1.3833 + 0.5550 = 1.9383

Phone 3: predicted = 1.3833 × 1 = 1.3833
         error = 1.5 - 1.3833 = 0.1167
         new_w = 1.3833 + 0.1 × 1 × 0.1167 = 1.3833 + 0.0117 = 1.3950

new_global_w = (1.4300 + 1.9383 + 1.3950) / 3
             = 4.7633 / 3
             = 1.5878

Each device's error is shrinking round over round (Phone 2's error, for instance, dropped from 3 to 1.85 as the shared weight moved closer to what everyone needs). If you keep this loop running, the global weight keeps drifting toward whatever single value best balances all three phones' data — the same destination ordinary gradient descent would reach if it were secretly allowed to see the pooled dataset directly, but reached here without the server ever touching that raw data.

Writing the Loop as an Algorithm

Everything above is just two nested loops: an outer loop over communication rounds, and an inner loop over clients (devices). That structure translates directly into code:

def local_update(w, x, y, lr=0.1):
    predicted = w * x
    error = y - predicted
    return w + lr * x * error

def federated_round(global_w, clients, lr=0.1):
    local_weights = [local_update(global_w, x, y, lr) for x, y in clients]
    new_global_w = sum(local_weights) / len(local_weights)
    return new_global_w, local_weights

clients = [(2, 3), (3, 6), (1, 1.5)]   # Phone 1, Phone 2, Phone 3
w = 1.0

for round_num in range(1, 3):
    w, locals_ = federated_round(w, clients)
    rounded = [round(v, 4) for v in locals_]
    print("Round", round_num, "locals =", rounded, "global =", round(w, 4))

Tracing this by hand: on the first pass through the loop, federated_round is called with global_w = 1.0; the list comprehension computes local_update(1.0, 2, 3, 0.1), local_update(1.0, 3, 6, 0.1), and local_update(1.0, 1, 1.5, 0.1) in turn, giving exactly the three numbers we computed above, 1.2, 1.9, 1.05; their average, 1.3833, becomes the new w. The printed output for the two rounds is:

Round 1 locals = [1.2, 1.9, 1.05] global = 1.3833
Round 2 locals = [1.43, 1.9383, 1.395] global = 1.5878

which matches our hand computation exactly. Notice what the function signature tells a programmer about the privacy property: local_update is the only function that ever touches x and y; federated_round, which is where the server-side logic lives, only ever handles the list local_weights — three plain numbers. If you were deploying this for real, local_update would run separately on each phone's own hardware, and only its single returned number would travel over the network. The server's code, structurally, is never even given a variable that could hold raw user data.

Weighting Clients Fairly When Data Amounts Differ

Our example gave each phone exactly one data point, so a plain average was reasonable. Real devices hold very different amounts of local data — a heavy streaming user might have thousands of logged sessions, a light user just a handful. If you average local weights with equal weight regardless of dataset size, a device with almost no data can drag the global model just as hard as a device with a huge, reliable dataset. The fix, and the actual definition McMahan's FedAvg paper uses, is to weight each client's contribution by how much local data it trained on:

def federated_average(weights, sizes):
    total = sum(sizes)
    return sum(w * n for w, n in zip(weights, sizes)) / total

Suppose, after some round, the three phones' local weights were 2.0, 3.5, and 1.5, but Phone 1 had trained on 50 logged sessions, Phone 2 on only 10, and Phone 3 on 40 (100 sessions total). A plain average would give (2.0+3.5+1.5)/3 = 2.3333. The size-aware FedAvg gives:

(50 × 2.0 + 10 × 3.5 + 40 × 1.5) / 100
= (100 + 35 + 60) / 100
= 195 / 100
= 1.9500

The weighted result, 1.95, sits closer to Phones 1 and 3 (who together supplied 90 of the 100 total sessions) than the plain average does — which is the statistically correct behaviour: a device's opinion about the best weight should count roughly in proportion to how much evidence it actually has.

How One Round Looks as a Diagram

Federated SGD vs. Federated Averaging

Our simulation ran exactly one gradient step per device per round — this simplest version is technically called Federated SGD (FedSGD). The algorithm McMahan's team actually proposed and named FedAvg lets each device run several local training passes (several "epochs" over its local data) before reporting back, not just one step. That matters for a very programming-flavoured reason: every round requires a network round-trip, and network round-trips are slow and expensive at the scale of millions of phones. By doing more local computation per round, FedAvg needs far fewer rounds to converge than FedSGD, trading cheap on-device computation for expensive network communication — a classic systems trade-off. The averaging step itself, and the privacy property, work identically either way; only how much local work happens between rounds changes.

Common Misconception: "No Raw Data Sent" Means "Perfectly Private"

It is tempting to conclude that because Phone 1's actual record (2, 3) never left the phone, an attacker who intercepts the network traffic, or a curious engineer looking at server logs, learns literally nothing about that user. This is false, and it is worth understanding exactly why, because it is the single most important nuance separating a Grade-9 summary of federated learning from a correct one.

The number a device sends back, w_local, is not random noise — it is a specific mathematical function of that device's private data. Look again at the update rule: new_w = old_w + lr × x × (y - old_w × x). If an attacker knows the starting weight, the learning rate, and receives the returned new_w, they have one equation relating the two unknowns x and y. With a single-example device like ours, that is not quite enough to pin down both numbers exactly, but it already narrows them down to a relationship between them; with more realistic models and repeated observations across rounds, researchers have shown it is often possible to substantially reconstruct or approximate the original training examples from the sequence of updates alone. This class of attack is called a gradient inversion or model inversion attack, and it is an active area of security research precisely because "we only send weights, not data" is not, by itself, a rigorous privacy guarantee.

Real federated learning deployments therefore add extra layers on top of the basic protocol described in this chapter: secure aggregation, a cryptographic protocol where the server is mathematically only able to compute the sum of all devices' updates in a round, never see any individual device's update in isolation; and differential privacy, where each device adds a small amount of carefully calibrated random noise to its update before sending it, giving a provable mathematical bound on how much any single device's data could have influenced the final result. The honest way to describe federated learning is: it removes the need to centralise raw data and shrinks the attack surface enormously, but it is a foundation that additional privacy techniques are layered on top of — not a magic guarantee on its own.

A Second Misconception Worth Clearing Up

Some students assume federated learning is a different kind of model — as if "federated" describes a special new type of neural network the way "convolutional" or "recurrent" does. It does not. Federated learning is a training protocol, a way of organising where computation happens and what gets communicated. You could train a one-weight linear model federatedly, as we just did, or a deep image-recognition network federatedly — the model architecture itself is unchanged; what changes is the loop that decides who computes gradients, on what data, and what gets sent over the network to combine them.

Why It's Genuinely Hard in Practice

Two challenges make federated learning noticeably harder to engineer than ordinary centralised training, and both follow directly from the fact that devices, not a data centre, are doing the work:

• Non-IID data. "IID" means independent and identically distributed — the textbook assumption that every data source looks statistically similar. Real phones violate this badly: a college student in Bengaluru and a retired person in a small town in Bihar have wildly different typing vocabularies and streaming habits. Averaging updates from very different populations can pull the global model toward a compromise that fits nobody particularly well, and researchers have developed variants of FedAvg specifically to handle this.
• Unreliable, uneven participation. A data centre's machines are always on and roughly equally fast. Phones drop off Wi-Fi, run out of battery, or are simply switched off, and older devices compute local updates far more slowly than new ones. Real systems must handle rounds where only a random, changing subset of devices actually respond in time, and must not let the model be dominated by whichever devices happen to be fastest or most consistently online.

Where This Runs Today

Federated learning is not a laboratory curiosity. Google's Gboard keyboard uses it to improve next-word and emoji predictions from what people type, without uploading keystrokes. Apple has published on using federated learning together with differential privacy to improve on-device features like predictive text and voice recognition, again without centralising raw voice or typing data. The same architecture is actively researched for exactly the hospital scenario this chapter opened with — letting multiple hospitals jointly improve a diagnostic model from medical images or records that legally, and ethically, cannot be pooled into one database.

Test Yourself: Trace the Algorithm

1. Three devices report local weights 2.0, 3.5, and 1.5 after a round, each having trained on an equal amount of local data. Compute the plain FedAvg global weight.
2. Same three local weights, but now Phone 1 trained on 50 examples, Phone 2 on 10, and Phone 3 on 40 (100 total). Compute the data-size-weighted FedAvg global weight, and explain in one sentence why it differs from your answer to Question 1.
3. Using the local_update function from this chapter, with lr = 0.2 instead of 0.1, and starting from w = 1.0, compute Phone 1's local weight after one round on its data point (x=2, y=3). Show your arithmetic.
4. A hospital administrator claims: "We use federated learning, so patient data is 100% anonymous and can never be reverse-engineered." Explain precisely what is wrong with this claim, and name one specific technique a real deployment would add to make the privacy guarantee rigorous.
5. In one sentence, explain the practical trade-off between FedSGD (one local step per round) and FedAvg (several local steps per round), in terms of what gets used more and what gets used less.

Answers
1. (2.0 + 3.5 + 1.5) / 3 = 7.0 / 3 = 2.3333.
2. (50×2.0 + 10×3.5 + 40×1.5) / 100 = (100 + 35 + 60) / 100 = 195 / 100 = 1.9500. It differs because the weighted version gives more influence to Phones 1 and 3, which together supplied 90% of the total training data, instead of treating all three phones as equally informative.
3. predicted = 1.0 × 2 = 2; error = 3 - 2 = 1; new_w = 1.0 + 0.2 × 2 × 1 = 1.0 + 0.4 = 1.4000.
4. Sending only weight updates instead of raw data reduces the attack surface but does not, by itself, guarantee privacy: an update is a specific mathematical function of the local data (through the update rule), so techniques like gradient inversion can partly reconstruct training examples from a sequence of updates. A rigorous deployment would add secure aggregation (server only ever sees the sum of all updates in a round, never any single device's update) and/or differential privacy (calibrated random noise added to each update with a provable bound on information leakage).
5. FedAvg trades more on-device computation (several local training passes per round) for fewer, cheaper network round-trips, compared to FedSGD, which does minimal computation per round but needs many more rounds, and therefore many more network trips, to reach the same result.

Summary

Federated learning solves a real conflict every large-scale ML system eventually hits: models improve with more data, but data often cannot or should not be centralised. The mechanism is a loop — broadcast the current global model, let each device improve it locally using an ordinary update rule like the one you traced by hand, then combine only the returned weights, usually via a data-size-weighted average called FedAvg, back into a new global model. This is a training protocol, not a different model type, and it can wrap around any learnable model, from the single-weight predictor in this chapter to full neural networks in production keyboards. Its privacy benefit is real but not absolute: sending weights instead of raw data shrinks what can leak, but weights are still a function of the data that produced them, which is why serious deployments add secure aggregation and differential privacy rather than relying on "no raw data sent" as a complete guarantee. The two genuinely hard engineering problems — devices whose data distributions differ wildly from each other (non-IID data), and devices that drop out or lag unpredictably — are what most current research in this area is actually about.