Active Learning: Smart Data Labeling

One Pathologist, One Hundred Thousand Photos

A farm-advisory app used across grape and tomato belts in Nashik, Maharashtra lets farmers photograph a leaf and get an instant disease diagnosis — early blight, powdery mildew, rust, or healthy. Behind that instant answer is a supervised learning model: a program that looks at a photo (the input) and predicts a category (the label). To learn this mapping, the model needs thousands of example photos that already have the correct label attached, the way a textbook needs worked examples before it can ask you to solve one yourself.

Here is the catch. Only a trained plant pathologist can look at a leaf photo and say, with confidence, "this is early blight, not just sun-scorch." A careful expert can label maybe 200 photos in a working day — examining each one, checking it against known symptoms, sometimes zooming in to compare lesion patterns. The app collected 100,000 farmer-submitted photos this growing season. At 200 labels a day, labeling all of them would take 500 working days — well over a year. By the time the model was ready, the season it was meant to help with would be long over.

This is the central problem this chapter solves: when labeling is slow and expensive, which photos should the expert spend their limited hours on? The answer is not "label whatever arrives first" or "label a random sample." It is a strategy called active learning — letting the model itself point at the examples that would teach it the most, and reserving the human expert's time for exactly those.

Why Random Sampling Wastes an Expert's Time

Suppose instead of active learning, the team picks 500 random photos out of the 100,000 and asks the pathologist to label those, then trains the model on just this random sample. This is called passive learning — the model passively receives whatever data collection happens to hand it, with no say in the matter.

The trouble is that most photos farmers submit are of ordinary, healthy leaves, or leaves with textbook-obvious disease symptoms — a fully yellowed, curling leaf with classic rust pustules is not a hard call for a model that has already seen fifty like it. A random sample of 500 photos will be dominated by these easy, unsurprising cases. The pathologist spends the afternoon confirming things the model would likely have gotten right anyway. Meanwhile, the photos that would actually teach the model something new — a leaf with early, faint lesions that could be either blight or ordinary sun damage, or a photo taken at an odd angle — are scattered randomly through the other 99,500 unlabeled photos and might not get picked at all.

Active learning flips this around. Instead of choosing which photos to label by chance, it uses the model's own current predictions to find the photos it is most confused about, and sends only those to the human expert. Every minute of the pathologist's time is then spent resolving a genuine point of confusion, not re-confirming the obvious.

Teaching a Model to Say "I'm Not Sure"

To find the photos the model is confused about, we first need the model to reveal how confident it is, not just what its final answer is. A classifier trained to distinguish "diseased" from "healthy" does not just output a yes/no verdict — during training it learns to output a probability, a number between 0 and 1 representing how strongly the evidence in the photo points to "diseased."

Consider two of the model's predictions:

• Photo A: probability of diseased = 0.93. The model is quite confident this leaf is diseased.
• Photo B: probability of diseased = 0.51. The model is basically flipping a coin — barely leaning toward "diseased" over "healthy."

Photo A does not need a human's attention: whichever way that 0.93 turns out, the model is unlikely to have been badly fooled, and even if it is occasionally wrong on confident cases, there is little for the pathologist to teach it by confirming what it already believes strongly. Photo B is exactly the opposite — a probability of 0.51 means the model has found almost no clear evidence either way, and a human's true label on this one photo will sharpen the model's understanding of the boundary between "diseased" and "healthy" far more than ten confidently-correct photos would.

This gives us a simple, numeric way to rank photos by how much they are worth labeling. Define the confidence gap of a prediction as how far its probability sits from the halfway point, 0.5:

confidence gap = |p - 0.5|

A small confidence gap means the model is near a coin flip — uncertain. A large confidence gap means the model is leaning strongly one way — confident. To find the most useful photos to label next, we sort unlabeled photos by confidence gap in ascending order and work down the list: smallest gap (most uncertain) first.

Let's work this out by hand for five photos the model has just scored:

• Photo101: p = 0.51 → |0.51 − 0.5| = 0.01
• Photo102: p = 0.93 → |0.93 − 0.5| = 0.43
• Photo103: p = 0.67 → |0.67 − 0.5| = 0.17
• Photo104: p = 0.50 → |0.50 − 0.5| = 0.00
• Photo105: p = 0.08 → |0.08 − 0.5| = 0.42

Sorting these five confidence gaps from smallest to largest gives the query order: Photo104 (0.00), Photo101 (0.01), Photo103 (0.17), Photo105 (0.42), Photo102 (0.43). Photo104 goes to the pathologist first — its probability sat exactly on the 50-50 line, meaning the model found no usable evidence at all in that image. Photo102, at the other end, is left for later or skipped entirely, because a probability of 0.93 already tells us the model is confident.

Now let's write this as a function, since in a real system there could be thousands of unlabeled photos to rank, not five:

def uncertainty_scores(predictions):
    scored = []
    for photo_id, p in predictions:
        gap = abs(p - 0.5)
        scored.append((photo_id, round(gap, 2)))
    scored.sort(key=lambda item: item[1])
    return scored

predictions = [
    ("Photo101", 0.51),
    ("Photo102", 0.93),
    ("Photo103", 0.67),
    ("Photo104", 0.50),
    ("Photo105", 0.08),
]

for photo_id, gap in uncertainty_scores(predictions):
    print(photo_id, gap)

Tracing this line by line: predictions is a list of five tuples, each pairing a photo's ID with the model's predicted probability. The for loop inside uncertainty_scores visits each tuple, computes abs(p - 0.5), and appends a new tuple of (photo_id, gap) to scored. The gap is rounded to 2 decimal places here on purpose — computers store numbers like 0.51 − 0.5 imprecisely in binary, so without rounding you would sometimes see a gap printed as something like 0.010000000000000009 instead of a clean 0.01. The call scored.sort(key=lambda item: item[1]) sorts the list of tuples using each tuple's second element (the gap) as the sort key, smallest first — this is exactly the "ascending by confidence gap" rule from the worked example above. Running this program prints:

Photo104 0.0
Photo101 0.01
Photo103 0.17
Photo105 0.42
Photo102 0.43

which matches the by-hand ranking exactly. The pathologist can now be handed this list and told: work from the top.

When There Are More Than Two Classes: Margin Sampling

The leaf-disease app does not just distinguish "diseased" from "healthy" — it chooses between several diseases: healthy, early blight, rust, and powdery mildew. With four possible classes, the model outputs four probabilities that add up to 1.0, not a single number, so "distance from 0.5" no longer makes sense. We need a different rule for measuring confusion when there are several classes to choose from.

The natural idea: look at the model's top two guesses. If the model's favourite class is far more likely than its second favourite, it has a clear opinion. If the top two are nearly tied, the model is genuinely torn between two diagnoses — exactly the situation a human expert should resolve. This is called margin sampling, and the margin is defined as:

margin = (probability of top-ranked class) - (probability of second-ranked class)

A small margin means the model can barely tell its top two candidates apart. As before, we sort ascending by margin: smallest margin (most confused) first.

Worked example — three photos, each scored across four classes [healthy, blight, rust, mildew]:

• PhotoA: [0.40, 0.38, 0.12, 0.10]. Top two probabilities: 0.40 and 0.38. Margin = 0.40 − 0.38 = 0.02.
• PhotoB: [0.85, 0.10, 0.03, 0.02]. Top two: 0.85 and 0.10. Margin = 0.85 − 0.10 = 0.75.
• PhotoC: [0.30, 0.29, 0.21, 0.20]. Top two: 0.30 and 0.29. Margin = 0.30 − 0.29 = 0.01.

Sorted ascending by margin: PhotoC (0.01), PhotoA (0.02), PhotoB (0.75). PhotoC is queried first — the model gave its top two classes almost identical probability (30% vs 29%), meaning it genuinely cannot decide between them. PhotoB, where one class dominates at 85%, is the last thing worth spending expert time on.

In code:

def margin_sampling(predictions):
    scored = []
    for photo_id, probs in predictions:
        top_two = sorted(probs, reverse=True)[:2]
        margin = round(top_two[0] - top_two[1], 2)
        scored.append((photo_id, margin))
    scored.sort(key=lambda item: item[1])
    return scored

predictions2 = [
    ("PhotoA", [0.40, 0.38, 0.12, 0.10]),
    ("PhotoB", [0.85, 0.10, 0.03, 0.02]),
    ("PhotoC", [0.30, 0.29, 0.21, 0.20]),
]

for photo_id, margin in margin_sampling(predictions2):
    print(photo_id, margin)

Tracing this: for each photo, sorted(probs, reverse=True)[:2] sorts that photo's four probabilities from largest to smallest and keeps only the first two — the top two candidates. Subtracting the second from the first gives the margin, which is rounded and stored alongside the photo ID. The final sort again orders the list by that margin, smallest first. Running the program prints:

PhotoC 0.01
PhotoA 0.02
PhotoB 0.75

which matches the hand-worked ranking. Notice that uncertainty_scores and margin_sampling are really the same idea in two different clothes: both compute a number that is small when the model is confused and large when it is confident, and both sort the unlabeled pool by that number so the most confusing examples rise to the top.

The Active Learning Loop

Uncertainty scoring and margin sampling are the engine inside a bigger, repeating process. Written out as a numbered sequence, active learning runs as a loop with six steps:

1. Seed. Start with a small labeled set — perhaps 300 photos the pathologist already labeled early in the project — and a much larger pool of unlabeled photos.
2. Train. Fit the model on whatever labeled set currently exists.
3. Score. Run the freshly trained model over every photo still in the unlabeled pool, computing a confidence gap or margin for each one.
4. Query. Pick the photo (or a small batch of photos) with the smallest confidence gap or margin — the ones the model is most confused about — and send just those to the human expert, called the oracle in active learning terminology. At this point the chosen photo is still sitting in the unlabeled pool; it has only been flagged for attention, not yet labeled.
5. Label and update. The oracle labels those photos. They move from the unlabeled pool into the labeled set, which is what actually shrinks the unlabeled pool and grows the labeled one.
6. Repeat. Go back to step 2 and retrain on the now slightly larger labeled set, continuing the cycle until a labeling budget runs out or the model's accuracy on a held-out test set stops improving.

Notice that it is step 5, Label and update — not step 4, Query — that actually removes a photo from the unlabeled pool. Step 4 only identifies which photo deserves the oracle's attention next; the photo is still unlabeled while it travels to the expert. It is only once the oracle has produced a true label in step 5 that the photo formally moves out of the unlabeled pool and into the labeled set, so it will not be scored or queried again in a later pass through the loop.

The diagram below traces this exact six-step cycle, including the loop-back arrow from step 6 to step 2 that keeps the process running.

A Common Misconception: Who Is Doing the Labeling?

The name "active learning" leads many students to assume it means the model labels the data itself, cutting the human out entirely. This is incorrect. Every single label in this loop still comes from the human oracle — the pathologist. What the model does is not labeling; it is choosing, using its own uncertainty, which unlabeled photo is most worth a human's time. The word "active" describes the model taking an active role in selecting training examples, in contrast to passive learning, where the model has no say and simply receives whatever labeled data it is handed. If you ever see a system where the model assigns its own labels to unlabeled data and trains on those guesses without human review, that is a different technique — sometimes called self-training or pseudo-labeling — and it is not active learning, because there is no oracle step at all.

A Deeper Catch: Uncertain Doesn't Always Mean Useful

There is a subtler trap worth naming explicitly. A small confidence gap or margin tells you the model is uncertain, but it does not automatically tell you why the model is uncertain. There are two very different reasons a probability can land near 0.5:

• The photo shows a genuine borderline case — early-stage symptoms that could plausibly be blight or could plausibly be ordinary environmental stress. This is exactly the kind of example active learning is designed to find, and a human label here meaningfully sharpens the boundary the model has learned.
• The photo is something the model has essentially never seen before — a leaf photographed at night with flash glare, or a crop the training data never covered. Here the model isn't torn between two well-understood possibilities; it simply has no reliable evidence to work with, and its probability near 0.5 reflects confusion, not a fine decision boundary.

Both situations produce the same small confidence gap, but only the first is the kind of example that improves the model efficiently. In a real deployment, teams often add a simple sanity check — flagging photos that look nothing like anything in the training set — before handing the "most uncertain" list to the oracle, so the expert's time goes toward real boundary cases rather than blurry or off-topic photos that happen to confuse the model for unrelated reasons.

Where This Shows Up Beyond Leaves

The same loop applies wherever labeling is the bottleneck rather than data collection. A bank or postal service building a handwritten-digit or handwritten-address reader could use active learning to route only the messiest, least legible scanned digits to a human reviewer, instead of asking staff to double-check every single digit a system already reads confidently. The mechanism is identical to the leaf-disease case: score every unlabeled example, rank by confidence gap or margin, send the top of that list to a human, fold the new labels back into training, and repeat.

Knowing When to Stop

The loop in the diagram above cannot run forever — it needs a stopping rule, decided before the project starts. The two most common ones are a labeling budget (for example, "the pathologist has 40 hours available this season; stop once that time is used") and an accuracy plateau (keep a small labeled test set aside that is never used for training, and stop once accuracy on that held-out set stops improving between rounds — a sign that additional labels are no longer teaching the model much). Choosing a stopping rule matters: without one, a team might keep querying the oracle long after each new label is adding almost nothing to model quality, defeating the entire purpose of using active learning to save expert time in the first place.

Test Yourself

Q1. Four leaf photos get these probabilities of "diseased" from the model: Photo201 = 0.22, Photo202 = 0.58, Photo203 = 0.49, Photo204 = 0.99. Compute the confidence gap for each and list the query order (most uncertain first).

Q2. A three-class model (healthy / rust / mildew) scores three photos as follows — PhotoX: [0.50, 0.45, 0.05], PhotoY: [0.70, 0.20, 0.10], PhotoZ: [0.34, 0.33, 0.33]. Using margin sampling, which photo should be sent to the oracle first, and why?

Q3. Using the Nashik leaf-labeling scenario, explain in one or two sentences why a random sample of 500 photos wastes more of the pathologist's time than an actively-queried batch of 500 photos.

Q4. A photo of a wilted spinach leaf — a crop the training data never included — gets a probability of 0.50 from the leaf-disease model. Does this automatically mean it is a good candidate to send to the oracle? Explain, referring to the two different causes of a small confidence gap.

Answers

A1. Gaps: Photo201 = |0.22 − 0.5| = 0.28; Photo202 = |0.58 − 0.5| = 0.08; Photo203 = |0.49 − 0.5| = 0.01; Photo204 = |0.99 − 0.5| = 0.49. Query order (smallest gap first): Photo203, Photo202, Photo201, Photo204.

A2. Margins: PhotoX = 0.50 − 0.45 = 0.05; PhotoY = 0.70 − 0.20 = 0.50; PhotoZ, sorted descending is [0.34, 0.33, 0.33], so margin = 0.34 − 0.33 = 0.01. PhotoZ has the smallest margin, so it is queried first — the model's top two classes are separated by only one percentage point, meaning it genuinely cannot decide between them.

A3. Most of the 100,000 submitted photos are of clearly healthy or textbook-obvious diseased leaves, which a random sample of 500 would mostly reproduce; the pathologist would spend the afternoon confirming cases the model likely already handles correctly, while the truly ambiguous, boundary-case photos that would actually sharpen the model stay scattered — and possibly unpicked — among the other 99,500 unlabeled photos.

A4. No, not automatically. A confidence gap of 0.00 could mean the model found a genuine borderline case worth a human's judgment, or it could mean the model has simply never encountered anything like this input (a crop outside its training data) and has no real basis for an opinion at all. Sending only "smallest gap" photos to the oracle without checking whether they resemble known training data risks wasting the expert's time on off-topic images rather than real decision-boundary leaves.

Summary

Active learning solves a very specific problem: labeling is expensive, but not all unlabeled examples are equally worth labeling. By scoring every unlabeled item with a confidence gap (two classes) or a margin (many classes), sorting from most to least confusing, and repeatedly sending only the top of that list to a human oracle through the six-step Seed → Train → Score → Query → Label and update → Repeat loop, a model can reach strong accuracy using a small fraction of the labels a random sample would have required — provided the team also filters out uncertainty caused by simply unfamiliar, off-distribution inputs, and stops the loop once a labeling budget or accuracy plateau says further labels are not worth the expert's time.

Think About It

Think about this: How would you explain active learning: smart data labeling to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where active learning: smart data labeling is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting active learning: smart data labeling to at least 3 other topics you have studied.