A Network With Too Much Freedom
Here is a neural network in its simplest possible form: a single artificial neuron with no activation function, just a weighted sum. It takes two inputs, x1 and x2, multiplies each by a weight, adds a bias, and outputs a prediction:
y_hat = w1·x1 + w2·x2 + b
This neuron has exactly three learnable numbers: w1, w2, b. Now suppose we train it on data from exactly two students, using a continuous "performance index" (0–100 scale) as the target:
- Priya: x1 = 5 (hours of self-study logged that week), x2 = 0.50 (an unrelated number — the last two digits of her admit-card barcode, divided by 100), target y = 68.0
- Rohan: x1 = 5 (also studied 5 hours that week), x2 = 0.53 (his barcode-derived number), target y = 68.3
x2 is nonsense as a predictor — a barcode digit cannot cause an exam score — but with only two data points, the network cannot tell that apart from a real pattern. Watch what happens when we ask: which (w1, w2, b) make this neuron match both students exactly? We need:
5w1 + 0.50w2 + b = 68.0
5w1 + 0.53w2 + b = 68.3
Subtracting the first equation from the second eliminates w1 and b in one stroke: 0.03·w2 = 0.3, so w2 = 10. Substituting back, b = 68.0 − 5w1 − 5 = 63 − 5w1. Notice w1 never got pinned down — it can be anything, and for every choice, setting b = 63 − 5w1 still fits both students perfectly. We have three unknowns but only two constraints, so one whole direction of weight-space is left completely free. This is the mathematical seed of overfitting: whenever a network has more adjustable parameters than the data can constrain, infinitely many weight settings achieve zero training error, and most of them are terrible.
Compare two members of this family. The modest choice w1 = 0 gives (w1, w2, b) = (0, 10, 63): y_hat = 10x2 + 63, which sensibly ignores the useless barcode feature's exact value and just leans on the (arbitrary) constant. Now try w1 = 100: then b = 63 − 500 = −437, giving (100, 10, −437). Check it: at Priya's point, 100(5) + 10(0.50) − 437 = 500 + 5 − 437 = 68.0. At Rohan's, 500 + 5.3 − 437 = 68.3. Both fit exactly — the training loss is zero for this solution too. But this network has learned that study hours matter enormously (weight 100) purely because both students happened to log the same 5 hours; the moment a new student's logged hours differ even slightly from exactly 5, the prediction goes wild. A 1% nudge in x1 — just 0.05 hours, about three minutes of rounding error in a logged timestamp — changes the prediction by 100 × 0.05 = 5.0 points on the index, more than sixteen times the entire 0.3-point gap the model was trained to explain between Priya and Rohan. That is a network that memorized two coordinates rather than learning anything transferable, and it is invisible if you only check training error, because training error is zero either way.
Why This Happens: Capacity Versus Data
Generalize the pattern above: a model's capacity is (loosely) how many independent ways it can adjust itself to fit data — here, three weights. When capacity exceeds the number of independent constraints the training data provides, the loss function stops having a single minimum and instead has a flat valley of equally-perfect solutions, as we just derived algebraically. Gradient descent will happily settle anywhere on that valley floor, and nothing in the plain training objective prefers the small, boring, generalizable solution over the large, erratic, memorized one — both score exactly zero training loss. Real neural networks with thousands or millions of weights face this same valley, just in far higher dimensions, and typically with noisy rather than exactly-fit data, which makes the symptom show up as low training error alongside high validation error rather than a literal flat valley. The fix has to change what "success" means to the optimizer: instead of rewarding zero training loss alone, we reward zero training loss achieved with small weights. That reward change is exactly what regularization is.
L2 Regularization (Ridge / Weight Decay): The Derivation
Take the mean squared error over m training examples, J(w, b) = (1/2m) · Σi=1..m (y_hat(i) − y(i))², and add a penalty proportional to the squared size of the weights (by convention, the bias b is excluded — penalizing it would just discourage the model from centering its predictions correctly, which has nothing to do with overfitting):
J_reg(w, b) = J(w, b) + (λ/2m) · Σj w_j²
λ (lambda) is a number you choose before training: how much you want to punish large weights. To use gradient descent on J_reg we need ∂J_reg/∂w_j — the rate J_reg changes as w_j alone moves, holding every other weight fixed. The first term contributes whatever the ordinary data gradient ∂J/∂w_j is. For the second term, treat every w_k with k ≠ j as a constant; only the w_j² piece survives differentiation, and d/dw_j[(λ/2m)w_j²] = (λ/2m)·2w_j = (λ/m)w_j. So:
∂J_reg/∂w_j = ∂J/∂w_j + (λ/m)w_j
Plug this into the ordinary gradient descent rule w_j := w_j − η·∂J_reg/∂w_j (η is the learning rate):
w_j := w_j − η(∂J/∂w_j + (λ/m)w_j) = w_j·(1 − ηλ/m) − η·∂J/∂w_j
Read this rearranged form carefully: every single step first multiplies w_j by a shrink factor (1 − ηλ/m), strictly less than 1, and only then subtracts the usual data-driven gradient step. That shrink happens whether or not the data gradient has anything to say, which is why this technique is called weight decay — the weight decays toward zero on every update, and only the pull of real signal in the data can counteract that decay and keep a weight large.
This is precisely the missing force from the two-student example. Recall that once the network reaches the valley floor (zero training loss), ∂J/∂w_j = 0 everywhere along that valley — plain gradient descent has nothing left to do and simply stops wherever it happens to land, which could be the reckless w1 = 100 solution. L2 regularization adds a restoring force, (λ/m)w_j, that keeps pushing even after the data gradient vanishes, along exactly the flat direction we found (w1 free, w2 = 10, b = 63 − 5w1). It steers the network toward the smallest-norm member of that family. Check the sizes: the modest solution has w1² + w2² = 0² + 10² = 100; the reckless one has 100² + 10² = 10100 — 101 times larger. The penalty term (λ/2m)Σw_j² with λ = 1, m = 2 would cost the modest solution just 25, versus 2525 for the reckless one. Regularized gradient descent will always prefer the cheap one.
Let's hand-trace one update step to see the decay factor in action. Picture a checkpoint partway through training on this same two-student dataset (so m = 2, matching the derivation above), where the weights currently sit at w1 = −20, w2 = 30, and — because this checkpoint happens to already be on the zero-loss valley — the raw data gradient is momentarily [0, 0]. Use λ = 1, η = 0.1:
import numpy as np
w = np.array([-20.0, 30.0]) # weights at this checkpoint
data_grad = np.array([0.0, 0.0]) # dJ/dw is zero here (on the valley floor)
lam = 1.0 # lambda: regularization strength
eta = 0.1 # learning rate
m = 2 # training set size: Priya, Rohan
reg_grad = (lam / m) * w # (lambda/m) * w -> 0.5 * w
w = w - eta * (data_grad + reg_grad)
print(w)
Trace it by hand exactly as the code does. reg_grad = 0.5 × [−20, 30] = [−10, 15]. total gradient = [0,0] + [−10, 15] = [−10, 15]. Then w − 0.1×[−10, 15] = [−20 − (−1), 30 − 1.5] = [−19.0, 28.5]. You can reach the same numbers via the decay-factor formula: the shrink factor here is 1 − ηλ/m = 1 − (0.1×1)/2 = 0.95, so each weight is simply multiplied by 0.95 (since the data gradient contributes nothing this step): −20 × 0.95 = −19.0, and 30 × 0.95 = 28.5. Both routes agree: print(w) reports [−19.0, 28.5]. Each weight lost exactly 5% of its value this step — a 5% "tax" the optimizer pays every iteration, purely for carrying large weights, regardless of what the data says.
L1 Regularization: Pushing Weights to Exactly Zero
L1 regularization penalizes the absolute value of the weights instead of their square:
J_reg(w, b) = J(w, b) + (λ/m) · Σj |w_j|
Differentiate |w_j|: for w_j > 0 the slope is +1; for w_j < 0 the slope is −1; that is, d|w_j|/dw_j = sign(w_j). (At exactly w_j = 0 the function has a sharp corner and no true derivative exists — implementations use a subgradient, most simply just defining the gradient as 0 there.) The update rule becomes:
w_j := w_j − η·∂J/∂w_j − η(λ/m)·sign(w_j)
This is a completely different kind of push than L2's. L2's penalty gradient is (λ/m)w_j — proportional to the weight, so as w_j shrinks toward zero, the push shrinks too, and it asymptotically approaches zero without ever quite reaching it (like the 5% haircut above: 5% of a small number is an even smaller number). L1's penalty gradient is (λ/m)·sign(w_j) — a constant-size push toward zero no matter how small w_j already is. That constant push can fully cancel a small residual data gradient and drive the weight to land exactly on zero and stay there (once at zero, the subgradient convention typically keeps it there unless the data gradient is large enough to overcome the fixed penalty). This is why L1 regularization produces sparse networks — many weights become exactly zero, effectively deleting those input connections — while L2 produces small but nonzero weights almost everywhere.
Seeing the Difference: Constraint Regions
There is a classic geometric way to see why L1 favors exact zeros and L2 doesn't. Minimizing J(w) + penalty(w) is equivalent to minimizing J(w) alone while being constrained to stay inside a region whose shape depends on the penalty: for L2 that region is a circle (in two dimensions) or a sphere/hypersphere in general; for L1 it is a diamond (a rotated square) or, in higher dimensions, a shape with sharp corners and flat edges aligned with the axes. The elliptical rings below represent contours of equal training loss J(w), shrinking toward the unconstrained best fit. The regularized solution is the point where the smallest loss-contour that still touches the constraint region first makes contact.
On the left, the loss contours (orange ellipses) are pulled off-center from the origin, and because a circle is smooth everywhere, the point of first contact with a shrinking ellipse is almost never exactly on an axis — both w1 and w2 end up nonzero, just smaller than the unconstrained best fit. On the right, the diamond has corners sitting exactly on the axes. Because the loss contours are elongated and tilted, it is geometrically common for the first point of contact to land precisely on one of those corners — and a corner on the w1-axis means w2 = 0 exactly. That single geometric fact — smooth boundary versus cornered boundary — is the entire reason L1 performs automatic feature selection and L2 does not.
Dropout: Regularizing by Deleting Neurons
L1 and L2 constrain the weights directly. Dropout, introduced for deep neural networks specifically, takes a different route: during training, before each forward pass, every neuron in a layer is independently switched off with some fixed probability p (a common choice is p = 0.5 for hidden layers). A neuron that is switched off outputs exactly 0 for that pass, as if it did not exist; which neurons get switched off is re-randomized on every single training step.
Why does randomly deleting neurons prevent overfitting? A neuron cannot rely on any one specific other neuron always being present to fix its mistakes or carry a memorized detail, because that partner might be dropped on any given step. This forces every neuron to learn a feature that is useful somewhat independently of exactly which other neurons happen to be active — which pushes the network away from co-adapted, memorized combinations and toward more robust, redundant representations.
There's a subtlety: if you simply zero out a fraction p of neurons during training and then use all of them (with no zeroing) at test time, the total signal reaching the next layer would suddenly be larger at test time than what the network trained on, because at training time only a (1−p) fraction of units were contributing on average. The standard fix, called inverted dropout, rescales the surviving activations during training by 1/(1−p), so the expected total output matches what will happen at test time (when dropout is switched off entirely).
Concretely: suppose a layer's raw activations are a = [4, 6, 2, 8], with p = 0.5, and this step's random mask happens to be [1, 0, 1, 0] (neurons 2 and 4 dropped). After masking: [4, 0, 2, 0]. Inverted dropout then scales the survivors by 1/(1−0.5) = 2: the layer actually passes on [8, 0, 4, 0]. Check that this preserves the expectation for, say, the first neuron: across many training steps, it survives with probability (1−p) = 0.5, contributing 4×2 = 8, and is dropped with probability p = 0.5, contributing 0. Its expected output is 0.5(8) + 0.5(0) = 4 — exactly the original, undropped activation. That is precisely the property inverted dropout is designed to guarantee, which is why test-time inference can run the full, undropped network with no rescaling at all and still match what training was, on average, optimizing for.
Early Stopping
The most operationally simple regularizer needs no penalty term at all: split off a validation set the network never trains on, and after each epoch, record the loss on it alongside the training loss. In a network with real spare capacity, training loss falls throughout training, but validation loss typically falls for a while and then starts rising again — the point where the network has stopped learning general patterns and started fitting quirks specific to the training set. Early stopping means simply saving the weights from the epoch where validation loss was lowest, and discarding whatever further training produced. It costs nothing extra to compute (you needed a validation set for model selection anyway) and requires no new hyperparameter beyond deciding how many epochs of rising validation loss to tolerate before stopping (a "patience" setting), making it a natural default alongside, not instead of, L1, L2, or dropout.
Two Misconceptions Worth Killing
Misconception 1: "Regularization adds to the loss value, so it must be hurting the network." It is true that J_reg(w) ≥ J(w) for every w, since we're adding a nonnegative penalty. But the number you should judge a model by is never the training loss reported during optimization — it is performance on data the model never trained on. The entire two-student example showed both the modest and reckless weight settings scoring an identical, perfect J(w) = 0; the regularized objective was higher for the reckless one specifically because it generalizes worse, which is exactly the signal you want your optimizer to respond to. A regularized network reporting a higher number during training and a lower error at test time is regularization working exactly as designed, not a sign something is broken.
Misconception 2: "L1 and L2 do basically the same thing, just with a different exponent." The exponent difference looks cosmetic but produces qualitatively different behavior, as the constraint-region diagram makes precise: L2's penalty gradient is proportional to the weight itself, so it applies a gentle, ever-weakening pull that leaves nearly every weight small but nonzero; L1's penalty gradient is a constant magnitude regardless of the weight's size, strong enough relative to a small weight's data gradient to zero it out completely and keep it there. If your goal is simply to keep weights from growing huge, use L2. If your goal is to also discover which input features the network can discard entirely — automatic feature selection — L1 (or a weighted combination of both, called elastic net) is the appropriate tool, and no amount of tuning L2's λ alone will produce that exact-zero behavior, because L2's penalty gradient vanishes as w_j → 0, so it never supplies the constant push needed to cross zero and stay there.
Where This Shows Up in Your Exams
CBSE's Artificial Intelligence curriculum treats overfitting, underfitting, and the general idea of model evaluation as core conceptual territory; expect questions asking you to identify overfitting from a description of training-versus-validation behavior, and to name techniques (regularization, more data, simpler models, dropout, early stopping) that address it — this chapter's derivations go a level deeper than the board syllabus typically demands, but the extra rigor is exactly what separates a guessed answer from one you can justify. Neither JEE Main/Advanced nor BITSAT currently include neural networks as a named topic, so you won't see "derive the L2 weight-decay update rule" on those papers directly — but the underlying mathematics you just used (minimizing a function built from a quadratic term plus a penalty, taking partial derivatives, reasoning about a function's derivative failing to exist at a corner) is squarely Applications of Derivatives and Maxima-Minima territory from the JEE Mathematics syllabus, so working through this derivation by hand is genuine, transferable JEE practice even though the label "neural network" won't appear on the question paper. GATE's Data Science and Artificial Intelligence (DA) paper, introduced in 2024 as a new GATE discipline, does test bias-variance tradeoff and L1/L2 regularization directly and numerically, making this chapter close to GATE-DA syllabus as written. If you are preparing under the INSPIRE scheme's talent-search track (which absorbed KVPY's scholarship role after KVPY was discontinued in 2022), the statistics-and-reasoning components draw on exactly this kind of "does this pattern generalize, or did I just memorize two points" thinking.
Check Yourself
- In the two-student example, verify algebraically that (w1, w2, b) = (−10, 10, 113) also fits both Priya's and Rohan's data points exactly. (Hint: check it satisfies b = 63 − 5w1.)
- For that same (−10, 10, 113) solution, compute the L2 penalty contribution (λ/2m)(w1²+w2²) with λ=1, m=2, and compare it to the modest solution's penalty of 25. Which would gradient descent on J_reg prefer?
- Starting from w = [8.0, −4.0] with a data gradient of [0.5, −0.2], λ = 2, η = 0.1, and m = 4, compute one L2-regularized gradient descent update by hand, then separately for L1 regularization using the same numbers. Explain in one sentence why the two results differ in kind, not just in magnitude.
- A classmate says, "My validation accuracy is lower than my training accuracy, so my regularization is broken." Using the ideas from this chapter, explain what's actually wrong with that reasoning, and describe what pattern in training-vs-validation accuracy over epochs would actually indicate a real problem.
- Explain, using the constraint-region diagram, why increasing λ in L1 regularization tends to zero out weights one at a time as λ grows, rather than all at once.
Summary
Overfitting is not simply "the network is too smart" — it is the precise, derivable consequence of a model having more free parameters than the training data can constrain, leaving a whole family of equally-perfect-on-training-data solutions, most of which generalize badly. L2 regularization adds a (λ/2m)Σw_j² penalty whose gradient (λ/m)w_j produces a proportional "weight decay" shrink on every update, steering the optimizer toward the smallest-norm solution among the equally-good ones. L1 regularization's (λ/m)Σ|w_j| penalty instead applies a constant-size push, sign(w_j), which can drive weights to exactly zero and keep them there, producing sparse, automatically feature-selecting networks — a difference traceable directly to the smooth-versus-cornered geometry of their constraint regions. Dropout regularizes by randomly deleting neurons during training (rescaled via inverted dropout so expected activations match test time), preventing brittle co-adaptation between specific neurons. Early stopping regularizes for free by halting training at the epoch where validation loss, not training loss, is lowest. All four techniques share one goal: make the optimizer answer to more than just "did you fit the training data," because fitting the training data, as the two-student example proved with cold arithmetic, is a test that a badly memorizing network can pass just as easily as a genuinely learning one.
Think About It
Think about this: How would you explain regularization: preventing overfitting in neural networks to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.