Regularization: L1 vs L2 and Sparsity

Suppose you are doing a Class 12 science-fair project. You survey 25 classmates and collect a spreadsheet with 40 columns for each of them — attendance percentage, average daily study hours, sleep hours, number of tuition classes, mother's and father's years of education, distance from home to school, number of siblings, phone screen-time, even things you added just because you had the data lying around: shoe size, the last digit of their Aadhaar-linked roll number, favourite cricket team encoded as a number. Your goal: predict their Class 12 board percentage from these 40 features using ordinary least squares (OLS) linear regression.

Here is the uncomfortable fact: with 40 features and only 25 students, plain least squares does not merely overfit — it cannot even find a unique answer. The normal equations that OLS solves are w = (XᵀX)⁻¹Xᵀy, where X is your 25×40 data matrix. The rank of XᵀX can never exceed the rank of X, and the rank of X can never exceed the smaller of its two dimensions — here, 25. So XᵀX, a 40×40 matrix, has rank at most 25, meaning it is singular: it has no inverse. There are infinitely many weight vectors w that fit your 25 training students with zero error, because you have more unknowns (40 weights) than equations (25 students) to pin them down. Least squares will happily hand you one of these — often one with enormous coefficients on features like "shoe size" — and it will predict brand-new students' marks terribly, because it fit noise, not signal.

This chapter is about the fix: regularization. We will derive, from first principles, exactly what L2 (Ridge) and L1 (Lasso) regularization do to the optimization problem, prove a genuinely surprising fact — that L1 regularization drives some weights to exactly zero while L2 never does — and see why that single difference in geometry is the reason Lasso is used for automatic feature selection and Ridge is used for stable shrinkage.

From Training Error to Generalization Error

Ordinary least squares chooses the weight vector w that minimizes the residual sum of squares on the training data: RSS(w) = Σᵢ(yᵢ − ŷᵢ)². This is the training error. What we actually care about is the generalization error — how well the model predicts on students it has never seen. When the number of features p is large relative to the number of samples n (or when features are highly correlated with each other, called multicollinearity), OLS can drive training error to zero or near-zero while generalization error explodes. The mechanism is precise: even when XᵀX is technically invertible (p ≤ n), if it is ill-conditioned — some of its eigenvalues are very close to zero — then (XᵀX)⁻¹ contains huge entries, and tiny amounts of noise in y get amplified into wild swings in the estimated weights. A model with huge, wildly-varying coefficients is a model that is extremely sensitive to the exact training sample it happened to see — the definition of high variance and overfitting.

Regularization repairs this by changing what we optimize. Instead of minimizing training error alone, we minimize training error plus a penalty on the size of the weights:

minimize over w:   Loss(w)  +  λ · penalty(w)

Here λ ≥ 0 (lambda) is a hyperparameter you choose, controlling how much you penalize large weights. Setting λ = 0 recovers plain OLS. As λ grows, the optimizer is forced to trade off fitting the data perfectly against keeping the weights small. The two most important choices of "penalty(w)" — the sum of squared weights (L2) and the sum of absolute weights (L1) — behave in surprisingly different ways, and we now derive exactly why, one algebra step at a time.

Ridge Regression (L2): Deriving the Shrinkage Formula

To see the mechanism with no clutter, work with a single feature x (no intercept, data mean-centered) predicting y, with n training points. Define the two sums S_xx = Σxᵢ² and S_xy = Σxᵢyᵢ. Plain OLS in this simplified 1-D case minimizes Σ(yᵢ − wxᵢ)², which expands to Σyᵢ² − 2w·S_xy + w²·S_xx. Setting the derivative to zero gives the familiar single-feature OLS solution w_OLS = S_xy / S_xx.

Ridge regression adds an L2 penalty (λ/2)w² to the objective (the 1/2 is just a convenience that keeps the arithmetic clean — some textbooks and libraries drop it and absorb the factor of 2 into λ instead; it changes nothing about the qualitative behaviour). The full objective is:

L(w) = (1/2)Σ(yᵢ − wxᵢ)²  +  (λ/2)w²

Expand the squared-error term exactly as before and collect all w² terms together:

L(w) = (1/2)S_xx·w² − S_xy·w + (λ/2)w² + constant
     = (1/2)(S_xx + λ)·w² − S_xy·w + constant

Differentiate with respect to w and set the result to zero to find the minimum:

dL/dw = (S_xx + λ)·w − S_xy = 0
   ⇒   w_ridge = S_xy / (S_xx + λ)

Compare this directly to w_OLS = S_xy / S_xx. Ridge simply adds λ to the denominator. Rewriting, w_ridge = w_OLS · [S_xx / (S_xx + λ)] — the ridge weight is the OLS weight multiplied by a shrinkage factor that is always strictly between 0 and 1 for any finite λ > 0. Three consequences follow immediately from this one formula, and all three are exact, not approximate:

• As λ → 0, the shrinkage factor → 1, so w_ridge → w_OLS: no regularization, no shrinkage.
• As λ → ∞, the shrinkage factor → 0, so w_ridge → 0: infinite regularization crushes every weight toward zero.
• For any finite λ, the shrinkage factor is strictly positive, so w_ridge can get arbitrarily close to zero but can equal exactly zero only in the knife-edge case w_OLS = 0 to begin with. Ridge shrinks; it essentially never zeroes out a genuinely nonzero coefficient.

Lasso Regression (L1): Deriving the Soft-Threshold and Why Zeros Appear

Now replace the penalty with the L1 form, λ|w|, keeping everything else identical:

L(w) = (1/2)Σ(yᵢ − wxᵢ)²  +  λ|w|
     = (1/2)S_xx·w² − S_xy·w + λ|w| + constant

The function |w| has a sharp corner at w = 0 — it is not differentiable there, since its slope is +1 for every w > 0 and −1 for every w < 0, and no single tangent line fits at the corner itself. (This is a standard continuity-and-differentiability example from the JEE/CBSE calculus syllabus: f(w) = |w| is continuous everywhere but fails the differentiability test at w = 0 because the left-hand derivative, −1, does not equal the right-hand derivative, +1.) Because of this kink, we cannot just set one derivative to zero — we must handle three separate cases.

Case 1: w > 0. Here |w| = w, so the objective is smooth and its derivative is S_xx·w − S_xy + λ. Setting this to zero:

w = (S_xy − λ) / S_xx     — valid only if this comes out positive, i.e. only if S_xy > λ

Case 2: w < 0. Here |w| = −w, so the derivative is S_xx·w − S_xy − λ. Setting this to zero:

w = (S_xy + λ) / S_xx     — valid only if this comes out negative, i.e. only if S_xy < −λ

Case 3: w = 0, checked separately. Since L(w) has no single derivative at the corner, we test whether w = 0 is a minimum directly, by checking the one-sided slopes. The right-hand slope of L at 0 (approaching from w > 0) is λ − S_xy; for L to be non-decreasing as w moves right from zero, we need this to be at least 0, i.e. S_xy ≤ λ. The left-hand slope of L at 0 (approaching from w < 0) is −S_xy − λ; for L to be non-increasing as w moves left toward zero (equivalently, non-decreasing as w moves away from zero to the left), we need this to be at most 0, i.e. S_xy ≥ −λ. Both conditions together mean: whenever |S_xy| ≤ λ, the function L is "V-shaped" around zero — sloping up on both sides — so w = 0 is exactly where the minimum sits.

Putting the three cases together gives the celebrated soft-thresholding solution:

w_lasso =  (S_xy − λ)/S_xx     if S_xy > λ
           (S_xy + λ)/S_xx     if S_xy < −λ
           0                    if |S_xy| ≤ λ

This is the entire reason Lasso produces sparsity, derived with nothing beyond one-sided derivatives. Ridge's denominator-shrinkage formula can only shrink a nonzero OLS estimate proportionally; it never crosses zero for any finite λ. Lasso's soft-threshold formula literally subtracts a fixed amount λ from the size of the signal (S_xy), and if the signal was weaker than λ to begin with, the result is clipped to exactly zero — not "close to zero," exactly zero. Any feature whose correlation with the target is too small to overcome the penalty λ gets deleted from the model entirely.

A Fully Worked Numerical Example

Let's make this concrete with two features and one value of λ = 5, using S_xx = 10 for both (imagine both features have been standardized to the same scale). Feature 1 is a real, informative feature with S_xy = 25. Feature 2 is essentially noise, weakly correlated with the target, with S_xy = 3.

Feature 1 (S_xy = 25): w_OLS = 25/10 = 2.5. Ridge: w = 25/(10+5) = 25/15 ≈ 1.667 — shrunk, but still clearly present. Lasso: since 25 > λ = 5, w = (25−5)/10 = 2.0 — reduced, but still clearly present.

Feature 2 (S_xy = 3): w_OLS = 3/10 = 0.3. Ridge: w = 3/(10+5) = 3/15 = 0.2 — shrunk, but still nonzero: the useless "shoe size" feature is still sitting in the model with a small but nonzero coefficient. Lasso: since |3| ≤ λ = 5, w = 0 — the feature is removed from the model completely.

This is exactly the behaviour you would want from an automatic feature-selection tool. The code below implements the two closed-form update rules exactly as derived above and reproduces these four numbers precisely, so you can trace every line yourself:

def ridge_1d(Sxy, Sxx, lam):
    return Sxy / (Sxx + lam)

def lasso_1d(Sxy, Sxx, lam):
    if Sxy > lam:
        return (Sxy - lam) / Sxx
    elif Sxy < -lam:
        return (Sxy + lam) / Sxx
    else:
        return 0.0

Sxx = 10
lam = 5

for name, Sxy in [("Feature 1 (real signal)", 25), ("Feature 2 (pure noise)", 3)]:
    w_ols   = Sxy / Sxx
    w_ridge = ridge_1d(Sxy, Sxx, lam)
    w_lasso = lasso_1d(Sxy, Sxx, lam)
    print(f"{name}: OLS={w_ols:.3f}  Ridge={w_ridge:.3f}  Lasso={w_lasso:.3f}")

# Output:
# Feature 1 (real signal): OLS=2.500  Ridge=1.667  Lasso=2.000
# Feature 2 (pure noise): OLS=0.300  Ridge=0.200  Lasso=0.000

In a real multi-feature dataset, Lasso is not solved feature-by-feature independently like this — the true objective involves all features simultaneously through the full loss. But the standard algorithm used to solve it, coordinate descent, works by cycling through each feature one at a time, holding all other weights fixed, and applying exactly this soft-threshold update using the "residual correlation" for that feature (the S_xy-like quantity computed against the current residuals rather than raw y). So the 1-D derivation above is not a toy simplification you throw away later — it is the literal inner loop of production Lasso solvers such as scikit-learn's and glmnet's.

The Geometric Picture: Diamonds, Circles, and Corners

There is an equivalent, and famously illuminating, way to see the same result: regularized regression can be rewritten as a constrained optimization — minimize the plain OLS loss subject to the weights lying inside some fixed region around the origin. For L2, that region is a circle (in 2-D; a sphere/ball in higher dimensions): w₁² + w₂² ≤ t. For L1, it is a diamond: |w₁| + |w₂| ≤ t. Larger λ corresponds to a smaller allowed region t; the two formulations — penalty and constraint — produce the same solutions for a matching pair of λ and t.

Picture the OLS solution as a point outside this region (since without a size limit, OLS would pick weights that are "too big" from the regularized model's point of view). The loss function's contours — the set of all weight vectors giving the same training error — are ellipses (or circles, in the special case of equally-scaled, uncorrelated features) centered on that OLS point, growing outward as error increases. The regularized solution is the point where the smallest such contour first touches the allowed region.

Here is the key geometric fact the diagram below makes visible: a circle has a smooth boundary everywhere, so the growing contour generically first touches it at some ordinary point on the curve — a point where, generically, neither coordinate is zero. A diamond, however, has four sharp corners sitting exactly on the axes. Because a corner is not smooth, a whole range of contour orientations can be "tangent" there simultaneously (there is no single tangent line at a corner — any line whose slope lies between the slopes of the two adjacent edges will touch only at that corner). This makes the corner a much more likely first point of contact than any other point on the diamond's boundary. And every corner of the diamond, by its very shape, lies exactly on an axis — meaning one of the two coordinates is exactly zero there. That is the geometric twin of the soft-thresholding algebra above.

Note that the loss contours are drawn as circles here for exact, verifiable tangency in the diagram; in a real regression with correlated or differently-scaled features they are generally tilted ellipses, but the qualitative story — smooth boundaries meet contours at generic points, corners attract tangencies — is unchanged and is a completely general property of convex regions.

Two Misconceptions, Corrected

Misconception 1: "L1 and L2 do basically the same thing, just with a different-looking formula." They do not. We proved this twice — once algebraically (the ridge shrinkage factor is a continuous multiplier that only reaches exactly zero in the limit λ → ∞, while the lasso soft-threshold subtracts a fixed amount and clips at zero for any finite λ once the signal is weak enough) and once geometrically (smooth circle vs. cornered diamond). In fact, for the Lasso there is a specific finite value λ_max (equal to the largest |S_xy| across all features, in the standardized-coordinate case) beyond which every single weight is driven to exactly zero — the whole model collapses to predicting the mean. Ridge has no such finite collapsing point; weights only approach zero asymptotically as λ → ∞.

Misconception 2: "You can regularize the raw features and the intercept exactly as they come." Two real pitfalls hide here. First, regularization penalizes the size of a coefficient, and "size" is meaningless without a common scale — a feature measured in rupees (values in the lakhs) will naturally get a tiny OLS coefficient, while a feature measured in years (values like 1–10) will naturally get a much larger one, purely because of units, not because of importance. Penalizing raw w² or |w| would then unfairly crush whichever feature happens to have large-valued units. The fix, always applied before Ridge or Lasso in practice, is to standardize every feature (subtract its mean, divide by its standard deviation) so all coefficients are being compared on the same footing. Second, the intercept term b₀ is conventionally excluded from the penalty entirely — it only sets the baseline output level (which depends on an arbitrary choice of where you measure y from), and penalizing it would bias predictions toward zero for no principled reason.

Choosing λ: The Bias-Variance Trade-off

λ is not learned by the optimizer — it is a hyperparameter you choose from outside, typically via k-fold cross-validation: train on part of the data for a grid of candidate λ values, measure error on a held-out part, and pick whichever λ minimizes that held-out error. As λ increases from 0, two things happen in opposite directions. Training error increases monotonically — you are handing the optimizer a shrinking amount of freedom to fit the data, so it necessarily fits the training set somewhat worse. Test (generalization) error typically traces a U-shape: it first decreases as reasonable regularization suppresses the wild, noise-fitting coefficients that caused overfitting, reaches a minimum at some well-chosen λ, and then increases again once λ grows so large that the model becomes too simple to capture real signal (this final regime is called underfitting, or high bias). The entire purpose of tuning λ is to sit at the bottom of that U — the sweet spot between a model too flexible to trust and one too rigid to be useful.

Elastic Net: Combining Both, Briefly

Lasso has one practical weakness worth knowing: when two features are highly correlated with each other (near-duplicates, statistically speaking), Lasso tends to pick one somewhat arbitrarily and zero out the other, even if both are genuinely informative — because once one of the pair explains the shared signal, the residual correlation left for the second one drops below the threshold λ and it gets clipped to zero. Ridge does not have this problem; it tends to spread weight evenly across correlated features rather than picking a winner. Elastic Net is the natural fix: it uses a combined penalty αλ|w| + (1−α)(λ/2)w² for some mixing parameter α between 0 and 1, inheriting Lasso's ability to zero out truly useless features while keeping Ridge's tendency to stabilize groups of correlated, jointly-informative ones.

Where This Shows Up in Your Exams

The differentiability failure of f(w) = |w| at w = 0, and the one-sided (left-hand/right-hand) derivative test used to handle it, is a standard CBSE Class 12 and JEE topic under "Continuity and Differentiability" — the soft-threshold derivation above is that exact technique applied to an optimization problem instead of a plain differentiability check. The general recipe of writing an optimization objective as a sum of two convex functions and reasoning about their combined minimum connects directly to the calculus-based optimization ("maxima and minima") problems that appear every year in CBSE boards and JEE Main. At the GATE-foundation and undergraduate-ML level, Ridge and Lasso regression, the bias-variance trade-off, and cross-validation for hyperparameter selection are core, frequently-tested topics in every introductory machine learning and data science curriculum, precisely because the closed-form derivations here generalize directly (via coordinate descent, as noted above) to real, high-dimensional datasets.

Check Yourself

1. Given S_xx = 8, S_xy = 20, and λ = 4, compute w_OLS, w_ridge, and w_lasso using the formulas derived in this chapter.
2. Given S_xx = 8, S_xy = 3, and λ = 4, compute the same three values. What happens to the Lasso weight, and why?
3. Two columns in your dataset are "distance from home to school in kilometres" and "distance from home to school in metres" — essentially the same feature, scaled by 1000. Explain, using the soft-threshold formula, why applying Lasso directly to the unstandardized raw values could arbitrarily zero out one of these two nearly-identical columns while keeping the other with a very different-looking coefficient.
4. Explain, using one-sided derivatives, why f(w) = |w| is not differentiable at w = 0, and then explain why this same property is precisely what allows the Lasso objective to have its minimum sit exactly at w = 0 for a whole range of S_xy values, rather than at just a single point.
5. A friend claims: "If Ridge regression gives every feature a small nonzero coefficient, it must be a worse model than Lasso, which gives a clean, simple set of nonzero features." Is this claim always true? Under what circumstance (hint: correlated features) might Ridge's behaviour actually be preferable?

Answer Key

1. w_OLS = 20/8 = 2.5. w_ridge = 20/(8+4) = 20/12 ≈ 1.667. Since S_xy = 20 > λ = 4, w_lasso = (20−4)/8 = 16/8 = 2.0.

2. w_OLS = 3/8 = 0.375. w_ridge = 3/(8+4) = 3/12 = 0.25 (shrunk but nonzero). Since |S_xy| = 3 ≤ λ = 4, w_lasso = 0 exactly — the weak feature is deleted from the model.

3. S_xy for the metre-scaled column will be roughly 1000× larger in raw units than for the kilometre-scaled column (since S_xy = Σxᵢyᵢ scales linearly with the feature's units), even though both encode identical information. Against the same fixed λ, the metre column's S_xy will very likely exceed λ (kept, with a tiny-looking coefficient) while the kilometre column's much smaller S_xy may fall below λ and be zeroed — an artifact of units, not of actual relevance. This is exactly why standardization must happen before regularization.

4. The right-hand derivative of |w| at 0 is +1 and the left-hand derivative is −1; since these disagree, no single derivative exists at w = 0, so |w| is not differentiable there. This same kink means the combined Lasso objective's slope jumps discontinuously from negative to positive as w crosses 0 (rather than passing smoothly through zero), so there is an entire interval of S_xy values, namely |S_xy| ≤ λ, for which the minimum is trapped exactly at that corner — unlike a smooth function, where the minimum sits at exactly one point for each S_xy.

5. No. Ridge's "worse" appearance (nonzero everywhere) is not automatically worse predictive performance — sparsity is an interpretability and feature-selection property, not a guarantee of lower error. When features are highly correlated, Lasso's tendency to arbitrarily pick one of the correlated group and zero the rest can make the model unstable (a tiny change in training data can flip which correlated feature gets kept), whereas Ridge spreads weight across the correlated group and is more stable. In that setting, Ridge — or Elastic Net, which blends both — is often the more reliable choice even though its coefficient list looks "messier."

Summary

Unregularized least squares can fail outright when the number of features approaches or exceeds the number of samples, because XᵀX becomes singular or ill-conditioned, and the resulting weights become extremely sensitive to noise in the training data. Regularization fixes this by adding a penalty on weight size to the training objective. Ridge (L2) adds (λ/2)w², whose derivative is a smooth, linear function of w; the resulting closed-form solution, w_ridge = S_xy/(S_xx+λ), shrinks every coefficient by a proportional factor but never sets a genuinely nonzero coefficient to exactly zero for finite λ. Lasso (L1) adds λ|w|, whose derivative jumps discontinuously at w = 0; solving the three resulting cases by hand gives the soft-threshold rule w_lasso = sign(S_xy)·max(|S_xy|−λ, 0)/S_xx, which clips weak coefficients to exactly zero once their signal falls below λ, producing genuinely sparse models useful for automatic feature selection. Geometrically, this is the same fact as a diamond constraint region having corners on the axes while a circular constraint region has none. Both penalties trade increased training bias for reduced variance, and the right amount of regularization — the value of λ — is chosen by cross-validation at the bottom of the resulting bias-variance U-curve, not by intuition or by default settings.