Learning Rate Scheduling: Dynamic Speed Control

You already know the update rule that drives every neural network you train:

w := w - lr * gradient

There is one number in that line you have probably been treating as a fixed setting you pick once and forget: lr, the learning rate. This chapter proves, with an example small enough to compute by hand, that treating lr as a constant is a mistake — and shows you the exact functions that fix it.

Why a Single Fixed Learning Rate Eventually Fails

Take the simplest possible loss surface: L(w) = w², a single weight sitting in a parabolic bowl. This is not a toy for its own sake — near any local minimum, a smooth loss function looks approximately like this (a second-order Taylor expansion around the minimum is exactly a paraboloid), so whatever we learn here about w² tells us something true about the last stretch of training any real network goes through.

The gradient is dL/dw = 2w. Plugging into gradient descent:

w_(t+1) = w_t - lr * 2 * w_t = w_t * (1 - 2*lr)

This is a geometric sequence. Starting from some w_0, after t steps:

w_t = w_0 * (1 - 2*lr)^t

Everything about whether training converges, diverges, or crawls is hiding inside the number r = 1 - 2*lr, because a geometric sequence r^t only shrinks to zero when |r| < 1. Let's test three concrete values of lr with w_0 = 1.

• lr = 1.2 (too large). r = 1 - 2.4 = -1.4, so |r| = 1.4 > 1. The sequence is 1, -1.4, 1.96, -2.744, ... — the magnitude grows by a factor of 1.4 every single step. This is not a metaphor for instability; it is literally an exploding geometric series. In a real network this is the "loss becomes NaN after epoch 3" bug every DL practitioner has hit.
• lr = 0.01 (safe but slow). r = 1 - 0.02 = 0.98. To shrink w_t to 1% of its starting value you need 0.98^t = 0.01, which gives t = ln(0.01) / ln(0.98) ≈ 228 steps. Safe, but you have paid for that safety with roughly 228 update steps to do what could, in principle, take far fewer.
• lr = 0.5 (exactly optimal for this bowl). r = 1 - 1.0 = 0, so w_1 = 0. One step. Done.

That third case is not a coincidence — it is a real, provable fact worth deriving because it explains where "optimal learning rate" numbers actually come from. The second derivative of L(w) = w² is L''(w) = 2; this number measures the curvature of the bowl. The optimal fixed step size for a quadratic is exactly lr = 1 / L''(w) — here, 1/2 = 0.5. This is the same idea Newton's method uses to jump straight to a minimum using curvature information.

Now here is the problem that no fixed learning rate can solve. A real network has thousands to billions of weights, and the loss surface curves differently along every one of those directions, and that curvature itself changes as training progresses and the weights move to a different region of the landscape. A learning rate tuned to be optimal for a steep, narrow direction (high curvature, needs a small lr or it overshoots and oscillates, exactly like our lr = 1.2 case) will crawl uselessly along a shallow, flat direction (low curvature, needs a large lr to make any progress). There is no single number that is simultaneously safe for the steep parts and fast for the flat parts, and worse — the mixture of steep and flat directions a network is currently traveling through changes as training proceeds. Early training typically needs bigger, more exploratory steps to cross flat plateaus and saddle regions quickly; late training, close to a minimum, needs small, careful steps so it doesn't bounce past the exact bottom the way our lr = 1.2 example did. A schedule that changes lr over time is how we get both behaviours out of one training run.

The Second Reason: Stochastic Noise Doesn't Let You Settle

There's a second, independent reason schedules are necessary, and it has nothing to do with curvature. You never train on the full dataset's exact gradient — you train on mini-batches, so every gradient you compute is a noisy estimate of the true gradient. Near the minimum, the true gradient is close to zero, but your mini-batch gradient is a zero-mean random variable with some non-zero variance σ² added on top. With a fixed learning rate, the update at every step still carries a random kick of size roughly lr · σ. The weights never actually settle at the minimum — they settle into a small random cloud of radius proportional to lr around it, sometimes called the noise ball. The only way to shrink that ball to a point is to shrink lr as training proceeds.

Beyond the syllabus — how small, precisely? The classical result governing this, due to Robbins and Monro (1951), gives two conditions a schedule lr_t must satisfy to provably converge to the exact minimum despite the noise:

Sum(lr_t) = infinity          (can still travel arbitrarily far from the start)
Sum(lr_t^2) < infinity        (steps shrink fast enough to average out the noise)

A fixed learning rate fails the second condition outright (an infinite sum of a constant squared is still infinite), which is the rigorous version of the noise-ball argument above. A schedule like lr_t = c/t satisfies both: Sum(1/t) is the harmonic series and diverges (condition one holds), while Sum(1/t²) is a convergent p-series (condition two holds). This is the theoretical ancestor of every decay schedule you're about to see. In practice we train for a fixed, finite number of epochs rather than an infinite horizon, so we don't need the literal Robbins-Monro conditions — we just need lr to be comfortably small by the end of the finite budget we've been given. But the intuition — big steps early, vanishing steps late — is exactly the same idea in both the idealized theorem and every practical schedule below.

Four Schedules You Will Actually Use

A learning rate schedule is simply a function lr_t = f(t) that replaces the constant lr, where t is the epoch or step number. Here are the four you'll meet in every serious deep learning codebase.

1. Step decay. Multiply the learning rate by a shrink factor gamma every fixed number of epochs:

lr_t = lr_0 * gamma ^ floor(t / step_size)

With lr_0 = 0.1, gamma = 0.5, step_size = 25: the rate holds at 0.1 for epochs 0–24, drops to 0.05 for epochs 25–49, to 0.025 for 50–74, and to 0.0125 for 75 onward. Simple, and still the default in many computer-vision training recipes because it's easy to reason about.

2. Exponential decay. Continuous shrinkage every single step instead of jumps:

lr_t = lr_0 * exp(-k * t)

With lr_0 = 0.1 and k = 0.025, by epoch 50 the rate has fallen to 0.1 * exp(-1.25) ≈ 0.0287, and by epoch 100 to roughly 0.0082. There are no sudden jumps for the optimizer to adjust to, which sometimes gives smoother loss curves than step decay.

3. Cosine annealing. Introduced by Ilya Loshchilov and Frank Hutter in their 2017 paper on stochastic gradient descent with warm restarts (SGDR), this schedule rides one arc of a cosine curve from the maximum rate down to a minimum over a fixed horizon T:

lr_t = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * t / T))

Check the shape: at t = 0, cos(0) = 1, giving lr_max exactly. At t = T, cos(pi) = -1, giving lr_min exactly. In between, because cos is strictly decreasing on [0, pi], lr_t decreases smoothly and monotonically — no algebra trick required, just the shape of the cosine function over its first half-period. The distinctive feature compared to exponential decay is the shape of the descent: cosine annealing decreases slowly at first, fastest through the middle of training, and slowly again right at the end — giving the network a long, gentle final approach into the minimum instead of a rushed one.

4. Linear warm-up, then decay. The first three schedules all start at their maximum on step 0 — but step 0 is exactly when your weights are freshly randomly initialized and the loss surface under your feet is the least trustworthy it will ever be. Large early updates, especially combined with adaptive optimizers like Adam whose internal variance estimates are themselves unreliable in the first handful of steps, can send training somewhere it never recovers from. The fix is to ramp lr up from (near) zero over a short warm-up window before starting the decay:

lr_t = lr_max * t / T_warmup                              for t < T_warmup
lr_t = lr_min + 0.5*(lr_max-lr_min)*(1+cos(pi*(t-T_warmup)/(T-T_warmup)))   for t ≥ T_warmup

Figure: The Four Schedules Over 100 Epochs

All four curves below share the same starting point (lr_max = 0.1) so you can compare their shapes directly.

Notice the warm-up curve (dashed, orange) is the only one that starts at zero and climbs — it spends epochs 0–10 rising to the same peak the other three start at, then follows the same cosine-shaped descent as the plain cosine curve, just compressed into a shorter remaining window.

Verifying the Numbers in Code

Trace this by hand before you trust any library. Here is a direct implementation of step decay and cosine annealing:

import numpy as np

def step_decay(t, lr0=0.1, drop=0.5, epochs_per_drop=25):
    return lr0 * (drop ** (t // epochs_per_drop))

def cosine_anneal(t, T, lr_max=0.1, lr_min=0.0):
    return lr_min + 0.5 * (lr_max - lr_min) * (1 + np.cos(np.pi * t / T))

for t in [0, 25, 50, 75, 99]:
    print(t, round(step_decay(t), 4), round(cosine_anneal(t, 100), 4))

Trace it: at t = 25, 25 // 25 = 1, so step_decay = 0.1 * 0.5¹ = 0.05; and cosine_anneal = 0.05 * (1 + cos(45°)) = 0.05 * 1.7071 ≈ 0.0854. At t = 99, 99 // 25 = 3, so step_decay = 0.1 * 0.5³ = 0.0125; and cos(178.2°) ≈ -0.9995, so cosine_anneal ≈ 0.05 * 0.0005 ≈ 0.0000. The full printed output is:

0 0.1 0.1
25 0.05 0.0854
50 0.025 0.05
75 0.0125 0.0146
99 0.0125 0.0

Now the version you'll actually use, wiring a schedule into a real PyTorch training loop. The one detail that trips almost everyone up the first time: scheduler.step() is called once per epoch (for most schedulers), after optimizer.step(), never before — call it too early and epoch 0 trains with the wrong rate.

import torch

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=100, eta_min=0.0
)

for epoch in range(100):
    train_one_epoch(model, optimizer)   # forward, loss.backward(), optimizer.step()
    scheduler.step()                    # called AFTER optimizer.step(), once per epoch
    print(epoch, optimizer.param_groups[0]['lr'])

Adaptive Schedules: When the Function Isn't Fixed in Advance

Every schedule so far is a function of t alone, decided before training even starts. ReduceLROnPlateau instead watches your validation loss and only cuts lr when progress actually stalls — it has no fixed shape at all:

scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
    optimizer, mode='min', factor=0.5, patience=5
)

for epoch in range(100):
    train_one_epoch(model, optimizer)
    val_loss = evaluate(model, val_data)
    scheduler.step(val_loss)   # metric passed explicitly, unlike the other schedulers

Here, patience=5 means: if validation loss hasn't improved for 5 consecutive epochs, multiply lr by factor=0.5 and reset the patience counter. This is the closest thing to "let the data decide when to slow down" rather than committing to a shape upfront.

A related but distinct idea is the one-cycle policy, proposed by Leslie Smith in his 2018 "super-convergence" paper: instead of only decaying, ramp lr up from a small value to a much larger-than-usual peak over the first 30–45% of training, then decay it back down (often even below the starting value) for the rest, usually while moving momentum in the opposite direction (high momentum when lr is low, low momentum when lr is high). The surprising empirical result is that this can train some networks to comparable accuracy in a fraction of the usual epoch budget — the large mid-training learning rates act as a regularizer that helps the optimizer skip past sharp, narrow minima and settle into wider, more generalizable ones.

Common Misconceptions, Corrected

Misconception 1: "A lower learning rate is always the safer, better choice." This confuses safety from divergence with actual training quality. Our own lr = 0.01 example above was perfectly safe and still needed roughly 228 steps to do what lr = 0.5 did in one. Every training run has a finite epoch budget; a learning rate that is technically stable but too small to reach a good minimum within that budget produces an underfit model, which is just as much a failure as a diverged one — it has simply failed slowly instead of loudly.

Misconception 2: "The learning rate schedule and the optimizer are the same knob." They are two separate, composable decisions. The optimizer (SGD, Adam, RMSProp, ...) decides how a raw gradient is transformed into a per-parameter update — Adam, for instance, rescales each parameter's step using running estimates of gradient mean and variance. The schedule then multiplies whatever the optimizer produces by a single global scale factor lr_t that changes over time. You can pair any schedule with any optimizer; they answer different questions ("what direction and relative size per parameter" versus "how big overall, right now").

Misconception 3: "Learning rate decay and weight decay are the same thing." The similar names cause real confusion in code reviews. Learning rate decay shrinks the step size lr_t used in every update — it changes how far you move. Weight decay is an entirely different mechanism: it adds a small pull toward zero directly into the update (equivalent to L2 regularization on the weights), penalizing large weight magnitudes to reduce overfitting. A training run can use both simultaneously, and changing one has no direct algebraic relationship to the other — they appear as separate arguments in every deep learning framework for exactly this reason.

Exam Connections

The geometric-sequence convergence argument at the start of this chapter — testing whether |1 - 2*lr| < 1 — is precisely the fixed-point iteration convergence check that appears in JEE and BITSAT numerical-methods questions on iterative sequences and recurrence relations; the skill of identifying the ratio that governs growth or decay in a recurrence transfers directly. The harmonic-series-versus-p-series argument in the Robbins-Monro box is standard series content from the Class 11–12 syllabus, applied here to a genuine research question rather than an abstract exercise. If you go on to GATE's Data Science and Artificial Intelligence (DA) paper, learning rate schedules, and the optimization theory behind why they work, are examinable machine-learning content, not folklore.

Check Your Understanding

1. For L(w) = 4w², find the fixed learning rate that reaches the minimum in exactly one gradient descent step. (Use the same curvature argument as the worked example.)
2. A step decay schedule uses lr_0 = 0.2, gamma = 0.1, step_size = 10. What is lr_t at t = 25?
3. Explain, in terms of the noise-ball argument, why training with a fixed learning rate can plateau in validation loss even though training loss is still (very slowly) decreasing.
4. A friend proposes the schedule lr_t = 0.1 for all t, but says "it satisfies Robbins-Monro because the sum of lr_t over infinite steps is infinity." Is your friend right or wrong, and why?
5. Why does warm-up matter more for Adam-style adaptive optimizers than for plain SGD in the very first few steps?

Answers: (1) L''(w) = 8, so lr = 1/8 = 0.125. (2) t = 25 gives floor(25/10) = 2, so lr_25 = 0.2 * 0.1² = 0.002. (3) Once weights enter the noise ball around the minimum, the fixed-size random kicks from mini-batch noise keep pushing them to nearby but not-identical points every step; training loss (measured on the batch just used) can still inch down from lucky batches while validation loss, measured on unseen data, stops improving because the model isn't actually settling closer to any true minimum. (4) Wrong: the first Robbins-Monro condition (Sum(lr_t) = infinity) does hold for a constant, but the second (Sum(lr_t²) < infinity) fails, because summing a positive constant infinitely many times still diverges — both conditions are required. (5) Adam maintains running estimates of the first and second moments of the gradient, and those estimates are statistically unreliable when only a handful of steps' worth of data has been averaged into them; a large early update computed from a noisy, immature variance estimate can push weights somewhere training never recovers from, so ramping lr up gives those internal estimates time to stabilize first.

Summary

• A fixed learning rate cannot be simultaneously safe (small enough to avoid the oscillation/divergence our |1 - 2*lr| ≥ 1 condition predicts) and fast (large enough to cross flat regions quickly) across an entire training run, because curvature differs across directions and across training time.
• Even on a single well-behaved bowl, mini-batch gradient noise traps a fixed learning rate in a "noise ball" around the minimum instead of letting it settle exactly; only a shrinking lr_t can close that ball, formalized by the Robbins-Monro conditions Sum(lr_t) = infinity and Sum(lr_t²) < infinity.
• Step decay and exponential decay are the simplest schedules: multiply by a fixed factor either at intervals or continuously.
• Cosine annealing rides a cosine arc from lr_max to lr_min, decreasing slowly at both ends and fastest in the middle — provably monotonic because cosine is strictly decreasing on [0, pi].
• Linear warm-up ramps lr up from near zero before any decay begins, protecting against the unreliable gradients and immature optimizer statistics of the first few steps.
• ReduceLROnPlateau and the one-cycle policy depart from fixed-in-advance functions of t — one reacts to stalled validation loss, the other deliberately overshoots to a high peak rate as an implicit regularizer.
• Learning rate schedule, choice of optimizer, and weight decay are three independent, composable knobs — conflating any two of them is a common and costly bug.

Think About It

Think about this: How would you explain learning rate scheduling: dynamic speed control to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.