Open a calculator app and type sin(37). An answer appears in a fraction of a second: 0.6018. There is no giant lookup table of sines stored on your phone, and the chip inside it cannot literally "measure an angle" the way a protractor does. What it does instead is evaluate a polynomial — a sum of powers of a number, multiplied and added — because polynomials are the only functions a processor can compute using nothing but multiplication and addition. Somehow, a polynomial stands in for sine well enough that you cannot tell the difference to four decimal places.
The same trick, in a different costume, is running every time you train a neural network. When an optimizer decides how to nudge a million weights to make the loss go down, it does not know the true shape of the loss function across the entire 1,000,000-dimensional space — that would be computationally impossible to explore. It only knows the loss and its slope at the current point. It builds a cheap, local, straight-line stand-in for the loss function, moves a small step guided by that stand-in, and repeats. Gradient descent is, quite literally, a Taylor series truncated after one term, applied over and over.
This chapter builds the Taylor series from the ground up — starting from a single tangent line you already understand from Class 11 calculus, and ending at the mathematical object that both your calculator and your neural network's optimizer depend on.
1. Where You Already Stand: The Tangent Line
You know that the derivative f'(a) is the slope of the tangent line to the curve y = f(x) at the point x = a. The equation of that tangent line, using point-slope form, is:
L(x) = f(a) + f'(a)(x - a)Here is the idea that unlocks everything else in this chapter: for x close to a, the curve and its tangent line are almost the same height. The tangent line is not just a geometric object you draw for fun — it is a usable approximation of the function itself, valid in a small neighbourhood around a.
Let's test this with a concrete number. Take f(x) = √x and approximate √4.1 without a calculator, using a = 4 (a nearby point where the square root is exact).
First, f(4) = 2. Next, f'(x) = 1/(2√x), so f'(4) = 1/4 = 0.25. The linear approximation is:
f(4.1) ≈ f(4) + f'(4)(4.1 - 4)
= 2 + 0.25 × 0.1
= 2.025The true value is √4.1 = 2.024846…. Our estimate is off by only 0.000154 — accurate to three decimal places, computed with nothing but a slope. This is the CBSE Class 12 "Approximations" result you may recognise as Δy ≈ (dy/dx)·Δx, and it is nothing more than the linear (first-degree) Taylor approximation in disguise.
2. The Tangent Line's Limitation, and How to Fix It
The tangent line is a straight line. The function √x curves. No matter how well the tangent line matches at a, it will always drift away as x moves further from a, because a straight line simply cannot bend to follow a curve. We need a better stand-in — one that is allowed to curve too.
The natural next step is a parabola. Suppose we look for a quadratic polynomial P(x) = c₀ + c₁(x − a) + c₂(x − a)² that matches f not only in value and slope at a, but also in curvature (second derivative) at a. We solve for the unknown coefficients by matching derivatives at x = a:
P(a) = c₀. We want this to equalf(a), so c₀ = f(a).P'(x) = c₁ + 2c₂(x − a), soP'(a) = c₁. We want this to equalf'(a), so c₁ = f'(a).P''(x) = 2c₂(constant), soP''(a) = 2c₂. We want this to equalf''(a), so c₂ = f''(a) / 2.
This gives the quadratic approximation:
f(x) ≈ f(a) + f'(a)(x - a) + [f''(a) / 2](x - a)²Notice the pattern in the denominators: 1 (for c₀, since 0! = 1), 1 (for c₁, since 1! = 1), and 2 (for c₂, since 2! = 2). This is not a coincidence, and it is the key that generalises to arbitrarily many terms.
3. The General Pattern: Why the Coefficient Is Always f⁽ᵏ⁾(a)/k!
Suppose we build a degree-n polynomial P(x) = c₀ + c₁(x−a) + c₂(x−a)² + … + cₙ(x−a)ⁿ and demand that all its derivatives up to order n match those of f at x = a. Differentiate P exactly k times and then substitute x = a. Two things happen to every term in the sum:
- Every term of degree lower than k becomes zero after k differentiations (you run out of powers to bring down).
- Every term of degree higher than k still has at least one factor of
(x − a)left after k differentiations, and that factor becomes zero the moment we plug inx = a.
Only the term of degree exactly k survives. Differentiating cₖ(x−a)ᵏ k times brings down the falling factorial k(k−1)(k−2)…1 = k!, leaving k! · cₖ — a constant, with the (x−a) factor gone entirely. So:
P^(k)(a) = k! · cₖ ⟹ cₖ = f^(k)(a) / k!This is the coefficient rule for every term, for every degree. Putting it all together gives the Taylor series of f centered at a:
f(x) = f(a) + f'(a)(x-a) + [f''(a)/2!](x-a)² + [f'''(a)/3!](x-a)³ + …
= Σ (from k=0 to ∞) f^(k)(a)/k! · (x-a)^kWhen a = 0, this is called a Maclaurin series — the special case centered at the origin, which is what you'll use most often in practice because the derivatives usually simplify nicely there.
4. Two Series Worth Memorising the Derivation Of
The exponential, ex. This function has the unique property that every derivative of e^x is e^x itself. So f^(k)(x) = e^x for every k, which means f^(k)(0) = e⁰ = 1 for every k. Substituting into the Maclaurin formula, every coefficient is 1/k!:
e^x = 1 + x + x²/2! + x³/3! + x⁴/4! + …This is exactly the series a math library computes internally when you call exp() — including, ultimately, inside the softmax and sigmoid activation functions used across every neural network you'll ever train.
Sine, sin(x). The derivatives of sine cycle with period 4: f = sin x → f' = cos x → f'' = −sin x → f''' = −cos x → f'''' = sin x (back to the start). Evaluated at 0: sin 0 = 0, cos 0 = 1, −sin 0 = 0, −cos 0 = −1, sin 0 = 0, and the pattern 0, 1, 0, −1 repeats forever. All even-order derivatives vanish, and the odd-order ones alternate in sign:
sin(x) = x - x³/3! + x⁵/5! - x⁷/7! + …This is precisely how your calculator computed sin(37°) at the start of this chapter — after converting 37° to radians, it summed a handful of terms of exactly this series.
5. Seeing All Three Approximations Together
The diagram below plots f(x) = eˣ (solid blue) alongside its Maclaurin expansions truncated to one term — the tangent line, degree 1 (dashed red) — and two terms of curvature — the quadratic, degree 2 (dashed green) — both centered at a = 0. Watch how each approximation hugs the true curve near a and peels away as x moves further out, and how the quadratic peels away more slowly than the line, because it has one extra degree of freedom to bend with.
Read the callout at x = 1 directly off the plot: the true value is 2.718, the quadratic guesses 2.500 (an error of 0.218), and the plain tangent line guesses only 2.000 (an error of 0.718). Adding one extra term more than tripled the accuracy — and this pattern, more terms shrinking the error, is the entire reason Taylor series is useful rather than a curiosity.
6. Two Misconceptions Worth Killing Now
Misconception 1: "A linear approximation means the function actually is a straight line near a." It doesn't. Look again at the diagram — eˣ never stops curving; it only looks straight over a small enough window, the same way the curved surface of the Earth looks flat when you're standing on a football field. The formal statement is that the error f(x) − L(x) shrinks faster than (x − a) itself as x → a — but for any fixed, non-zero distance from a, the gap is real and computable, as you saw at x = 1 above (error 0.718, not zero).
Misconception 2: "Adding more Taylor terms always gives a better approximation, no matter how far x is from a." This is false in general. A Taylor series only converges to the true function value within a specific interval called the radius of convergence — for ln(1 + x) centered at 0, for example, the series only converges for −1 < x ≤ 1; feed it x = 3 and adding more terms makes the estimate worse, not better, because the series diverges out there entirely. "More terms = more accurate" is only true inside the region where the series is valid in the first place — always check that you're close enough to the center before trusting extra terms.
7. The ML Payoff: Gradient Descent Is a One-Term Taylor Series
Now the connection this chapter has been building toward. Let L(w) be a loss function of a single weight w (the multi-weight case works the same way, with the derivative replaced by a gradient vector). Suppose we're currently at weight w₀ and we're considering a step Δw. Taylor-expand the loss at the new point around the current point:
L(w₀ + Δw) ≈ L(w₀) + L'(w₀)·Δw + [L''(w₀)/2]·Δw² + …Truncate after the first-order term — exactly the tangent-line approximation from Section 1, just relabelled:
L(w₀ + Δw) ≈ L(w₀) + L'(w₀)·ΔwThis local linear model tells us the loss decreases when we move opposite to the sign of L'(w₀). But notice a problem: a genuinely linear function has no minimum — it just keeps decreasing forever in one direction. If we trusted this approximation completely and tried to minimize it exactly, we'd take an infinitely large step, which is nonsense, because the linear model is only valid near w₀. The fix is to take a deliberately small, bounded step in the descending direction:
Δw = -η · L'(w₀)This is the gradient descent update rule, and η (the learning rate) is not an arbitrary knob — it exists specifically to keep Δw small enough that the linear Taylor approximation stays trustworthy. This is precisely why a learning rate that's too large causes training to diverge: the optimizer takes a step so big that it exits the region where "the loss looks like a straight line" was ever a valid description, and the true (curved) loss can shoot back up instead of going down.
What if we keep the quadratic term instead of discarding it? Then we can do better than guessing a step size — we can solve for the exact minimum of the quadratic approximation itself. Differentiate the truncated series with respect to Δw and set it to zero:
d/d(Δw) [L(w₀) + L'(w₀)Δw + (L''(w₀)/2)Δw²] = L'(w₀) + L''(w₀)Δw = 0
⟹ Δw = -L'(w₀) / L''(w₀)This is Newton's method (Newton–Raphson) applied to optimization, and it converges dramatically faster near a minimum than plain gradient descent because it uses curvature information, not just slope. So why doesn't every neural network use it? Because L''(w₀), in the multi-weight case, is the Hessian matrix — a matrix with one entry for every pair of weights. A model with 10 million parameters has a Hessian with 10 million × 10 million ≈ 1014 entries — computing and inverting that per step is completely infeasible. First-order gradient descent (and its refinements like Adam, which cheaply estimates curvature statistically rather than computing it exactly) is a scalability trade-off, not a claim that the linear model is somehow "more correct."
8. Where This Shows Up in Your Exams
- CBSE Class 12, Application of Derivatives: the "approximate value" questions (
Δy ≈ (dy/dx)Δx) are exactly the degree-1 Taylor truncation from Section 1 — you've already been doing Taylor series without the name. - JEE Main / Advanced: Maclaurin expansions of
eˣ,sin x,cos x, andln(1+x)appear directly in limit evaluation (replacing a function with its series to resolve a 0/0 form is often faster than L'Hôpital's rule) and in estimation problems. - KVPY / Olympiad: problems asking you to bound the error of a truncated series, or to identify how many terms are needed for a given precision, test the remainder-term intuition from Section 6.
- GATE (foundation level): Newton–Raphson root-finding, which you derived in Section 7 as the zero of a first-order Taylor expansion of a function (not a loss), is a standard Numerical Methods topic.
9. Verifying the Series Numerically
The code below computes the Maclaurin series for e¹ using a growing number of terms, and compares each partial sum against Python's built-in math.exp. Trace it by hand before running it: term starts at 1.0 (representing x⁰/0! = 1), and after adding it to the running total, it's updated by multiplying by x/(k+1) — which turns xᵏ/k! into x^(k+1)/(k+1)! in one step, since xᵏ/k! × x/(k+1) = x^(k+1)/(k+1)!.
import math
def taylor_exp(x, n_terms):
total = 0.0
term = 1.0 # holds x^k / k!, starting at k = 0
for k in range(n_terms):
total += term
term *= x / (k + 1) # advance x^k/k! to x^(k+1)/(k+1)!
return total
for n in [1, 2, 4, 8, 15]:
approx = taylor_exp(1.0, n)
exact = math.exp(1.0)
print(f"n={n:2d} approx={approx:.6f} error={abs(exact-approx):.6f}")Sample output:
n= 1 approx=1.000000 error=1.718282
n= 2 approx=2.000000 error=0.718282
n= 4 approx=2.666667 error=0.051615
n= 8 approx=2.718254 error=0.000028
n=15 approx=2.718282 error=0.000000Follow the trace for n=4 by hand to confirm it: k=0 adds 1 (total 1, term becomes 1); k=1 adds 1 (total 2, term becomes 0.5); k=2 adds 0.5 (total 2.5, term becomes 1/6); k=3 adds 1/6 ≈ 0.166667 (total 2.666667). Four terms, four lines of arithmetic, and the error has already dropped from 1.72 to 0.05 — this is the shrinking-error behaviour from Section 5, now verified in code instead of read off a graph.
10. Check Your Understanding
Q1. Using linear approximation, estimate ln(1.05) by first finding the first three terms of the Maclaurin series of ln(1+x).
Solution: f(x)=ln(1+x), f(0)=0. f'(x)=1/(1+x), f'(0)=1. f''(x)=−1/(1+x)², f''(0)=−1, so the degree-2 coefficient is −1/2!=−0.5. f'''(x)=2/(1+x)³, f'''(0)=2, so the degree-3 coefficient is 2/3!=1/3. Series: ln(1+x) ≈ x − x²/2 + x³/3. At x=0.05: 0.05 − 0.00125 + 0.0000417 ≈ 0.048792. The true value is 0.048790 — accurate to five decimal places with three terms.
Q2. A toy loss function is L(w) = w² − 4w + 7. Using the first-order Taylor expansion around w₀ = 0 with learning rate η = 0.1, compute the gradient descent step Δw, and check whether it moves toward the true minimum.
Solution: L'(w) = 2w − 4, so L'(0) = −4. Δw = −η·L'(0) = −0.1×(−4) = 0.4, giving w₁ = 0.4. The exact minimum is where L'(w)=0, i.e. w = 2. The step moved from 0 toward 2 — correct direction, and it will keep closing the gap on further steps.
Q3. Why doesn't gradient descent just always use the Newton step Δw = −L'(w₀)/L''(w₀) if it converges faster?
Solution: It uses the quadratic (curvature) Taylor term, which for a network with many parameters becomes a full Hessian matrix — one entry per pair of weights. For millions of parameters this matrix is too large to compute or invert every step, so practical training uses cheaper first-order methods instead, trading convergence speed for feasibility.
Q4. True or false: "A degree-6 Taylor polynomial is always a better approximation of f(x) than a degree-2 one, for any x." Justify.
Solution: False. This only holds inside the function's radius of convergence, where the remainder shrinks as degree increases. Outside that radius the series may not converge to f(x) at all, so adding terms doesn't help and can even make the partial sum wander further from the true value.
Summary
A Taylor series replaces a function with a polynomial that matches it — and all of its derivatives — at one point, using coefficients f^(k)(a)/k!. Truncated at degree 1, it's the tangent-line approximation you already knew from Class 11–12 calculus. Truncated at degree 2, it captures curvature and roughly triples the accuracy for the same effort, as the eˣ diagram showed numerically. Extended to infinitely many terms, it reproduces functions like eˣ and sin x exactly, within a radius of convergence — which is why more terms are not automatically better outside that radius. And in machine learning, this isn't background theory: gradient descent is the degree-1 truncation applied to a loss function, the learning rate exists to keep that truncation trustworthy, and Newton's method is the degree-2 truncation solved exactly — abandoned at scale only because the Hessian it requires is too expensive to compute for models with millions of weights.