Automatic Differentiation and Computational Graphs

Every time you unlock your phone with your face, or a payments app flags a suspicious UPI transaction in milliseconds, a neural network with anywhere from a few thousand to several billion internal numbers — called weights — is running underneath. Training that network means searching for weight values that make its predictions accurate, and the only method we know that works at this scale is gradient descent: repeatedly nudge every single weight a little in the direction that reduces the model's error fastest. To compute that nudge correctly you need the exact derivative of the error with respect to every weight, recomputed after every training example or batch. A modest image classifier with 5 million weights needs 5 million partial derivatives, freshly computed, thousands of times over during training. Try writing out the derivative of even a 50-line neural network computation by hand and you will run out of paper long before you run out of chain rule. Yet PyTorch or TensorFlow computes all 5 million of those derivatives — exactly, not approximately — in a fraction of a second, every single time. That is not a clever shortcut or a numerical trick; it is a precise algorithm called automatic differentiation (AD). By the end of this chapter you will have derived that algorithm yourself on a small example by hand, and then implemented its core in about fifteen lines of Python.

Three ways to differentiate a program — and why two of them fail

Suppose a program computes some function f(x) — it could be one line or ten thousand lines of code. There are exactly three general strategies for getting its derivative, and understanding why two of them collapse at scale is what motivates AD.

1. Numerical differentiation. You already know the definition of a derivative: f′(x) = lim_h→0 [f(x+h) − f(x)] / h. A computer can approximate this directly by picking a small h and evaluating f twice. Take f(x) = x² at x = 2, with h = 0.001: f(2) = 4, f(2.001) = 4.004001, so the estimate is (4.004001 − 4) / 0.001 = 4.001 — close to the true value 4, but not exact. This gap is truncation error, and it shrinks as h shrinks — so why not just make h tiny, like 1e-15? Because computers store real numbers as floating-point values with roughly 15–17 significant decimal digits of precision. When h is smaller than the precision available near x, the computer literally cannot represent x+h as a distinct number from x — the subtraction f(x+h) − f(x) becomes 0 or pure rounding noise, and dividing that noise by a tiny h amplifies it into garbage. So numerical differentiation is trapped between two errors that both get worse at the extremes: truncation error if h is too large, catastrophic rounding error if h is too small. There is no value of h that eliminates both. And there is a second, more decisive problem for machine learning: estimating the derivative with respect to one weight this way costs one extra full evaluation of f. For 5 million weights you need roughly 5 million extra evaluations of the entire network just to get one gradient. Completely infeasible.

2. Symbolic differentiation. This is what Wolfram Alpha or a computer algebra system does: apply the rules of calculus (sum rule, product rule, chain rule) directly to the algebraic expression and produce a new algebraic expression for the derivative. This is exact, unlike the numerical method. But it has its own fatal flaw for real programs: expression swell. Consider building up a function by repeated squaring: h₁ = x, and h_k+1 = h_k · h_k for k = 1, 2, 3, …. By the product rule, the derivative of h_k+1 with respect to x is h_k′·h_k + h_k·h_k′. Notice h_k′ appears twice. If a symbolic engine naively substitutes the full written-out expression for h_k′ at both occurrences instead of remembering "these are the same subexpression, computed once," the size of the derivative expression roughly doubles at every step. After just 20 squarings — a function that takes only 20 multiplications to evaluate — the fully expanded symbolic derivative has on the order of 2²⁰, about a million, terms. The function is cheap to compute; its naively-expanded symbolic derivative is not. Real neural networks are exactly this kind of deeply repeated composition (the same weight matrix used at every layer, the same activation function applied millions of times), so naive symbolic differentiation is also a dead end at scale.

3. Automatic differentiation. This is the third way, and it is exact like the symbolic method but cheap like neither of the other two. The core insight: instead of manipulating an algebraic expression as text, decompose the program into a sequence of elementary operations — addition, multiplication, sin, exp, and so on — each with a known, simple derivative rule, and apply the chain rule numerically, node by node, as the program actually executes. Because each elementary operation is visited (and its derivative computed) exactly once no matter how many times its result is reused downstream, AD's cost is only a small constant multiple of the cost of running the original function once — regardless of whether you have 2 inputs or 2 billion. This is the punchline of the entire chapter, and everything below is building the machinery to make it precise.

The computational graph: turning a formula into a DAG

Any expression built from elementary operations can be drawn as a directed acyclic graph (DAG): nodes are values (inputs, intermediate results, and the final output), and a directed edge from node A to node B means "B's value is computed directly from A's value." This is not an abstraction invented for this chapter — it is literally the order of operations your compiler or interpreter already follows when it evaluates an expression; AD simply makes that structure explicit and reusable.

Take a concrete function we will use throughout this chapter:

f(x, y) = (x + y)·(x·y)

Decomposed into elementary operations, this becomes three steps:

• v1 = x + y
• v2 = x · y
• f = v1 · v2

The graph has five nodes: two input (leaf) nodes x and y, two intermediate nodes v1 and v2, and one output node f. Both x and y feed into two different downstream nodes each (x feeds v1 and v2; y feeds v1 and v2) — this "fan-out" is the single most important structural feature of the graph, and it is exactly what the rest of this chapter is about handling correctly.

Forward-mode AD: carrying derivatives alongside values

The most direct way to use this graph for differentiation is forward mode: alongside every value v_i in the graph, carry its derivative with respect to one chosen input, computed using the elementary rules you already know (sum rule: d(a+b) = da+db; product rule: d(a·b) = a·db + b·da), propagating from the inputs toward the output in the same order the program actually runs. A beautifully compact way to formalize this is with dual numbers: represent a value together with its derivative as a pair a + a′ε, where ε is a symbol satisfying ε² = 0 (it is not zero itself, just its square is defined to vanish). Multiplying two dual numbers: (a + a′ε)(b + b′ε) = ab + (ab′ + a′b)ε + a′b′ε² = ab + (ab′+a′b)ε, since ε²=0. Read off the ε coefficient: it is exactly ab′+a′b, the product rule, produced automatically by ordinary algebra rather than by invoking calculus as a separate rule. Addition works the same way and reproduces the sum rule. This is not a trick specific to one example — every elementary operation, run on dual numbers, automatically outputs the correct derivative alongside the correct value.

Forward mode has one crucial limitation: a single forward pass, seeded with one input's derivative set to 1, gives you the derivative of every downstream node with respect to that one input only. To get the gradient with respect to all N inputs, you need N separate forward passes. For a neural network with millions of inputs (weights), that is millions of passes — no better than numerical differentiation's cost problem.

Reverse-mode AD: the algorithm behind backpropagation

Reverse mode flips the direction and solves exactly this problem. First, run one ordinary forward pass, storing every intermediate value (v1 = 7, v2 = 12, f = 84 in our example — computed below). Then, starting from the output and seeding its own derivative with respect to itself as 1 (∂f/∂f = 1), walk backward through the graph, and at each node multiply the incoming gradient by that node's local derivative — the partial derivative of the node's own operation with respect to each of its direct inputs, computed while treating those inputs as independent at that single step (this local derivative is not yet the full derivative through the whole graph — that distinction matters and we return to it below).

The subtle part — and the mathematical heart of backpropagation — is what happens at a node like x that fans out to two downstream nodes, v1 and v2. A small change dx in x causes a small change dv1 = (∂v1/∂x)·dx and a small change dv2 = (∂v2/∂x)·dx simultaneously, because both v1 and v2 depend on x. The resulting change in f, to first order (the total differential), is df = (∂f/∂v1)·dv1 + (∂f/∂v2)·dv2. Substituting the two expressions for dv1 and dv2:

df = (∂f/∂v1)(∂v1/∂x)·dx + (∂f/∂v2)(∂v2/∂x)·dx

Dividing both sides by dx gives the multivariable chain rule that reverse-mode AD implements at every fan-out node:

∂f/∂x = (∂f/∂v1)(∂v1/∂x) + (∂f/∂v2)(∂v2/∂x)

In words: whenever a variable influences the output through more than one path, its total gradient is the sum of the contributions along every path, each contribution being a product of local derivatives along that path. Miss a path, and you get a wrong — specifically, an undercounted — gradient.

The payoff for organizing the computation this way: one forward pass plus one backward pass gives you the gradient with respect to every input simultaneously, regardless of how many inputs there are. The cost depends only on the size of the graph (roughly twice the cost of one forward evaluation), never on the number of inputs. This is precisely why training a network with a billion weights is affordable: reverse-mode AD is what practitioners call backpropagation — the two names refer to the same algorithm, backpropagation being reverse-mode AD applied specifically to the layered graphs of neural networks.

Fully worked example, traced step by step

Take f(x, y) = (x + y)·(x·y) at x = 3, y = 4.

Forward pass (compute values, left to right through the graph):

• v1 = x + y = 3 + 4 = 7
• v2 = x·y = 3·4 = 12
• f = v1·v2 = 7·12 = 84

Reverse pass (seed ∂f/∂f = 1, then walk backward, applying the product/sum rule at each node):

• f = v1·v2, so by the product rule ∂f/∂v1 = v2 = 12, and ∂f/∂v2 = v1 = 7.
• v1 = x + y, so the local derivatives are ∂v1/∂x = 1 and ∂v1/∂y = 1.
• v2 = x·y, so the local derivatives are ∂v2/∂x = y = 4 and ∂v2/∂y = x = 3.
• x reaches f through both v1 and v2, so its total gradient sums both paths: ∂f/∂x = (∂f/∂v1)(∂v1/∂x) + (∂f/∂v2)(∂v2/∂x) = (12)(1) + (7)(4) = 12 + 28 = 40.
• y reaches f through both v1 and v2 as well: ∂f/∂y = (12)(1) + (7)(3) = 12 + 21 = 33.

Verification against ordinary calculus. Expand the original function directly: f(x,y) = (x+y)(xy) = x²y + xy². Differentiating this expanded form the way you already know how: ∂f/∂x = 2xy + y² = 2(3)(4) + 4² = 24 + 16 = 40, and ∂f/∂y = x² + 2xy = 3² + 2(3)(4) = 9 + 24 = 33. Both match exactly. This is not a coincidence — it is the guarantee AD makes: it always agrees with ordinary symbolic calculus, to floating-point precision, because it applies exactly the same rules, just node by node instead of on one giant expression.

The diagram below shows this entire graph: forward values and local derivatives in black and blue, and the accumulated reverse-mode gradients in red at each node.

Building a tiny autodiff engine

Every idea above fits in a small Python class. Each Value remembers its own number, its accumulated gradient, and the list of children it was built from together with the local derivative for each child. backward() implements exactly the reverse pass derived above: add the incoming seed to this node's own gradient, then recurse into every child, passing along seed × local_derivative — this recursive call is precisely the sum-over-paths rule, because a node with two parents gets backward() called on it twice, and each call adds to its gradient rather than overwriting it.

class Value:
    def __init__(self, data, children=()):
        self.data = data
        self.grad = 0.0
        self.children = children  # tuple of (child_Value, local_derivative)

    def __add__(self, other):
        return Value(self.data + other.data,
                     children=((self, 1.0), (other, 1.0)))

    def __mul__(self, other):
        return Value(self.data * other.data,
                     children=((self, other.data), (other, self.data)))

    def backward(self, seed=1.0):
        self.grad += seed
        for child, local_derivative in self.children:
            child.backward(seed * local_derivative)


x = Value(3.0)
y = Value(4.0)
v1 = x + y          # v1.data = 7.0
v2 = x * y          # v2.data = 12.0
f  = v1 * v2         # f.data  = 84.0

f.backward()

print(f.data, x.grad, y.grad)
# 84.0 40.0 33.0

Trace it exactly as Python would: f.backward() calls backward(seed=1.0) on f, setting f.grad = 1.0, then loops over f's children, which multiplication stored as ((v1, v2.data), (v2, v1.data)) = ((v1, 12.0), (v2, 7.0)). The first child call is v1.backward(seed = 1.0 × 12.0 = 12.0), setting v1.grad = 12.0, and recursing into v1's children ((x, 1.0), (y, 1.0)), giving x.grad += 12.0 and y.grad += 12.0. The second child call from f is v2.backward(seed = 1.0 × 7.0 = 7.0), setting v2.grad = 7.0, and recursing into v2's children ((x, 4.0), (y, 3.0)) — note these are y.data and x.data respectively, from the multiplication rule — giving x.grad += 28.0 (now 12.0 + 28.0 = 40.0) and y.grad += 21.0 (now 12.0 + 21.0 = 33.0). The printed result is exactly 84.0 40.0 33.0, matching both the hand-worked graph and the direct calculus check above. This fifteen-line class, scaled up with more operations (subtraction, division, sin, exp, matrix multiplication) and wrapped around GPU-parallel arrays, is structurally the same idea inside PyTorch's autograd, TensorFlow's GradientTape, and JAX's grad — and the same idea behind small educational engines such as Andrej Karpathy's micrograd. None of them are doing anything mathematically different from what you just traced by hand.

It is worth also seeing numerical differentiation coded, so the contrast is concrete rather than just asserted:

def numerical_derivative(f, x, h=1e-3):
    return (f(x + h) - f(x)) / h

f = lambda x: x ** 2
print(numerical_derivative(f, 2.0))
# approximately 4.001 (the true derivative at x = 2 is exactly 4)

Compare: the autodiff engine above returned exactly 40.0 and 33.0 with no tunable step size and no approximation error, while numerical_derivative returns a number that is only close to the truth, and whose accuracy is fragile in ways an exact method never has to worry about.

A misconception worth naming and correcting

The single most common mistake students make when first learning this material is believing that automatic differentiation is another kind of approximation, like numerical differentiation with a very small step size h. This is false, and the distinction matters. Numerical differentiation has truncation error baked into its definition — the finite-difference formula (f(x+h)−f(x))/h is only an approximation of the limit as h → 0, and that error is present at any positive h, however small. Automatic differentiation has no such term anywhere in its derivation: at every single node it applies the exact sum rule, product rule, or chain rule from calculus, using values that were computed exactly during the forward pass. The only error present in an AD computation is ordinary floating-point rounding error — the same tiny error present in literally any computer arithmetic, including 2.0 + 2.0 in Python, and it does not grow the way finite-difference truncation error does as you shrink a step size. AD is exact calculus, mechanized; it is not calculus approximated.

A second, related trap: when a node has more than one downstream consumer — like x feeding both v1 and v2 in our example — it is tempting to compute the gradient contribution from just one path and stop, especially if you are tracing the graph by hand and the first path you find "closes the loop." The correct rule, derived above from the total differential, is that you must sum the contribution from every path from that node to the output. This is exactly why the reference implementation above uses self.grad += seed rather than self.grad = seed — using = instead of += is a genuine, common bug in from-scratch autodiff implementations, and it silently produces an undercounted gradient that is very hard to notice just by looking at the final trained model's behavior.

From this graph to training real neural networks

A neural network's training loop is, structurally, nothing more than an enormous version of the five-node graph worked through above. The "loss function" L takes the network's current weights and a batch of training examples and outputs one number measuring how wrong the predictions are. Reverse-mode AD runs one forward pass through the whole network to compute L, then one backward pass to get ∂L/∂w for every single weight w simultaneously — whether the network has 500 weights or 500 billion. Each weight is then updated by a small step opposite its gradient, w ← w − η·(∂L/∂w), where η (the learning rate) controls the step size; this update rule, gradient descent, is the subject of its own chapter, but notice it is entirely dependent on having the exact gradient available cheaply — which is precisely what this chapter's algorithm provides. Every layer of a real network (matrix multiplications, activation functions like ReLU or sigmoid, convolutions) is just another elementary operation with a known local derivative, slotted into a much larger version of the same DAG.

Exam relevance

• CBSE Class 12 Boards: the chain rule, product rule, and quotient rule (from the "Continuity and Differentiability" chapter) are exactly the tools applied at each node of the graph in this chapter — fluency with those rules by hand is a direct prerequisite for everything above.
• JEE Main / Advanced: these exams test the single-variable chain and product rules extensively (including implicit differentiation and related-rates problems), but do not formally test multivariable partial derivatives or automatic differentiation as a named topic — treat this chapter as depth beyond the syllabus that makes the syllabus's own rules click harder.
• Olympiad-style problems (KVPY-style rate-of-change questions): the "sum over multiple paths" idea derived here from the total differential is the same reasoning used in related-rates Olympiad problems where one quantity affects an outcome through two independent mechanisms at once.
• GATE-foundation / engineering coursework: both the newer GATE Data Science and AI paper and machine-learning electives in GATE Computer Science assume backpropagation and computational graphs as background; this chapter is that background, derived rather than asserted.

Summary

• Any program is a composition of elementary operations (+, −, ×, /, sin, exp, …), each with a simple, known derivative rule.
• Writing that composition as a directed acyclic graph — inputs, intermediate nodes, output — makes the structure of the computation explicit and reusable for differentiation.
• Numerical differentiation is approximate and needs one extra pass per input; symbolic differentiation is exact but its expression size can blow up exponentially through repeated composition. Automatic differentiation is exact and cheap because it works on the graph, computing each node once.
• Forward-mode AD carries a value's derivative alongside the value itself, but needs one full pass per input variable.
• Reverse-mode AD (backpropagation) runs one forward pass to compute values, then one backward pass, seeded with ∂f/∂f = 1, that computes the gradient with respect to every input in a single traversal — this is why it scales to networks with billions of parameters.
• At any node with more than one downstream consumer, the total gradient is the sum of the contributions from every path (the multivariable chain rule, derived here from the total differential) — accumulate with +=, never overwrite.
• AD's only error is ordinary floating-point rounding, not the truncation error inherent to finite differences — it is exact calculus, mechanized, not an approximation.

Check your understanding

• Q1. For g(x) = sin(x²), decompose it into a two-node graph (v1 = x·x, v2 = sin(v1)) and compute dg/dx at x = 1 using reverse-mode AD by hand. Verify your answer against the direct derivative d/dx[sin(x²)] = 2x·cos(x²).
• Q2. A loss function has 1,000 inputs and 1 output. Explain precisely why forward-mode AD needs 1,000 passes to get the full gradient while reverse-mode needs only 2 (one forward, one backward), and state which mode a real training loop uses and why.
• Q3. In a graph, node c is used by two other nodes, d and e. A student computes ∂f/∂c using only the path through d and ignores the path through e. Explain exactly what is mathematically wrong with this, and state the correct rule.
• Q4. For f(x, y) = x²y, decompose it as v1 = x·x and f = v1·y. At x = 2, y = 5, compute the forward values and the reverse-mode gradients ∂f/∂x and ∂f/∂y, being careful that v1 = x·x is itself a fan-out of x into both operand slots of the multiplication. Verify against the direct derivatives ∂f/∂x = 2xy and ∂f/∂y = x².

Answers.

• A1. Forward: v1 = 1·1 = 1, v2 = sin(1) ≈ 0.8415. Reverse: seed ∂g/∂v2 = 1; ∂v2/∂v1 = cos(v1) = cos(1) ≈ 0.5403, so ∂g/∂v1 ≈ 0.5403. Since v1 = x·x fans x into both multiplication slots, ∂v1/∂x = x + x = 1 + 1 = 2 (matching the known rule d(x²)/dx = 2x). So dg/dx ≈ 0.5403 × 2 = 1.0806. Direct check: 2(1)cos(1) ≈ 2 × 0.5403 = 1.0806. Matches.
• A2. Forward mode propagates the derivative with respect to only one chosen input per pass, so covering all 1,000 inputs needs 1,000 separate forward passes. Reverse mode computes values in one forward pass and then, in a single backward traversal, accumulates the gradient with respect to every input at once, because every node's backward call only depends on graph structure, not on which input you eventually care about. Real training loops use reverse mode (backpropagation) because loss functions have millions of inputs (weights) and effectively one output (the loss), which is exactly the case reverse mode is cheap for.
• A3. This is wrong because c influences f through both d and e simultaneously; the correct total derivative, from the multivariable chain rule, is ∂f/∂c = (∂f/∂d)(∂d/∂c) + (∂f/∂e)(∂e/∂c). Dropping the e-path silently discards part of c's real effect on f, giving an undercounted, incorrect gradient — which is why real implementations accumulate gradients with += rather than overwrite them with =.
• A4. Forward: v1 = 2 × 2 = 4, f = 4 × 5 = 20. Reverse: seed ∂f/∂f=1; ∂f/∂v1 = y = 5, ∂f/∂y = v1 = 4. v1 = x·x fans x into both slots: ∂v1/∂x = x + x = 4. So ∂f/∂x = (∂f/∂v1)(∂v1/∂x) = 5 × 4 = 20, and ∂f/∂y = 4. Direct check: ∂f/∂x = 2xy = 2(2)(5) = 20 ✓, ∂f/∂y = x² = 4 ✓. Both match.

Think About It

Think about this: How would you explain automatic differentiation and computational graphs to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where automatic differentiation and computational graphs is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting automatic differentiation and computational graphs to at least 3 other topics you have studied.