Information Geometry: Differential Geometry of Probability Families

Why "0.01" Doesn't Mean the Same Thing Everywhere

Two cricket analysts are each modelling a coin-toss-like event with a Bernoulli distribution — heads with probability θ. Analyst A revises her estimate of a fair-looking coin from θ = 0.50 to θ = 0.51. Analyst B revises his estimate of an almost-always-heads coin from θ = 0.98 to θ = 0.99. Both changed their number by exactly 0.01. Question: did they change their belief by the "same amount"?

Treated as points on a number line, yes — both moved 0.01 units. But think about what these numbers predict. Analyst A's two coins (fair and slightly-biased) behave almost identically over any short run of tosses; you would need a huge number of flips to statistically tell θ=0.50 from θ=0.51 apart. Analyst B's two coins behave very differently: one produces about 2 tails in 100 tosses, the other about 1 — a change that is comparatively easy to detect, because near certainty, small probability shifts translate into large shifts in what you actually expect to observe. The same 0.01 step in the parameter carries very different amounts of "distinguishing information" depending on where you take it.

This is the founding observation of information geometry: the space of probability distributions is not a flat, uniformly-ruled space like the real number line, even when we label its points with ordinary real-number parameters. It behaves like a curved surface, where the "true" distance between two nearby points depends on where on the surface you are standing. Building the correct ruler for this surface — and discovering that it is a genuine object of differential geometry, complete with a metric, tangent vectors, and geodesics — is exactly what this chapter does, using nothing beyond derivatives, integrals, and a bit of matrix algebra.

From Likelihood to a Local Ruler: the Score Function

Let P(x; θ) be a probability distribution over outcomes x, depending on a parameter θ. Define the log-likelihood ℓ(θ; x) = ln P(x; θ), and the score function:

s(θ; x) = ∂ℓ/∂θ = ∂/∂θ [ln P(x; θ)]

The score measures how sensitively the log-probability of a specific outcome x reacts to a nudge in θ. Before using it, we need one basic fact: averaged over outcomes drawn from P(·; θ) itself, the score has mean zero. Here is the one-line proof, and it is worth doing carefully because everything downstream depends on it. Since P(x; θ) is a probability distribution for every θ, it must sum to 1 for every θ:

Σ_x P(x; θ) = 1   for all θ

Differentiate both sides with respect to θ. On the right, the derivative of the constant 1 is 0. On the left, assuming we may differentiate term-by-term (true for the well-behaved families used here):

Σ_x ∂P/∂θ = 0

Now use the identity ∂P/∂θ = P · (∂lnP/∂θ) = P · s(θ;x), which is just the chain rule for d(ln u)/du = 1/u run backwards. Substituting:

Σ_x P(x; θ) · s(θ; x) = 0,  i.e.  Eθ[s(θ; X)] = 0

So the score has zero expectation at the true parameter value. But its spread around zero is not zero — and that spread is precisely what tells you how sharply the distribution changes shape as θ moves. That spread is the Fisher information.

Fisher Information: Two Formulas, One Number

The Fisher information I(θ) is defined as the variance of the score (its mean is already 0, so variance is just the expected square):

I(θ) = Eθ[ s(θ; X)² ]

There is a second, equivalent formula that is often faster to compute. Differentiate the zero-mean identity Σ_x P·s = 0 once more with respect to θ, using the product rule on each term (P·s):

Σ_x [ (∂P/∂θ)·s + P·(∂s/∂θ) ] = 0

Again substitute ∂P/∂θ = P·s:

Σ_x P·s² + Σ_x P·(∂s/∂θ) = 0
⇒  E[s²] = −E[∂s/∂θ] = −E[ℓ″(θ)]

So Fisher information equals either the expected squared score, or the negative expected curvature (second derivative) of the log-likelihood:

I(θ) = E[(ℓ′)²] = −E[ℓ″]

The second form has a clean reading: ℓ″(θ) measures how sharply peaked the log-likelihood is around its maximum. A sharply peaked (highly curved, large negative ℓ″) log-likelihood means the data pins down θ very precisely — high information. A flat log-likelihood means many values of θ explain the data almost equally well — low information. Both formulas must agree, since we just derived one from the other by differentiating the same identity twice — a good consistency check to run on any concrete example.

Worked Example: The Fisher Information of a Coin

For a single Bernoulli(θ) trial (X = 1 with probability θ, X = 0 with probability 1−θ), the log-likelihood is:

ℓ(θ; x) = x·lnθ + (1−x)·ln(1−θ)

Route 1 (second-derivative formula). Differentiate once:

ℓ′(θ) = x/θ − (1−x)/(1−θ)

Differentiate again:

ℓ″(θ) = −x/θ² − (1−x)/(1−θ)²

Take the negative expectation, using E[X] = θ and E[1−X] = 1−θ:

I(θ) = θ/θ² + (1−θ)/(1−θ)² = 1/θ + 1/(1−θ) = 1 / [θ(1−θ)]

Route 2 (squared-score formula), as a cross-check. The score only takes two possible values: s(1) = 1/θ (when x=1, which happens with probability θ), and s(0) = −1/(1−θ) (when x=0, probability 1−θ). So:

E[s²] = θ·(1/θ)² + (1−θ)·(1/(1−θ))² = 1/θ + 1/(1−θ) = 1/[θ(1−θ)]

Both routes give the identical result, exactly as the derivation in the previous section guarantees. This is the formula that quantifies our opening hook precisely. At θ=0.5, I(0.5) = 1/0.25 = 4. At θ=0.98, I(0.98) = 1/(0.98×0.02) = 1/0.0196 ≈ 51.0 — nearly thirteen times larger. A rough way to feel this: the standard error of a sample proportion after n flips is √(θ(1−θ)/n), so to reliably resolve a shift of size δ you roughly need n much bigger than θ(1−θ)/δ². For δ=0.01, that is about 2500 flips near θ=0.5 but only about 196 flips near θ=0.98 — a ratio of ~12.75, matching I(0.98)/I(0.5) ≈ 12.76. Bigger Fisher information means fewer samples needed to distinguish nearby distributions: the space is "stretched" there.

You can also verify I(θ) computationally by simulating the squared-score formula directly:

import random

def bernoulli_score(x, theta):
    return x/theta - (1-x)/(1-theta)

def monte_carlo_fisher(theta, n=200000):
    total = 0.0
    for _ in range(n):
        x = 1 if random.random() < theta else 0
        s = bernoulli_score(x, theta)
        total += s * s
    return total / n

print(monte_carlo_fisher(0.3))   # analytic value: 1/(0.3*0.7) = 4.7619...

Tracing it: each loop draws X=1 with probability theta (via random.random() < theta), computes the score at that outcome, and accumulates its square. Averaging over 200,000 draws gives a Monte Carlo estimate of E[s²] that, for large n, lands close to the exact value 1/(0.3×0.7) = 4.7619 — the law of large numbers converging the simulation onto the calculus.

The Real Payoff: KL Divergence Is (Locally) This Metric

Fisher information was defined from the log-likelihood, which looks like a tool for estimation, not geometry. The bridge to geometry runs through the Kullback–Leibler (KL) divergence, the standard way to measure how one distribution differs from another:

KL(Pθ || Pθ+ε) = Σ_x P(x;θ) · ln[ P(x;θ) / P(x;θ+ε) ]

KL divergence is not symmetric — KL(P||Q) generally differs from KL(Q||P) — so on its own it cannot be a distance in the geometric sense (a true metric needs symmetry). But watch what happens for a small step ε. Write the divergence as:

KL(θ, θ+ε) = Eθ[ln P(X;θ)] − Eθ[ln P(X;θ+ε)]

and Taylor-expand ln P(x; θ+ε) around ε=0 to second order in ε:

ln P(x;θ+ε) ≈ ln P(x;θ) + ε·ℓ′(θ;x) + (ε²/2)·ℓ″(θ;x)

Take E_θ of both sides. The zero-order term gives E[ln P(X;θ)]; the first-order term vanishes because E_θ[ℓ′] = 0, which we proved two sections ago; the second-order term gives (ε²/2)·E[ℓ″] = −(ε²/2)·I(θ). Substituting back:

KL(θ, θ+ε) ≈ (ε²/2) · I(θ)

This is the central result of information geometry. To leading order, the KL divergence between two nearby distributions is exactly a quadratic form in the parameter step, with the Fisher information as its coefficient. Notice that this leading term depends only on ε², not on the sign of ε — so even though KL divergence is globally asymmetric, its leading local behaviour is automatically symmetric in the direction of the step. The asymmetry of KL only shows up at third order and beyond (a subtlety explored in full information-geometry courses via "dual connections" — beyond our scope, but good to know it exists rather than assume the asymmetry vanishes completely). To the order that matters for measuring local distance, KL divergence behaves exactly like a squared-length, and I(θ) is exactly the coefficient that turns a parameter step into a squared statistical distance.

Building a Ruler for Probability Space: the Fisher–Rao Metric

This is where the vocabulary of differential geometry enters, and it maps onto what we've built term for term. The set of all distributions {P_θ : θ in some range} is called a statistical manifold — a smooth family of distributions labelled by continuously-varying coordinates θ. A small change dθ is a tangent vector at the point θ. The Fisher information I(θ) plays the role of a metric tensor: it converts a tangent vector into a squared length via the Fisher–Rao line element:

ds² = I(θ) · dθ²

This is precisely the KL-divergence result from the previous section, rewritten as a differential: ds² = 2·KL(θ, θ+dθ) in the infinitesimal limit. It is not an arbitrary choice of ruler bolted onto the parameter space — it is the ruler that the distributions themselves demand, forced on us by how distinguishable nearby distributions actually are.

Once you have ds² = I(θ)dθ², the distance between two distributions θ₁ and θ₂, measured along the family, is the arc length:

d(θ1, θ2) = ∫θ1θ2 √I(θ) dθ

This is a geodesic distance — the natural, curvature-respecting distance along the manifold, as opposed to the naive |θ₂ − θ₁| you'd get by pretending the space were flat.

Solving the Geodesic: the Bernoulli Family Is Secretly a Circular Arc

Let's actually solve this integral for the Bernoulli family, using I(θ) = 1/[θ(1−θ)]:

d(θ1, θ2) = ∫θ1θ2 dθ / √[θ(1−θ)]

Substitute θ = sin²φ, so dθ = 2 sinφ cosφ dφ, and √[θ(1−θ)] = √(sin²φ cos²φ) = sinφ cosφ (for φ in [0, π/2], where sin and cos are both non-negative). The integrand becomes:

dθ / √[θ(1−θ)] = (2 sinφ cosφ dφ) / (sinφ cosφ) = 2 dφ

Every trigonometric factor cancels exactly. Integrating gives d(θ₁,θ₂) = 2(φ₂ − φ₁), where φ = arcsin(√θ). Equivalently, in terms of θ directly:

d(θ1, θ2) = 2 | arcsin(√θ2) − arcsin(√θ1) |

You can check the antiderivative by differentiating back: d/dθ[2 arcsin(√θ)] = 2 · 1/√(1−θ) · 1/(2√θ) = 1/√[θ(1−θ)], which is exactly √I(θ), confirming the integral is correct.

What this means geometrically is remarkable: if you re-coordinate the Bernoulli family using φ = arcsin(√θ) instead of θ itself, the metric becomes ds = 2dφ — completely flat, no curvature left. The substitution θ=sin²φ is exactly the parametrization of a quarter-circle: the Bernoulli family, measured with its natural statistical ruler, is isometric to an arc of a circle. Equal steps in φ are equal statistical distances — but because θ=sin²φ is a nonlinear (compressing-near-the-ends) function of φ, equal steps in φ correspond to unequal, edge-clustered steps in the raw parameter θ. That is exactly the stretching we predicted from the coin-toss hook, now derived exactly rather than just estimated.

As a sanity check on the endpoints: d(0,1) = 2(π/2 − 0) = π ≈ 3.1416, a finite number, even though the Fisher information I(θ) itself blows up as θ→0 or 1 (since θ(1−θ)→0 in the denominator). This is not a contradiction: near the endpoints, √I(θ) grows like θ^−1/2, and ∫θ^−1/2dθ is a convergent (integrable) singularity — the same way a cusp can have infinite curvature at a point yet finite arc length running through it.

Beyond One Parameter: the Fisher Information Matrix

Most families you meet in statistics and machine learning have more than one parameter, e.g. Normal(μ, σ). The construction generalises directly: the score becomes a vector of partial derivatives, and Fisher information becomes a matrix:

Iij(θ) = E[ (∂ℓ/∂θi) · (∂ℓ/∂θj) ] = −E[ ∂²ℓ/∂θi∂θj ]

and the line element becomes the general quadratic form (a Riemannian metric in the full sense — this is the matrix/quadratic-form machinery from your linear algebra syllabus doing real work):

ds² = Σi,j Iij(θ) dθi dθj

Let's compute I for the Normal family with mean μ and standard deviation σ. The log-density is:

ℓ(μ,σ;x) = −lnσ − (1/2)ln(2π) − (x−μ)²/(2σ²)

μμ entry. ∂ℓ/∂μ = (x−μ)/σ². Differentiating again: ∂²ℓ/∂μ² = −1/σ², a constant, so I_μμ = 1/σ² directly.

σσ entry. ∂ℓ/∂σ = −1/σ + (x−μ)²/σ³. Differentiating again: ∂²ℓ/∂σ² = 1/σ² − 3(x−μ)²/σ⁴. Taking expectation and using E[(X−μ)²] = σ² (the definition of variance): E[∂²ℓ/∂σ²] = 1/σ² − 3σ²/σ⁴ = −2/σ², so I_σσ = 2/σ².

Cross term. ∂²ℓ/∂μ∂σ = ∂/∂σ[(x−μ)/σ²] = −2(x−μ)/σ³. Taking expectation and using E[X−μ]=0: I_μσ = 0.

So the Fisher information matrix for the Normal family, in (μ,σ) coordinates, is diagonal:

I(μ,σ) = [ 1/σ²      0    ]
              [   0     2/σ² ]

The line element is ds² = dμ²/σ² + 2dσ²/σ². Read this: near a small σ (a tight, confident distribution), the same numerical shift dμ or dσ produces a much larger statistical distance than near a large σ (a spread-out, uncertain distribution) — a two-parameter version of exactly the stretching we found for the coin. The zero off-diagonal entry says μ and σ directions are, at each point, geometrically perpendicular — a clean, checkable structural fact, not a coincidence: it follows from E[X−μ]=0, the same zero-mean-deviation fact used throughout.

Correcting a Persistent Misconception

The natural but wrong intuition is: "if two probability distributions have parameters that are close together, the distributions themselves must be similarly close, and if the parameters are far apart, the distributions must be far apart, in roughly the same proportion everywhere." Everything above refutes this precisely. |θ₂−θ₁| is not an invariant, meaningful measure of how different two distributions are — it is an artefact of which coordinate you happened to write the family in. Reparametrize the same Bernoulli family using φ=arcsin(√θ) instead of θ, and the raw-coordinate distance |φ₂−φ₁| changes to a completely different number, even though the underlying pair of distributions hasn't changed at all. What does not change under reparametrization is the Fisher–Rao geodesic distance d(θ₁,θ₂) computed from the metric — because that quantity is built from KL divergence, which only ever compares actual distributions, never the labels we chose for them. This reparametrization-invariance is the entire reason information geometry is worth building: it gives a notion of "how different are these two models" that survives no matter how you choose to write your parameters, which a naive coordinate distance never does.

Why This Matters: Natural Gradient Descent

This isn't only elegant mathematics — it changes how machine learning models are actually trained. When you train a model whose output is a probability distribution (logistic regression, a softmax classifier, a language model's next-token distribution), plain gradient descent updates parameters as Δθ = −η∇L, treating the parameter space as flat Euclidean space. But we've just shown the space of distributions is not flat in parameter coordinates — it's stretched exactly by I(θ). A fixed-size step Δθ near an over-confident region (small θ(1−θ), or small σ) moves you through a huge distance in distribution-space, risking wild overshoot and instability, while the identical step size in a low-information region barely moves the model at all, making learning sluggish there. Natural gradient descent corrects for this by preconditioning the update with the inverse Fisher information matrix:

Δθ = −η · I(θ)−1 · ∇L(θ)

This rescales the naive gradient so that each step moves a consistent distance through distribution-space rather than through parameter-space — exactly the correction our geodesic analysis says is needed. It's a direct, practical consequence: the same reasoning that told us the Bernoulli family curves near 0 and 1 is the reasoning that explains why classifiers become numerically unstable to train when they get very confident, and why an optimizer that respects the Fisher metric behaves better there than one that doesn't. (A full derivation of why the natural gradient is the steepest-descent direction under the Fisher-Rao metric uses the calculus of variations and is beyond this chapter's scope — but the intuition you now have, that I(θ) measures the true local "cost" of a parameter step, is exactly the idea the full derivation formalizes.)

Where This Meets Your Exams

None of CBSE, JEE, or BITSAT will ask you to state the Fisher–Rao metric by name. What they will test, repeatedly, is the exact toolkit this chapter forced you to use with real stakes attached: differentiating ln f(x) and compositions via the chain rule (used for every score-function derivative here), evaluating definite integrals via trigonometric substitution θ=sin²φ (the standard JEE substitution family for integrands with √[x(1−x)]-type expressions), and reading off structure from a symmetric matrix — diagonal entries, off-diagonal (cross) terms, and quadratic forms x^TAx — which is core Class 12 matrices content and a GATE-foundation staple. If you can redo the Bernoulli I(θ) derivation and the θ=sin²φ substitution from memory, you've practised exactly the mechanics these exams reward, on a problem substantially harder than what they'll actually ask.

Check Your Understanding

1. Compute: Find I(0.2) for a Bernoulli distribution using the formula I(θ)=1/[θ(1−θ)].
   Answer: 1/(0.2×0.8) = 1/0.16 = 6.25.
2. Compute: Find the Fisher–Rao distance between θ₁=0.1 and θ₂=0.9.
   Answer: Using d=2|arcsin(√0.9)−arcsin(√0.1)|: arcsin(√0.9)=arcsin(0.949)≈1.249 rad, arcsin(√0.1)=arcsin(0.316)≈0.322 rad. d ≈ 2(1.249−0.322) = 2(0.927) ≈ 1.85. (Compare: the naive parameter distance would say 0.8 — a different number, illustrating exactly why the naive measure isn't invariant.)
3. Conceptual: Why does a symmetric-looking quantity (KL divergence's leading-order term) emerge from an asymmetric one (KL divergence itself)?
   Answer: Because the leading term in the Taylor expansion is proportional to ε² (from the second derivative of the log-likelihood), and ε² is identical whether you step from θ to θ+ε or from θ+ε to θ (i.e. ε→−ε). Asymmetry can only enter through odd-order terms, which are higher order and vanish faster than the quadratic term as ε→0.
4. Derive: Using I_μμ=1/σ² from the Normal family, what happens to the "statistical cost" of a fixed shift dμ as σ→0?
   Answer: ds² = dμ²/σ² → ∞ as σ→0 for any fixed nonzero dμ — an infinitely confident (zero-variance) model is infinitely sensitive to any shift in its mean, matching the intuition that a near-certain prediction becomes trivially easy to falsify with even a tiny relocation.
5. True/False with justification: "If two Bernoulli parameters differ by the same |Δθ|, the corresponding distributions are always equally distinguishable."
   Answer: False. Distinguishability is governed by ∫√I(θ)dθ, not by |Δθ| itself; since I(θ)=1/[θ(1−θ)] is not constant, equal Δθ produces unequal statistical distance depending on where θ sits, as shown numerically for θ=0.5→0.51 versus θ=0.98→0.99.

Summary

A family of probability distributions indexed by parameters θ forms a statistical manifold. The score function ∂ℓ/∂θ has zero mean, and its variance — the Fisher information I(θ) — equals both E[(ℓ′)²] and −E[ℓ″], two formulas that must and do agree. Expanding KL divergence between nearby distributions to second order shows KL(θ,θ+ε) ≈ (ε²/2)I(θ), which means Fisher information is exactly the metric tensor of the manifold: ds²=I(θ)dθ². For the Bernoulli family this integrates, via the substitution θ=sin²φ, to the exact closed form d(θ₁,θ₂)=2|arcsin√θ₂−arcsin√θ₁|, revealing that the family is isometric to a circular arc — raw parameter spacing is misleading, but φ-spacing is the truth. For multi-parameter families like the Normal, Fisher information becomes a matrix, computed here explicitly as diag(1/σ², 2/σ²). The payoff is a distance between distributions that is invariant under reparametrization — unlike naive coordinate distance — and this same metric, used to precondition gradient descent as natural gradient, is a working tool in how real models are trained today.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where information geometry: differential geometry of probability families is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting information geometry: differential geometry of probability families to at least 3 other topics you have studied.