Energy-Based Models: Learning Probability through Energy Functions

A Different Question to Ask a Machine

Suppose you want a program to judge whether a handwritten digit actually looks like a "3," whether a sentence is grammatical Hindi, or whether a proposed chess move leaves the king safe. The obvious approach is to ask the program for a probability: "what is P(this is a valid 3)?" But probabilities are expensive. To output a genuine probability, a model has to know how that value compares to every other possible input it could have received, because probabilities must sum to 1 across all possibilities. For a 28-by-28 pixel image, "every other possible input" means every one of 2⁷⁸⁴ black-and-white pictures — a number with more digits than there are atoms in the observable universe.

Energy-based models (EBMs) sidestep this by asking a cheaper question. Instead of "what is the probability of x?", they ask "how much do I dislike x?" and answer with a single number called the energy, written E(x). A well-formed digit gets low energy. A scribble gets high energy. A grammatical sentence gets low energy. A word-salad gets high energy. Crucially, E(x) does not need to be normalized against every other possible x — you can build it, train it, and use it for comparison without ever computing a true probability. Later in this chapter you will see exactly how a probability distribution can be recovered from energy when you do need one, and why that recovery is often the hardest part of the whole framework.

Borrowing "Energy" from Physics, on Purpose

The word "energy" here is not a metaphor invented for machine learning — it is deliberately borrowed from statistical mechanics, and the borrowing is exact. You already know, from potential energy in mechanics, that a ball placed on a hilly surface rolls downhill and settles at a local minimum of potential energy. That resting point is a stable equilibrium: small nudges away from it cost energy to make, so the system prefers to stay there. Rolling downhill along the steepest direction is, mathematically, the same update rule as gradient descent, which you may already have met when training neural networks by minimizing a loss function. An EBM makes this literal: it builds a function E(x) over the space of possible inputs x (images, sentences, board positions, molecule structures — anything), shapes that function during training so that "good," realistic, or observed configurations sit in valleys, and lets everything else sit on the hillsides. Recognizing a good configuration becomes finding a nearby valley; generating a plausible new configuration becomes rolling a random starting point downhill until it settles.

This single idea — build a landscape, put the data at the bottom of the valleys — is powerful because you never had to specify what makes a valley "good" in words. You only need a function that scores compatibility, and a training procedure that carves the landscape so real data sits low. What is missing so far is the bridge back to probability: given an energy landscape, how likely is any particular point on it? That bridge is not arbitrary — it can be derived from a very short, very reasonable requirement, which is what the next section does properly rather than just asserting the standard formula.

Why Exponential? Deriving the Boltzmann Form

Here is the requirement: imagine two independent systems, A and B, that do not interact — say, the pixels in the top half of an image and the pixels in the bottom half, modeled as if they varied independently. Two facts about independent systems are non-negotiable:

• Energies add. If A is in configuration a with energy E_A(a), and B is in configuration b with energy E_B(b), and the two don't interact, the energy of the combined system is just E(a, b) = E_A(a) + E_B(b).
• Probabilities multiply. For genuinely independent systems, P(a and b) = P(a) · P(b).

Now suppose probability is determined entirely by energy through some single function f, so that P(a) = f(E_A(a)), P(b) = f(E_B(b)), and P(a,b) = f(E_A(a) + E_B(b)). Combining the two facts above forces:

f(E_A(a) + E_B(b)) = f(E_A(a)) · f(E_B(b))

for every possible pair of energy values. This is a functional equation of exactly the type you meet in Olympiad algebra: f(u + v) = f(u)·f(v) for all real u, v. Setting u = v = 0 gives f(0) = f(0)², so f(0) is 0 or 1; f(0)=0 would make every probability zero, which is useless, so f(0)=1. Checking that f(nu) = f(u)ⁿ for integers n and extending by continuity shows the only continuous solutions are exponential: f(u) = c^u for some constant c, which is usually rewritten as f(u) = e^−βu for some constant β. We want higher energy to mean lower probability — "I dislike this configuration" should translate to "this is unlikely" — so we need f to be decreasing, which pins down β > 0. There is no other well-behaved function satisfying "energies add, probabilities multiply": the exponential is not a convenient choice, it is the only choice. This is exactly the same argument that shows compound growth must be exponential once you require "growth over consecutive equal periods multiplies."

The Full Formula, and What Temperature Does

So the unnormalized weight of a configuration x must be proportional to e^−βE(x). To turn these weights into an honest probability distribution — one where every value is non-negative and all values sum to 1 — divide by the sum of the weights over every possible configuration:

p(x) = e^−βE(x) / Z,    where   Z = Σ_x′ e^−βE(x′)

This is the Boltzmann distribution (also called the Gibbs distribution), and Z is called the partition function — it just "partitions" the total probability mass of 1 across every configuration in proportion to e^−βE(x). In physics, β = 1/(kT), where k is Boltzmann's constant and T is temperature; in machine learning, people usually fold β into the energy function itself (redefine E to already include the β factor) and write the simpler p(x) = e^−E(x)/Z, which is the convention this chapter will use from here on.

The constant β (or equivalently "temperature") is worth understanding on its own, because it explains something important about how EBMs relate to plain optimization. As β → ∞ (temperature → 0), the gap between e^−βE(x) for the lowest-energy state and every other state grows without bound, so p(x) collapses onto the single global minimum of E — the distribution becomes a spike, and "sampling from the model" becomes identical to "solving the optimization problem: find x that minimizes E(x)." This is why classical optimization can be viewed as the zero-temperature limit of an energy-based model. As β → 0 (temperature → ∞), e^−βE(x) → 1 for every x regardless of its energy, so p(x) becomes uniform — the energy function stops mattering at all, and every configuration becomes equally likely. Real EBMs live between these extremes: energy differences matter, but they don't make anything strictly impossible.

Seeing It: An Energy Landscape and Its Probability Twin

The top curve is drawn the way a physical landscape looks — a deep dip means very low energy. The bottom curve is the ordinary way a probability distribution is drawn — a tall spike means high probability. The dashed guide lines show the exponential formula doing its job: whichever x sits lowest on the energy curve sits highest on the probability curve, directly underneath it. Curve shapes here are illustrative; the exact numbers are worked out by hand and by code in the next two sections.

Worked Example: Three Energies, by Hand

Take three configurations A, B, C with energies E(A) = 0, E(B) = 1, E(C) = 3 (in whatever unitless scale the model has learned, with β=1 already folded in). Compute the unnormalized weights first:

• e⁻⁰ = 1
• e⁻¹ ≈ 0.36788
• e⁻³ ≈ 0.04979

Sum them to get the partition function: Z = 1 + 0.36788 + 0.04979 = 1.41767. Now divide each weight by Z:

• p(A) = 1 / 1.41767 ≈ 0.7054
• p(B) = 0.36788 / 1.41767 ≈ 0.2595
• p(C) = 0.04979 / 1.41767 ≈ 0.0351

Check: 0.7054 + 0.2595 + 0.0351 = 1.0000, as required. Notice the ratio p(A)/p(B) = e⁻⁰/e⁻¹ = e ≈ 2.718 — exactly one factor of e, because A and B differ in energy by exactly 1 unit. This is a general fact worth keeping: the ratio of two probabilities never needs Z at all, since Z cancels in the division: p(x₁)/p(x₂) = e^−E(x₁)/e^−E(x₂) = e^{E(x₂) − E(x₁)}. You can always tell which of two configurations is more likely by comparing their energies directly — you only need Z when you want an absolute probability value, not a comparison.

Verifying by Code

The hand calculation above should match a direct computation. Here is a small, fully general function that takes any list of energies and returns Boltzmann probabilities:

import math

def boltzmann_probs(energies):
    weights = [math.exp(-e) for e in energies]
    Z = sum(weights)
    return [w / Z for w in weights], Z

probs, Z = boltzmann_probs([0, 1, 3])
for e, p in zip([0, 1, 3], probs):
    print(f"E={e}: p={p:.4f}")
print(f"Z={Z:.4f}")

Tracing this line by line: weights becomes [math.exp(0), math.exp(-1), math.exp(-3)] = [1.0, 0.36788..., 0.04979...]. Z sums these to 1.41767.... The list comprehension divides each weight by Z, giving [0.70539..., 0.25948..., 0.03512...]. The loop then prints, rounding to four decimal places:

E=0: p=0.7054
E=1: p=0.2595
E=3: p=0.0351
Z=1.4177

This matches the hand calculation exactly (small differences beyond the fourth decimal are just rounding). The point of writing boltzmann_probs as a reusable function rather than a one-off script is that the rest of this chapter reuses it conceptually: any energy function over any finite set of configurations turns into a probability distribution by exactly these three steps — exponentiate, sum, divide.

The Partition Function Problem

The three-state and two-bit examples in this chapter enumerate every configuration and sum directly, which works because there are only a handful of them. Real problems do not cooperate. A 28×28 black-and-white image has 2⁷⁸⁴ possible configurations — computing Z would mean evaluating the energy function on 2⁷⁸⁴ images and adding up the results, which is not merely slow, it is physically impossible: no computer built or conceivable could finish that sum before the universe's stars burn out. A model over 20-character passwords drawn from 62 possible characters has 62²⁰ configurations, a number with 36 digits. This combinatorial explosion — the same kind you meet when counting permutations and combinations — is why Z is called intractable for any realistic EBM, and it is the central practical difficulty of the entire framework.

The escape route is the fact already proven in the worked example above: comparing two configurations, or ranking many configurations, or generating new configurations by rolling downhill on the energy surface, never requires Z, because Z cancels out of every ratio and drops out of every gradient of E itself. Practical systems built on EBMs are carefully designed to only ever need such ratios or gradients — never the actual normalized probability value — which is exactly the trick that makes the framework usable despite an unusable Z.

A Tiny Energy-Based Model over Two Bits

To see energy applied the way a trainable model would use it — not just handed a list of energies, but computing them from weighted interactions between variables — consider two binary variables x₁, x₂ ∈ {0, 1}, and an energy function with a shape borrowed directly from Hopfield networks (covered later in this chapter):

import itertools
import math

def energy(x1, x2, w1=1, w2=2, w12=3):
    return -(w1 * x1 + w2 * x2 + w12 * x1 * x2)

states = list(itertools.product([0, 1], repeat=2))
energies = {s: energy(*s) for s in states}
Z = sum(math.exp(-e) for e in energies.values())

for s in states:
    p = math.exp(-energies[s]) / Z
    print(s, "energy =", energies[s], "prob =", round(p, 4))
print("Z =", round(Z, 3))

Tracing it: itertools.product([0,1], repeat=2) generates (0,0), (0,1), (1,0), (1,1) in that order. The energy function negates a weighted sum, so a configuration that makes the weighted sum large and positive ends up with energy that is large and negative — that is, very low energy, very high probability. Evaluating:

• E(0,0) = −(1·0 + 2·0 + 3·0·0) = 0
• E(0,1) = −(1·0 + 2·1 + 3·0·1) = −2
• E(1,0) = −(1·1 + 2·0 + 3·1·0) = −1
• E(1,1) = −(1·1 + 2·1 + 3·1·1) = −6

The w12 = 3 term only switches on when both bits are 1, so (1,1) gets an extra bonus on top of its individual bits' contributions, dragging its energy down to −6 — by far the lowest, meaning the model has effectively "memorized" that x₁ and x₂ like to be 1 together. Z = e⁰ + e² + e¹ + e⁶ = 1 + 7.389 + 2.718 + 403.429 = 414.536. Dividing each weight by Z and rounding to four places gives the printed output:

(0, 0) energy = 0 prob = 0.0024
(0, 1) energy = -2 prob = 0.0178
(1, 0) energy = -1 prob = 0.0066
(1, 1) energy = -6 prob = 0.9732
Z = 414.536

The model assigns 97.3% of all probability mass to the single pattern it "likes" — a direct, tiny demonstration of the same associative-memory behaviour that makes Hopfield networks and modern EBM-based generative models useful: strong co-occurring evidence collapses the distribution onto the configurations consistent with it.

Training an EBM: Carving the Valley at the Data

Everything so far assumed E(x) was already given. Training means adjusting E's parameters θ (the weights, like w1, w2, w12 above) so that real, observed data ends up at low energy. The natural objective is to maximize log p(x_data; θ) for the training examples. Starting from the definition and differentiating with respect to θ is where the calculus actually earns its keep:

log p(x; θ) = −E(x; θ) − log Z(θ)

Differentiate both terms. The first term's derivative is simply −∂E(x)/∂θ. The second term needs the chain rule on the logarithm, then substituting the definition of Z:

∂/∂θ log Z(θ) = (1/Z) · ∂Z/∂θ = (1/Z) Σ_x′ e^−E(x′) · (−∂E(x′)/∂θ)

Since e^−E(x′)/Z is exactly p(x′; θ) by definition, this sum is nothing but an average, weighted by the model's own current distribution:

∂/∂θ log Z(θ) = −Σ_x′ p(x′; θ) · ∂E(x′)/∂θ = −E_x′~pθ[∂E(x′)/∂θ]

Putting the two derivative terms back together, the sign flips cancel neatly, leaving:

∂/∂θ log p(x; θ) = −∂E(x)/∂θ + E_x′~pθ[∂E(x′)/∂θ]

Read this the way a gradient-ascent update reads it. The first term pushes θ in the direction that lowers E at the real training example x — carving a valley exactly where the data sits. The second term pushes θ in the direction that raises the average energy over configurations the model itself currently finds plausible — flattening out any valleys the model has carved in the wrong places. This is the well-known "positive phase, negative phase" structure behind contrastive divergence, the classic algorithm used to train Boltzmann machines: pull down on the data, push up on the model's own guesses, repeat. Note what did not appear anywhere in the final gradient: Z itself. The intractable partition function only ever enters as an expectation E_x′~pθ[·], and that expectation can be approximated by drawing a handful of sample configurations from the current model (via short runs of stochastic hill-descending, i.e. Gibbs sampling or Langevin dynamics) rather than summing over every configuration that exists. This is precisely how EBMs are trained in practice without ever computing Z explicitly.

A Common Misconception

It is tempting to think the "energy" in an energy-based model measures something physical — joules, calories, some quantity with units that could in principle be measured with an instrument. It does not. E(x) is an arbitrary learned scalar function with no intrinsic units; the name and the exponential formula are borrowed from statistical mechanics because the mathematics is identical, not because the model is doing physics. A second, equally common confusion is treating the energy function as if it were the training loss. It is not: the energy function E(x; θ) is the model — it defines the probability distribution through the Boltzmann formula. The loss function (negative log-likelihood, contrastive divergence, score matching, and others) is separate machinery used to adjust E's parameters so that low energy lines up with real data. Confusing the two leads to the wrong intuition that "minimizing the energy of a training batch" and "training the model" are the same instruction, when actually — as the gradient derived above shows — correct training requires simultaneously lowering energy at the data and raising it everywhere the model currently overestimates probability; lowering energy at the data alone, with nothing pushing back up elsewhere, simply drives every energy value toward negative infinity without ever producing a meaningful distribution.

Where Energy-Based Models Actually Show Up

The Hopfield network used as this chapter's running example is not a toy invented for teaching — it is a real, historically important architecture. John Hopfield introduced it in 1982 as a model of associative memory, using exactly the energy function shape in the code above (generalized to many neurons): E(s) = −½ Σ_i≠j w_ij s_i s_j, with binary neurons s_i. Update neurons one at a time, each always moving to whichever value lowers E, and the network provably settles into a local energy minimum — one of the patterns it was trained (via its weights) to "remember." In 2024, Hopfield shared the Nobel Prize in Physics with Geoffrey Hinton specifically for this foundational energy-based approach to neural networks. Hinton's own major contribution, the Restricted Boltzmann Machine, is a direct descendant: an EBM with a layer of visible units and a layer of hidden units, trained with contrastive divergence, historically used to pretrain early deep networks layer by layer before end-to-end backpropagation became practical at scale.

The framework's influence continues in current generative modeling. Score-based and diffusion models — the family behind most modern image generators — learn a function called the "score," ∇_x log p(x), and by the Boltzmann formula ∇_x log p(x) = −∇_xE(x) exactly, so learning the score is mathematically equivalent to learning the gradient of an energy function, sidestepping the need to ever compute Z while still capturing the same underlying energy landscape. Yann LeCun, a pioneer of convolutional networks, has long argued for treating energy-based models as a general lens for machine learning: classification, generation, and prediction can all be framed as different questions asked of the same underlying energy function — "which label makes this energy lowest?" versus "which image makes this energy lowest given this label?" — rather than as fundamentally different kinds of models requiring separate machinery.

Exam Connections

The exponential-of-negative-energy form is not confined to computer science electives — it is one of the more quietly recurring formulas across the CBSE senior-secondary science curriculum, which makes it worth flagging explicitly. Class 11 Physics' kinetic theory of gases derives the Maxwell–Boltzmann speed distribution, whose exponential factor e^{−mv²/(2kT)} is the Boltzmann distribution applied to kinetic energy of gas molecules. Class 12 Chemistry's Chemical Kinetics chapter introduces the Arrhenius equation, k = A·e^−E_a/(RT), where the activation energy E_a plays exactly the role of E(x) here: a larger E_a makes the exponential factor smaller, which makes the rate constant k smaller — higher energy, lower likelihood, the identical logic this chapter derived from the additivity argument. For JEE and Olympiad preparation specifically, the derivation in "Why Exponential?" above is a genuine instance of a Cauchy-type functional equation (f(u+v) = f(u)f(v)), a recurring pattern in Olympiad algebra problems, and recognizing it here is good practice for recognizing it elsewhere. Finally, the combinatorial explosion behind an intractable partition function — 2⁷⁸⁴ configurations for a small image — is the same counting intuition tested in permutations-and-combinations problems, just applied at a scale large enough to be practically, not just theoretically, significant.

Summary

• An energy-based model scores any configuration x with a single scalar E(x); low energy means "preferred" or "likely," borrowing the exact mathematics of potential energy and equilibrium from physics.
• Requiring that energies of independent systems add while probabilities multiply forces the conversion from energy to probability to be exponential — p(x) ∝ e^−E(x) is not a convenient guess, it is the only function satisfying that requirement.
• The full formula is p(x) = e^−E(x)/Z, the Boltzmann distribution, where Z = Σ_x′ e^−E(x′) is the partition function that makes probabilities sum to 1.
• Z is intractable for any realistic input space because it requires summing over every possible configuration — but ratios of probabilities, and gradients used for training, never need Z, because it cancels out.
• Training maximizes log-likelihood by lowering energy at real data (the positive phase) while raising average energy over the model's own current samples (the negative phase) — the derivation shows Z drops out of the gradient entirely.
• Hopfield networks (Nobel Prize in Physics, 2024), Restricted Boltzmann Machines, and the score functions inside modern diffusion-based image generators are all instances or direct descendants of this same energy-to-probability framework.

Check Your Understanding

1. Two states have energies E = 0 and E = 2. Compute Z and both probabilities by hand, and check that they sum to 1.
2. Explain, without recomputing Z, why increasing the energy of one state always decreases that state's probability, regardless of what the other states' energies are.
3. In the two-bit code example, if w12 is changed from 3 to −3 (keeping w1=1, w2=2), recompute all four energies. Which state now has the lowest energy, and why does that make sense given what the weight change means?
4. True or false, with justification: you must compute Z before you can determine which of two configurations is more probable.
5. The Arrhenius equation is k = A·e^−E_a/(RT). Which quantity is playing the role of E(x) from this chapter, and does a larger activation energy correspond to a faster or slower reaction? Explain using the "higher energy, lower probability" logic developed above.

Answers: (1) Z = e⁰ + e⁻² = 1 + 0.1353 = 1.1353; p(E=0) = 1/1.1353 ≈ 0.8808; p(E=2) = 0.1353/1.1353 ≈ 0.1192; sum ≈ 1.0000. (2) Because p(x₁)/p(x₂) = e^{E(x₂)−E(x₁)} depends only on the difference between the two energies, not on Z or on any other state's energy; raising E(x₁) always shrinks this ratio, and since every state's probability is that state's weight divided by the same shared Z, a smaller weight always means a smaller probability regardless of what else is in the sum. (3) E(0,0)=0, E(0,1)=−2, E(1,0)=−1, E(1,1)=−(1+2−3)=0; the lowest is now (0,1) at −2. This makes sense because a negative w12 now penalizes both bits being 1 together, removing (1,1)'s previous advantage and leaving (0,1) — which relies only on the individually strong w2=2 weight — as the preferred pattern. (4) False: the ratio p(x₁)/p(x₂) = e^{E(x₂)−E(x₁)} never involves Z, since it cancels in the division; Z is only needed to know the absolute value of a probability, not to rank two configurations against each other. (5) E_a plays the role of E(x). A larger E_a makes e^−E_a/(RT) smaller, so k is smaller and the reaction is slower — exactly matching "higher energy means lower probability (here, lower rate)," the same exponential logic derived for the Boltzmann distribution.