Logistic Regression: The Foundation of Classification

Why Linear Regression Breaks When the Answer Is Yes/No

Here are seven students from a CBSE Class 10 mock Math test, recorded as (hours studied, result):

(2, Fail), (3, Fail), (4, Fail), (5, Pass), (6, Pass), (7, Pass), (8, Pass)

Write Pass as 1 and Fail as 0. This looks like a job for regression — you have a number (hours) and you want to predict an outcome. So try the tool you already know: fit a straight line y = mx + c through the data using ordinary least squares, treating "1" and "0" as if they were just numbers to be predicted.

Do the calculation properly instead of guessing. The mean of x is x̄ = (2+3+4+5+6+7+8)/7 = 5, and the mean of y is ȳ = 4/7 ≈ 0.5714. The least-squares slope is m = Σ(x−x̄)(y−ȳ) / Σ(x−x̄)². Computing the numerator term by term: (−3)(−0.5714) + (−2)(−0.5714) + (−1)(−0.5714) + (0)(0.4286) + (1)(0.4286) + (2)(0.4286) + (3)(0.4286) = 1.714 + 1.143 + 0.571 + 0 + 0.429 + 0.857 + 1.286 = 6.0 exactly. The denominator is 9+4+1+0+1+4+9 = 28. So m = 6/28 = 3/14 ≈ 0.2143. The intercept is c = ȳ − m·x̄ = 4/7 − (3/14)(5) = 8/14 − 15/14 = −7/14 = −0.5 exactly.

So the best-fit line is y = 0.2143x − 0.5. Now use it to predict. A student who studied 0 hours: y = −0.5. A "probability" of negative one-half makes no sense. A student who studied 10 hours: y = 0.2143(10) − 0.5 ≈ 1.643, a 164% chance of passing — also meaningless. The line itself is not wrong as arithmetic; it is the wrong *shape* for the job. A straight line's range is all of ℝ, but a probability must live strictly inside [0, 1]. No choice of m and c can fix this, because whatever line you draw, it eventually leaves the [0,1] band on one side or the other. We need a function that takes any real number as input and always outputs something between 0 and 1. That function is the foundation of this entire chapter.

Squashing the Line: The Sigmoid Function

Define the sigmoid (logistic) function:

σ(z) = 1 / (1 + e^−z)

Check its behaviour at the extremes. As z → +∞, e^−z → 0, so σ(z) → 1/(1+0) = 1. As z → −∞, e^−z → ∞, so σ(z) → 1/∞ = 0. For any finite z, e^−z is strictly positive, so the denominator is strictly greater than 1, which forces 0 < σ(z) < 1 always — the curve gets arbitrarily close to 0 and 1 but mathematically never touches either. At z = 0: σ(0) = 1/(1+e⁰) = 1/(1+1) = 0.5. So z = 0 always maps to a 50% probability, which is exactly the "undecided" point.

Now derive its slope, because we will need it for training. Write σ(z) = (1+e^−z)⁻¹ and differentiate using the chain rule:

σ′(z) = −1·(1+e^−z)⁻² · (−e^−z) = e^−z / (1+e^−z)²

Split this into two factors: e^−z/(1+e^−z)² = [1/(1+e^−z)] · [e^−z/(1+e^−z)]. The first bracket is just σ(z). For the second, notice e^−z/(1+e^−z) = (1+e^−z−1)/(1+e^−z) = 1 − 1/(1+e^−z) = 1 − σ(z). So:

σ′(z) = σ(z)·(1 − σ(z))

This is a remarkably tidy result: the slope of the sigmoid at any point is computable purely from the sigmoid's own output there, with no separate exponential to recompute. It's also always positive (since 0<σ<1 makes both factors positive), confirming the curve is strictly increasing — more input never decreases the output probability. Keep this formula close; it does the heavy lifting when we compute gradients later.

The Logistic Regression Model

Logistic regression combines the two ideas: first compute a linear score, exactly as in linear regression, then squash it through σ.

z = w·x + b

p = σ(z) = P(y = 1 | x)

Here w (weight) and b (bias) are the parameters to be learned — precisely the m and c you know from linear regression, just renamed. w controls how strongly x pushes the score up or down; b shifts the whole score left or right. Predicting the class is a second, separate step: pick a threshold (usually 0.5) and declare ŷ = 1 if p ≥ 0.5, else ŷ = 0. Everything up to computing p is a genuine regression on a continuous quantity; only the final thresholding step turns it into classification.

Why It's Still Called "Regression": The Log-Odds

This isn't just a naming leftover — logistic regression really does fit a straight line, just not to p itself. Start from p = 1/(1+e^−z) and compute 1−p:

1 − p = [(1+e^−z) − 1] / (1+e^−z) = e^−z / (1+e^−z)

Now divide p by (1−p):

p/(1−p) = [1/(1+e^−z)] ÷ [e^−z/(1+e^−z)] = 1/e^−z = e^z

The quantity p/(1−p) is called the odds (familiar from cricket betting lines — "3 to 1 odds" means p/(1−p) = 3). We've just shown odds = e^z. Take the natural log of both sides:

ln(p/(1−p)) = z = w·x + b

The left side is called the logit, or log-odds. So logistic regression fits a straight-line equation to the log-odds of the outcome — a genuine linear regression happening on a transformed scale. The sigmoid is simply the algebra needed to convert that linear log-odds value back into a bounded probability. This is the precise, correct answer to "why is it called regression if it does classification?"

Worked Example: Predictions Before Any Training

Return to the 7-student dataset. Before training, initialize the parameters at w = 0, b = 0 (a standard, neutral starting point). Then for every student, z = 0·x + 0 = 0, so p = σ(0) = 0.5 for all seven — the untrained model is exactly as confident as a coin flip, regardless of hours studied. That's expected: with w = 0, x has no influence on z at all.

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

hours  = [2, 3, 4, 5, 6, 7, 8]
passed = [0, 0, 0, 1, 1, 1, 1]

w, b = 0.0, 0.0
predictions = [sigmoid(w * x + b) for x in hours]
print(predictions)
# [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]

Tracing this by hand: for every x, w*x + b evaluates to 0.0*x + 0.0 = 0.0, and sigmoid(0.0) = 1/(1+math.exp(-0.0)) = 1/(1+1.0) = 0.5. The list comprehension therefore produces seven copies of 0.5, matching the printed output exactly.

Measuring Wrongness: Why Not Mean Squared Error?

In linear regression you minimize mean squared error (MSE): the average of (prediction − actual)². It's tempting to reuse it here: average of (p_i − y_i)². The problem is not that MSE gives a wrong number — it's that when p = σ(w·x+b), the MSE surface as a function of w and b is generally non-convex: it can have multiple valleys (local minima), so gradient descent can get stuck somewhere that isn't the best possible fit, with no guarantee of reaching it. We need a loss function that stays convex — bowl-shaped, with one global minimum — even after the sigmoid squashing.

The right loss comes from a completely different idea: maximum likelihood estimation (MLE). Instead of asking "how far is the prediction from the label," ask "how probable is the data we actually observed, given these parameters?" and pick the parameters that make the observed data most probable.

Each label y_i is treated as a coin flip with bias p_i: P(y_i=1) = p_i and P(y_i=0) = 1−p_i. Both cases are captured in one compact expression: P(y_i) = p_i^y_i(1−p_i)^1−y_i — when y_i=1 the second factor becomes (·)⁰=1, leaving p_i; when y_i=0 the first factor becomes 1, leaving 1−p_i. Assuming the students' outcomes are independent, the likelihood of the whole dataset is the product:

L(w,b) = Π_i p_i^y_i(1−p_i)^1−y_i

Products of many small probabilities underflow numerically and are hard to differentiate, so take the log (log turns products into sums and is monotonically increasing, so maximizing log L is the same as maximizing L):

ln L = Σ_i [y_i ln p_i + (1−y_i) ln(1−p_i)]

Maximizing ln L is equivalent to minimizing its negative. Averaging over n examples gives the binary cross-entropy loss, the actual quantity logistic regression minimizes:

L = −(1/n) Σ_i [y_i ln p_i + (1−y_i) ln(1−p_i)]

Sanity-check this on the untrained model above, where every p_i = 0.5 regardless of y_i. Each individual term becomes −[y·ln(0.5) + (1−y)·ln(0.5)] = −ln(0.5), since ln(0.5) is common to both branches whatever y is. And −ln(0.5) = ln(2) ≈ 0.6931. So every one of the 7 terms equals 0.6931, and the average loss is exactly ln(2) ≈ 0.693. This is not a coincidence specific to this dataset — a model predicting p=0.5 for everything always has cross-entropy loss ln(2) on any binary-labeled data. It's a standard sanity check: if your loss doesn't start near ln(2) at initialization (for a roughly balanced dataset), something in your code is already wrong before training even begins.

The Gradient: An Unexpectedly Clean Result

To run gradient descent we need ∂L/∂w and ∂L/∂b. Work through the chain rule for a single example first, then average. Write L_i = −[y ln p + (1−y) ln(1−p)] where p = σ(z) and z = wx+b.

Step 1 — differentiate L_i with respect to p:

∂L_i/∂p = −y/p + (1−y)/(1−p) = [−y(1−p) + (1−y)p] / [p(1−p)] = (p−y) / [p(1−p)]

Step 2 — differentiate p with respect to z. This is exactly the sigmoid derivative proved earlier: ∂p/∂z = p(1−p).

Step 3 — chain them together:

∂L_i/∂z = ∂L_i/∂p · ∂p/∂z = [(p−y)/(p(1−p))] · p(1−p) = p − y

The p(1−p) terms cancel exactly, leaving the strikingly simple result ∂L_i/∂z = p − y: the gradient with respect to the linear score is just the prediction error. Finally, since z = wx+b, ∂z/∂w = x and ∂z/∂b = 1, giving:

∂L_i/∂w = (p_i − y_i)·x_i    and    ∂L_i/∂b = (p_i − y_i)

Averaged over the dataset: ∂L/∂w = (1/n)Σ(p_i−y_i)x_i, ∂L/∂b = (1/n)Σ(p_i−y_i). This cancellation is exactly why cross-entropy was paired with sigmoid in the first place: it produces a gradient that is just "error × input," identical in form to linear regression's gradient, while the loss surface itself remains convex.

Worked Example: One Step of Gradient Descent by Hand

Continue the dataset with w=0, b=0, where every p_i = 0.5. Compute (p_i − y_i)·x_i for each of the 7 points:

(0.5−0)·2=1.0, (0.5−0)·3=1.5, (0.5−0)·4=2.0, (0.5−1)·5=−2.5, (0.5−1)·6=−3.0, (0.5−1)·7=−3.5, (0.5−1)·8=−4.0

Sum = 1.0+1.5+2.0−2.5−3.0−3.5−4.0 = −8.5. Divide by n=7: ∂L/∂w = −8.5/7 ≈ −1.2143. For the bias, (p_i−y_i) sums to 0.5+0.5+0.5−0.5−0.5−0.5−0.5 = −0.5, giving ∂L/∂b = −0.5/7 ≈ −0.0714.

n = len(hours)
dw = sum((p - y) * x for p, y, x in zip(predictions, passed, hours)) / n
db = sum((p - y) for p, y in zip(predictions, passed)) / n
print(round(dw, 4), round(db, 4))
# -1.2143 -0.0714

alpha = 0.1
w = w - alpha * dw
b = b - alpha * db
print(round(w, 4), round(b, 4))
# 0.1214 0.0071

The negative gradient means the loss decreases as w increases — sensible, because the y=1 students (5,6,7,8 hours) sit at larger x than the y=0 students (2,3,4 hours), so a positive slope reduces error on the majority-weighted side. With learning rate α=0.1, gradient descent updates w := w − α·∂L/∂w = 0 − 0.1·(−1.2143) = 0.1214, and b := 0 − 0.1·(−0.0714) = 0.0071. After a single step, the model already leans toward "more hours → higher pass probability" — exactly the pattern in the data. Repeating this update thousands of times (in practice, with a computer) is what "training" a logistic regression model means.

The Decision Boundary

Once trained, a prediction is made by thresholding p at 0.5. Since σ(z)=0.5 exactly when z=0, the boundary between predicted classes is the set of points where w·x+b = 0. Solve for x: x = −b/w. In one feature this boundary is a single point on the number line; with two features (x₁, x₂) it's the straight line w₁x₁+w₂x₂+b=0; with more features it's a flat hyperplane. In every case it is linear — a straight cut through feature space — never curved, regardless of how many features you use.

To see this concretely, imagine gradient descent has run for many iterations (far beyond the single step above) and converged to roughly w ≈ 0.9, b ≈ −4.2. The decision boundary sits at x = −b/w = 4.2/0.9 ≈ 4.67 hours — students predicted to study more than about 4 hours 40 minutes are classified "Pass," matching the data's actual pass/fail split at 4–5 hours almost exactly.

w, b = 0.9, -4.2
x_boundary = -b / w
print(round(x_boundary, 2))
# 4.67

Common Misconception: "The Boundary Must Be Curved"

Because the sigmoid graph is visibly an S-shaped curve, many students assume the region separating "Pass" from "Fail" in feature space must also be curved. This is false, and the diagram above shows exactly why. The curve you see is a plot of probability p against the score z — that curve bends because σ is nonlinear in z. But the boundary is not drawn in (z, p) space; it's drawn in the original feature space (hours studied), and it is defined by the equation w·x+b = 0, which is linear by construction. In the diagram, the boundary is the single vertical dashed line at x ≈ 4.67 — a point, not a curve, because there is only one feature. With two features it would be a straight line; with three, a flat plane. The curviness of σ only affects how confidently the model predicts as you move away from the boundary — it never bends the boundary itself. This is also why logistic regression is called a "linear classifier": the decision rule w·x+b ≥ 0 is linear even though the probability estimate is not.

Beyond Two Classes: A Note on Softmax

Real classification problems often have more than two outcomes — for instance, classifying a handwritten digit as 0 through 9. Binary logistic regression generalizes to this case through the softmax function, which computes one linear score z_k = w_k·x + b_k per class k, then converts the whole vector of scores into a probability distribution: P(y=k) = e^z_k / Σ_j e^z_j. Setting the number of classes to 2 and fixing one class's score at 0 recovers the sigmoid exactly, so binary logistic regression is the special case of softmax regression with two classes — not a different algorithm. The loss generalizes the same way, from binary cross-entropy to categorical cross-entropy. This chapter focuses on the two-class case because every idea here — the linear score, the convex loss, the p−y gradient, the linear boundary — carries over unchanged; softmax just repeats it once per class.

Where This Shows Up in Your Exams

Logistic regression itself sits in the CBSE Class 11–12 Artificial Intelligence curriculum (Code 843) as the standard first classification algorithm taught after linear regression, and it appears in the same role in GATE's Data Science and AI paper. But the mathematics you just used is pure Class 11–12 calculus: differentiating e^−z and ln(x) via the chain rule is precisely the skill tested in JEE Main's Application of Derivatives and Continuity & Differentiability chapters, and recognizing that a function is monotonically increasing because its derivative is always positive (as we showed for σ) is a standard JEE technique for proving injectivity. The convexity argument — why cross-entropy is preferred over squared error here — echoes the second-derivative test used in JEE optimization problems, just applied to a loss surface instead of a single-variable function. Practicing the algebra in this chapter (odds, log-odds, chain-rule gradients) is directly transferable exam preparation, not a detour from it.

Summary

• Linear regression cannot model probabilities directly because its output range is unbounded, while probability must lie in [0,1] — demonstrated here with an actual fitted line producing a −0.5 "probability."
• The sigmoid σ(z) = 1/(1+e^−z) squashes any real number into (0,1), with σ(0)=0.5 and derivative σ′(z)=σ(z)(1−σ(z)).
• Logistic regression computes z=w·x+b (exactly like linear regression) and then p=σ(z); classification is a separate thresholding step on p.
• ln(p/(1−p)) = z, so the model is a genuine linear regression on the log-odds — the source of its name.
• Cross-entropy loss, derived from maximum likelihood, is used instead of MSE because it keeps the loss surface convex when paired with sigmoid.
• The gradient of cross-entropy loss with respect to the score is simply (prediction − actual): ∂L/∂z = p−y, because the σ′ term cancels algebraically.
• The decision boundary w·x+b=0 is always linear (a point, line, or hyperplane) no matter how curved the probability surface looks.
• Softmax is the direct multi-class generalization of the same sigmoid-plus-cross-entropy machinery.

Test Your Understanding

1. Algebraically show that σ(−z) = 1 − σ(z), starting from σ(z) = 1/(1+e^−z).
2. Using the illustrative trained model w=0.9, b=−4.2, compute p for a student who studied 3 hours and for one who studied 9 hours. (Hint: both values already appear as points on the plotted curve.)
3. We derived ∂L_i/∂w = (p−y)x from the chain rule ∂L/∂p · ∂p/∂z · ∂z/∂w. Write out the analogous three-step chain rule for ∂L_i/∂b and confirm it simplifies to (p−y).
4. A classmate says: "Since σ(z) never actually reaches exactly 0 or 1, logistic regression can never be perfectly confident." Is this mathematically correct? Explain using the asymptotic behaviour of σ.
5. For a model with w=1.5 and b=−6, find the exact hours-studied value at the decision boundary.

Answers: (1) 1−σ(z) = e^−z/(1+e^−z); multiplying numerator and denominator by e^z gives 1/(1+e^z) = σ(−z). (2) At x=3, z=0.9(3)−4.2=−1.5, p≈0.182; at x=9, z=0.9(9)−4.2=3.9, p≈0.980. (3) ∂L/∂p · ∂p/∂z · ∂z/∂b = [(p−y)/(p(1−p))] · p(1−p) · 1 = p−y. (4) Correct — σ is asymptotic, so it approaches but never equals 0 or 1 for any finite z; the model can become arbitrarily confident but never absolutely certain. (5) x = −b/w = 6/1.5 = 4.0 hours.