Logistic Regression: The Foundation of Neural Network Classifiers

A Straight Line Cannot Answer a Yes/No Question

Suppose you are tracking ten students preparing for a JEE Main mock test. For each student you record hours studied in the final week and whether they cleared the cutoff (1) or did not (0):

Hours: 1, 2, 3, 4, 5, 6, 7, 8, 9    Cleared: 0, 0, 0, 0, 1, 1, 1, 1, 1

This looks like a job for linear regression — fit a straight line through the points and use it to predict the probability of clearing the cutoff for any number of hours studied. Fitting a least-squares line to this data (the same method you used for continuous regression) gives:

p̂ = 0.17h − 0.28

Two things go wrong immediately if you try to read this as a probability. At h = 0 hours, the line predicts p̂ = −0.28 — a negative probability, which is meaningless. At h = 10 hours, it predicts p̂ = 1.39 — a probability greater than 1, also meaningless. A line has no ceiling and no floor, but a probability is trapped between 0 and 1 by definition.

Now add one more student to the dataset: someone who studied 9 hours but choked under pressure and did not clear the cutoff. Refitting the line to these eleven points changes the slope from 0.17 to about 0.11 — a 35% drop caused by a single data point. Every other prediction in the dataset shifts because of one outlier, because ordinary least squares tries to minimize the vertical distance to every point equally, including points that are already correctly and confidently classified. This is the core failure of using linear regression for a classification problem: it optimizes the wrong thing (minimizing squared distance to 0 or 1) and produces outputs that are not valid probabilities at all. Logistic regression exists to fix both problems at once, and it does so by changing what function of the inputs we choose to make linear.

What We Actually Need: A Bounded, Interpretable Squashing Function

We still want the decision to depend linearly on the input — more hours studied should smoothly increase the predicted chance of clearing the cutoff, and we want a single formula that combines multiple features (hours studied, previous mock-test score, sleep hours) into one score. What we do not want is for that raw linear score to be reported directly as a probability. So the strategy is: keep computing a linear score z = w·x + b exactly as before, but pass z through a function that squashes any real number into the range (0, 1) before calling it a probability.

Deriving the Sigmoid Function from Odds

Statisticians already had a bounded quantity that behaves linearly under the right transformation: the log-odds, also called the logit. If p is a probability, the odds of the event are p / (1−p) — a cricket commentator saying "odds of 3 to 1" means p/(1−p) = 3, i.e. p = 0.75. Odds range from 0 to infinity, which is still not the whole real line, but the logarithm of the odds does range over all real numbers: as p → 0, the log-odds → −∞; as p → 1, the log-odds → +∞. So logistic regression makes the following assumption, called the logit link:

ln( p / (1−p) ) = z = w·x + b

This says the log-odds of the event is a linear function of the features — exactly the kind of relationship linear regression is built to model. To get back a usable probability, solve this equation for p algebraically, step by step:

ln(p / (1-p)) = z
p / (1-p) = e^z                     [exponentiate both sides]
p = e^z (1 - p)                     [multiply both sides by (1-p)]
p = e^z - e^z p                     [expand]
p + e^z p = e^z                     [collect p terms on one side]
p(1 + e^z) = e^z                    [factor]
p = e^z / (1 + e^z)                 [divide]
p = 1 / (1 + e^(-z))                [divide numerator and denominator by e^z]

That last line is the sigmoid function, usually written σ(z) = 1 / (1 + e^(−z)). It was not invented arbitrarily to "look like an S" — it falls directly out of insisting that the log-odds be linear. Three properties are worth checking by direct substitution: σ(0) = 1/(1+1) = 0.5, σ(z) → 1 as z → +∞ (since e^{-z} → 0), and σ(z) → 0 as z → -∞ (since e^{-z} → ∞). No matter how large or negative the linear score z gets, σ(z) stays strictly between 0 and 1 — the exact defect that broke linear regression is repaired.

The Decision Boundary

Once probabilities are computed, classification needs a rule: predict "pass" if p ≥ 0.5, else predict "fail". Because σ(z) = 0.5 exactly when z = 0 (check: σ(0) = 1/(1+e^0) = 1/2), the 0.5 probability threshold corresponds precisely to w·x + b = 0. This is a linear equation in the features — a point for one feature, a line for two features, a plane for three, and a hyperplane in general. This is why logistic regression is called a linear classifier: even though the probability curve is an S-shaped sigmoid, the boundary that separates the two predicted classes is perfectly straight (linear) in feature space. The curve is only curved when you plot probability against the input; the decision itself is a straight cut.

Why Not Just Minimize Squared Error?

A natural instinct is to reuse the mean squared error cost from linear regression: J(w,b) = (1/m)Σ(σ(z_i) − y_i)². This works numerically but is a poor choice for two reasons. First, plotting squared error against the weights when a sigmoid is involved produces a non-convex surface with multiple local minima and flat plateaus, so gradient descent can get stuck far from the best solution. Second, squared error punishes a confidently wrong prediction only mildly — if the true label is 1 and the model predicts p = 0.01, squared error charges just (0.01−1)² ≈ 0.98, barely more than a middling wrong guess. We want a cost function that is both convex (so gradient descent reliably finds the global minimum) and that punishes confident, wrong predictions severely. Maximum likelihood estimation delivers exactly that.

Deriving the Cost Function: From Likelihood to Cross-Entropy

For a single training example with true label y ∈ {0, 1} and predicted probability p = σ(z), the model assigns probability p to the event "label is 1" and probability 1−p to the event "label is 0". Both cases can be written in one expression using y as an on/off switch:

L(y, p) = p^y · (1−p)^(1−y)

Check it: if y=1, this reduces to p¹·(1−p)⁰ = p; if y=0, it reduces to p⁰·(1−p)¹ = 1−p. Both are correct. Assuming the training examples are independent, the likelihood of the whole dataset is the product of these terms across all m examples. Products of many small probabilities underflow numerically and are awkward to differentiate, so we take the logarithm, which turns the product into a sum without changing where the maximum occurs (log is strictly increasing):

log-likelihood = Σ [ y_i ln(p_i) + (1−y_i) ln(1−p_i) ]

Maximizing this log-likelihood is identical to minimizing its negative average, which is the cost function actually used in logistic regression, called binary cross-entropy or log loss:

J(w,b) = −(1/m) Σ [ y_i ln(p_i) + (1−y_i) ln(1−p_i) ]

Look at what happens to a single term when the model is confidently wrong: if y=1 but p=0.01, the term is −ln(0.01) ≈ 4.6 — more than four times the penalty that squared error gave the same mistake, and the penalty grows without bound as p → 0. This is precisely the sharp, convex penalty we needed.

The Gradient: Why Training Turns Out So Clean

To run gradient descent we need ∂J/∂w. This requires the derivative of the sigmoid itself. Using the quotient rule on σ(z) = 1/(1+e^{-z}), with u=1 and v = 1+e^{-z} so that u′=0 and v′=−e^{-z}:

sigma'(z) = (u'v - uv') / v^2 = (0 - 1*(-e^-z)) / v^2 = e^-z / v^2

Now compare this to σ(z)·(1−σ(z)):

sigma(z)*(1-sigma(z)) = (1/v) * (1 - 1/v) = (1/v) * ((v-1)/v) = (v-1)/v^2

Since v − 1 = e^{-z}, this equals e^{-z}/v² — the same expression found above. This gives the compact and famous identity σ′(z) = σ(z)(1−σ(z)). Now apply the chain rule to the cost for one example, L = −[y ln(σ) + (1−y) ln(1−σ)]:

dL/dsigma = -(y/sigma - (1-y)/(1-sigma))
dL/dz = dL/dsigma * sigma'(z)
       = -(y/sigma - (1-y)/(1-sigma)) * sigma*(1-sigma)
       = -[ y(1-sigma) - (1-y)*sigma ]
       = -[ y - y*sigma - sigma + y*sigma ]
       = -[ y - sigma ]
       = sigma - y

Every term involving the sigmoid's own derivative cancels out, leaving the strikingly simple result ∂L/∂z = p − y — the prediction error itself. Applying ∂z/∂w_j = x_j via the chain rule once more, and averaging over all m examples, gives the gradient descent update rules:

dJ/dw_j = (1/m) * sum( (p_i - y_i) * x_ij )
dJ/db   = (1/m) * sum( p_i - y_i )

w_j := w_j - alpha * dJ/dw_j
b   := b   - alpha * dJ/db

where α is the learning rate. This is mechanically identical in form to the linear regression gradient — only p_i is now σ(z_i) instead of z_i itself. The heavy algebra of the sigmoid's derivative and the log-loss derivative was necessary precisely so that this final formula would come out this clean.

Worked Example: One Gradient Descent Step by Hand

To make the arithmetic tractable by hand, shrink to four students: hours studied x = [1, 2, 3, 4], cleared y = [0, 0, 1, 1]. Initialize w = 0, b = 0, learning rate α = 0.1.

Step 1 — forward pass. With w=0, b=0, every z_i = 0·x_i + 0 = 0, so every prediction is p_i = σ(0) = 0.5, regardless of x_i. This makes sense: with no information yet learned, the model guesses a coin-flip for everyone.

Step 2 — errors. p_i − y_i gives [0.5, 0.5, −0.5, −0.5] for the four students.

Step 3 — gradients.

dJ/dw = (1/4) * (0.5*1 + 0.5*2 + (-0.5)*3 + (-0.5)*4)
      = (1/4) * (0.5 + 1.0 - 1.5 - 2.0)
      = (1/4) * (-2.0) = -0.5

dJ/db = (1/4) * (0.5 + 0.5 - 0.5 - 0.5) = 0

Step 4 — update. w := 0 − 0.1×(−0.5) = 0.05, and b := 0 − 0.1×0 = 0. The weight moved in the positive direction, exactly as it should: students with more hours (x=3,4) were under-predicted (negative error), students with fewer hours (x=1,2) were over-predicted (positive error), and increasing w raises predictions for large x more than for small x, nudging the model in the right direction. Note b stayed at 0 because the errors happened to cancel out symmetrically on this particular dataset — that will not generally be true.

Training to Convergence: Verified Code

Running the same update rule for many iterations on this dataset (verified by direct execution, not estimated) produces:

import math

x = [1, 2, 3, 4]
y = [0, 0, 1, 1]
m = len(x)

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

w, b = 0.0, 0.0
lr = 0.1

for it in range(10000):
    preds = [sigmoid(w*xi + b) for xi in x]
    errors = [p - yi for p, yi in zip(preds, y)]
    dw = sum(e*xi for e, xi in zip(errors, x)) / m
    db = sum(errors) / m
    w -= lr*dw
    b -= lr*db

print(w, b)
print([round(sigmoid(w*xi+b), 4) for xi in x])
print(-b/w)   # decision boundary

This prints w ≈ 5.7975, b ≈ −14.3087, final probabilities [0.0002, 0.0622, 0.9562, 0.9999] for the four students, and a decision boundary at −b/w ≈ 2.468 hours. Compare this to where training started: the initial cost (with w=b=0) was −ln(0.5) = 0.6931 for every point — note this equals ln 2, which is always the starting cross-entropy loss when a logistic model begins at zero weights, since every prediction starts at exactly 0.5. After 10,000 iterations the average cross-entropy loss fell to about 0.0273. The model correctly learned that clearing the cutoff becomes very likely somewhere between 2 and 3 hours of study for this toy dataset, and it grew increasingly confident (probabilities pushed toward 0 and 1) because these four points are perfectly separable — a straight boundary can split them with zero errors, so gradient descent keeps sharpening the boundary indefinitely. In a real dataset with overlapping classes, the loss would plateau above zero because no boundary can achieve perfect separation.

Correcting a Common Misconception

A mistake many students make from the name alone: "logistic regression is a regression algorithm, so it must predict a continuous number like house price." This is wrong. Logistic regression is a classification algorithm — its output y is discrete (pass/fail, spam/not-spam, fraud/not-fraud). The word "regression" in its name refers to the internal mechanism, not the output: it is literally performing linear regression on the log-odds of the outcome, as the derivation above showed. The confusion is understandable, but the fix is to remember what is linear: not the probability itself (which is S-shaped), but the log-odds that feeds into the sigmoid.

A second, subtler misconception: that 0.5 is always the "correct" threshold for turning a probability into a decision. It is only the threshold that minimizes total misclassifications when both types of error are equally costly. In a UPI fraud-detection system, missing an actual fraud (false negative) is usually far more costly than flagging a genuine transaction for review (false positive), so banks often use a much lower threshold, like 0.2, to catch more true frauds at the cost of more false alarms. The mathematics of logistic regression only produces the probability p; choosing the threshold on p is a separate business or safety decision layered on top.

From One Neuron to a Network

Here is why this chapter's title calls logistic regression "the foundation" of neural network classifiers rather than just a stand-alone algorithm. A single artificial neuron, as used in a neural network, computes exactly two operations: a linear combination of its inputs, z = w·x + b, followed by a nonlinear activation function applied to z. If that activation function is the sigmoid, then one neuron is a logistic regression model, computed exactly the way this chapter derived it. The output layer of a binary-classification neural network is, almost always, literally a logistic regression unit sitting on top of whatever features the earlier layers have learned to construct.

A full neural network is built by stacking many such units into layers: the outputs of one layer of neurons become the inputs x to the next layer, each neuron computing its own z = w·x+b followed by an activation. The earlier "hidden" layers typically use a different activation (ReLU is common in modern networks; sigmoid and tanh were historically used and are still important to understand), while the sigmoid usually reappears at the very output for binary classification, converting the network's final linear score into a genuine probability — exactly the role it plays here. The gradient derivation you just worked through, ∂L/∂z = p − y, is also the exact starting point of the backpropagation algorithm at the output layer of a neural network: the error signal that gets propagated backward through the earlier layers begins as this same simple difference between prediction and true label. Understanding logistic regression rigorously, including deriving its cost function and gradient by hand as done above, is therefore not a side topic before neural networks — it is the output layer of a neural network, examined in isolation.

Multiple Features and the General Form

Everything above used a single feature (hours studied) for clarity, but the same derivation holds unchanged with n features, such as hours studied, previous mock-test percentile, and hours of sleep. The score becomes z = w_1x_1 + w_2x_2 + … + w_nx_n + b = w·x + b (a dot product), the sigmoid, cost function, and gradient formulas are identical in form, and the decision boundary generalizes from a single threshold point to a straight line, then a plane, then a hyperplane, as the number of features grows. When there are more than two classes to predict (say, classifying a student's likely rank band as Excellent/Good/Needs Improvement), logistic regression generalizes to softmax regression, which replaces the single sigmoid with a function that outputs a probability for each class such that all class probabilities sum to 1 — the same log-odds idea extended across multiple classes at once, and the same building block used at the output layer of multi-class neural network classifiers.

Where This Fits Your Exams

The derivations in this chapter draw directly on CBSE Class 12 Mathematics: the quotient rule and chain rule from Continuity and Differentiability and Application of Derivatives are exactly what produced σ′(z) = σ(z)(1−σ(z)); properties of logarithms and exponentials from earlier algebra underlie the odds-to-sigmoid derivation; and the likelihood construction connects to the Probability chapter's treatment of independent events. For IIT-JEE Main and Advanced, expect questions that test whether you can manipulate ln and e^x algebraically (as in solving the logit equation for p) rather than questions naming "logistic regression" explicitly — the underlying calculus and algebra are squarely in syllabus. For students continuing toward a GATE-style computer science foundation or an Olympiad in computational thinking, the convexity argument for why cross-entropy is preferred over squared error, and the derivation of the gradient via chain rule, are the kind of first-principles reasoning those exams reward over memorized formulas.

Check Your Understanding

• A model trained on exam data outputs σ(z) = 0.92 for a particular student. What value of z produced this (solve 0.92 = 1/(1+e^{-z}) for z, to two decimal places)? (Hint: rearrange to e^{-z} = (1−0.92)/0.92, then take ln of both sides.)
• Explain, without just quoting the formula, why ∂L/∂z = p − y being so simple is not a coincidence of this particular example, but a structural consequence of pairing the sigmoid with cross-entropy loss specifically.
• A hospital is training a logistic regression model to flag a rare but serious condition from test results. Should the decision threshold be set above 0.5, below 0.5, or exactly at 0.5? Justify your answer in terms of the relative cost of false negatives versus false positives.
• For the four-student worked example in this chapter, compute the gradient dJ/dw and dJ/db after the first update (using w=0.05, b=0) by hand, following the same four steps shown. Do not just read off the second value from the code output — derive it.
• Why is the decision boundary of logistic regression always linear in the input features, even though the probability curve itself is visibly curved (S-shaped)? Answer in terms of where z=0 occurs.

Summary

• Linear regression fails at classification because its output is unbounded, producing invalid "probabilities" below 0 or above 1, and it is overly sensitive to points that are already correctly classified.
• Logistic regression assumes the log-odds of the outcome is linear in the features: ln(p/(1−p)) = w·x+b. Solving this for p algebraically yields the sigmoid function, σ(z) = 1/(1+e^{-z}), which is bounded strictly between 0 and 1 for all real z.
• The decision boundary (p=0.5) always corresponds to z=0, which is a linear equation — so the boundary is a straight line (or hyperplane), even though probability-versus-input is curved.
• Squared error is a poor cost function for a sigmoid model because it is non-convex and under-penalizes confident wrong answers. Maximum likelihood estimation instead yields binary cross-entropy (log loss) as the correct, convex cost function.
• The sigmoid's derivative simplifies to σ(z)(1−σ(z)), and combining it with the cross-entropy loss via the chain rule produces the remarkably clean gradient ∂L/∂z = p − y, giving the update rule w := w − α(1/m)Σ(p_i−y_i)x_i.
• A single sigmoid-activated neuron in a neural network is a logistic regression unit; stacking such units into layers, with the sigmoid typically retained at the output for binary classification, is how logistic regression becomes the output stage — and the historical starting point — of neural network classifiers.
• "Regression" in the name refers to regressing the log-odds linearly, not to predicting a continuous output; and the 0.5 threshold is a default, not a law — real systems tune it based on the relative cost of false positives versus false negatives.

Think About It

Think about this: How would you explain logistic regression: the foundation of neural network classifiers to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where logistic regression: the foundation of neural network classifiers is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting logistic regression: the foundation of neural network classifiers to at least 3 other topics you have studied.