Feature Selection: Choosing What Matters

When Adding a Column Makes Your Model Worse

Suppose you're building a model to predict a student's Class 10 board exam score. You collect a spreadsheet: attendance percentage, weekly study hours, sleep hours, distance from home to school. A friend, trying to be helpful, says: "Why not also add roll number, the student's mobile number, and the PIN code of their house? More columns means more information for the model, right?"

This feels reasonable and is completely wrong. In this chapter you will prove, with actual numbers you compute yourself, that adding the wrong columns doesn't just fail to help a model — it actively makes predictions worse, harder to trust, and slower to compute. You will also see the reverse trap: a feature that looks statistically useless in isolation can be exactly the piece a model needs. Feature selection is the discipline of telling these two situations apart, and it is one of the few places in machine learning where a wrong choice is invisible until it quietly wrecks your results on data the model hasn't seen yet.

What Counts as a "Feature"?

A feature (also called an attribute or an independent variable) is any measured quantity you feed into a model to help it predict a target. If you're predicting house price, features might be area, number of bedrooms, and distance from the nearest metro station; the target is the price itself. A dataset with n samples and p features is a table of n rows and p columns, and every model you train — linear regression, a decision tree, a neural network — learns some function of those p columns to produce an output.

Feature selection is the process of choosing a subset of those p columns — discarding the ones that are irrelevant, redundant, or actively harmful to prediction quality — before or during model training. It is different from feature engineering, which creates new features (like turning "date of birth" into "age"), and different from dimensionality reduction techniques like PCA, which combine existing features into new ones rather than choosing among the originals. Feature selection keeps the original, interpretable columns; it just decides which ones earn a place in the model.

Why Bother? The Curse of Dimensionality, Counted Exactly

Here is the argument for why irrelevant features actively hurt a model, not just waste storage. Imagine each feature's range is divided into 10 equal bins (a coarse histogram). With 1 feature, your data can fall into 10 possible bins. With 2 features, the bins form a grid: 10 × 10 = 10² = 100 cells. With 5 features, it's 10⁵ = 100,000 cells. With 20 features, it's 10²⁰ — a number bigger than the count of grains of sand on Earth.

If you need, say, 5 data points per cell for a model to reliably estimate what's "typical" in that region of feature space, then 1 feature needs about 50 samples, but 5 features need 500,000 samples for the same reliability. Real datasets almost never scale their sample count this way — a CBSE school dataset might have 200 students and 40 features. The result is that most regions of the 40-dimensional feature space contain zero or one data point. The model has nothing to learn from in most of its own input space, so it starts fitting to coincidences in the sparse data it does have. This is overfitting, and irrelevant features are one of its most common causes. Fewer, well-chosen features mean denser, more learnable data.

There is a second, independent reason to remove features: redundancy. If your dataset has both "area in square feet" and "area in square metres," these are the same information twice (one is just the other divided by 10.764). Keeping both doesn't add knowledge — it can destabilize a linear model's coefficient estimates, because the model can't tell how to split credit between two columns that move in lockstep. This problem is called multicollinearity, and it's a second, distinct motivation for feature selection beyond simply discarding useless columns.

Filter Methods: Ranking Features by Correlation

The simplest family of feature selection techniques, called filter methods, ranks each feature by a statistical score computed independently of any model, then keeps the top-scoring ones. For a numeric target, the most common score is the Pearson correlation coefficient:

r = [ Σ (x_i - x̄)(y_i - ȳ) ] / √[ Σ(x_i - x̄)² · Σ(y_i - ȳ)² ]

where x is a feature, y is the target, and x̄, ȳ are their means. The numerator measures how much x and y move together (their covariance); the denominator normalizes by how much each varies on its own, so r always lands between −1 and +1, regardless of the units used. This is the same coefficient taught in the CBSE Class 11 Statistics chapter on correlation — here you're using it as a tool, not just computing it as an exercise.

Let's compute it by hand on five houses, with three candidate features for predicting price:

Mean area = 8.4, mean price = 39.4. The deviations from the mean are (−3.4, −1.4, −0.4, 1.6, 3.6) for area and (−14.4, −6.4, −1.4, 6.6, 15.6) for price. Multiplying matching pairs and summing: 48.96 + 8.96 + 0.56 + 10.56 + 56.16 = 125.20. The sum of squared deviations is 29.20 for area and 537.20 for price. So:

r_area = 125.20 / √(29.20 × 537.20) = 125.20 / 125.24 ≈ 0.9996

Almost perfect correlation — exactly what you'd expect, since bigger houses genuinely cost more. Now do the same for Listing ID (just the row number, 1 through 5, which carries zero real information about the house):

r_ID = 73.00 / √(10 × 537.20) = 73.00 / 73.29 ≈ 0.9960

A meaningless column scores 0.996 — nearly as "relevant" as area itself. And the phone-number digits, which are truly random with respect to price:

r_phone = -0.40 / √(4820.80 × 537.20) = -0.40 / 1609.27 ≈ -0.0002

correctly comes out at essentially zero. You can check the area calculation in code:

import numpy as np

area  = np.array([5, 7, 8, 10, 12])
price = np.array([25, 33, 38, 46, 55])

r = np.corrcoef(area, price)[0, 1]
print(round(r, 4))   # 0.9996 -- matches the hand calculation

Common Misconception: "High Correlation Means Relevant, Low Correlation Means Useless"

The Listing ID result above is not a fluke you can dismiss — it's the whole point. The houses in the table happen to be listed in the same order as their price (smaller, cheaper houses listed first). Because of that, the row number is accidentally correlated with price, purely from how the data was sorted, not from anything about the house. A filter method applied blindly would rank "Listing ID" as almost exactly as useful as "Area." The fix isn't a better formula — it's domain reasoning: ask whether a feature could plausibly cause or relate to the target before trusting a high correlation score. This is exactly why real datasets should be shuffled before any correlation-based ranking, and why filter scores are a starting point for investigation, never a final verdict.

The opposite failure is just as important and catches even careful students: a feature can have exactly zero linear correlation with the target and still be essential, if it only matters in combination with another feature. Consider a target defined as the XOR of two binary features A and B (Y = 1 if exactly one of A, B is 1, else Y = 0):

If all four rows occur equally often, A alone tells you nothing about Y (when A = 0, Y is 0 or 1 with equal chance; same when A = 1) — the correlation between A and Y works out to exactly 0, and the same is true for B alone. Yet A and B together determine Y perfectly. A univariate filter method, which scores one feature at a time, would discard both A and B as useless and destroy the model. This is a genuine limitation of filter methods, not a rare edge case — real interaction effects (like "study hours matter a lot, but only if attendance is also high") show up in exactly this shape. It's the main reason wrapper and embedded methods, covered next, exist at all.

Filter Methods for Categorical Targets: Information Gain

Pearson correlation assumes numeric, roughly linear relationships. When the target is categorical (pass/fail, admitted/not admitted), filter methods instead use information gain, built from Shannon's entropy. For a target with class probabilities p₁, ..., p_k, entropy is defined as:

H(S) = - Σ p_i · log&sub2;(p_i)

Entropy measures uncertainty, in bits: it is the theoretical minimum average number of bits needed to encode which class an outcome belongs to, given those probabilities (Shannon's source coding theorem). A perfectly balanced coin (p = 0.5, 0.5) has maximum uncertainty, H = 1 bit. A coin that always lands heads (p = 1, 0) has H = 0 — no uncertainty, no information needed.

Information gain for a feature measures how much a split on that feature reduces this uncertainty: IG(feature) = H(parent) − weighted average of H(children). Take 8 students, split by whether they attended extra tuition:

Parent entropy, with an exact 4/8, 4/8 split: H(parent) = −0.5·log&sub2;(0.5) − 0.5·log&sub2;(0.5) = 0.5 + 0.5 = 1 bit (maximum uncertainty — a coin flip).

Entropy within the "attended tuition" group (3 pass, 1 fail out of 4): using log&sub2;(3/4) ≈ −0.4150 and log&sub2;(1/4) = −2,

H = -(3/4)(-0.4150) - (1/4)(-2) = 0.3113 + 0.5 = 0.8113 bits

The "no tuition" group is the mirror image (1 pass, 3 fail), so it has the same entropy, 0.8113 bits. Since both groups have 4 of the 8 students, the weighted average child entropy is simply 0.8113 bits, and:

IG(tuition) = H(parent) - H(children) = 1 - 0.8113 = 0.1887 bits ≈ 0.189 bits

A feature that produced a perfect split (all "yes" pass, all "no" fail) would drive child entropy to exactly 0 and information gain to exactly 1 bit — the theoretical maximum here. A feature that produced the exact same pass/fail ratio in both groups as the overall dataset would have information gain of exactly 0 — it tells you nothing new. Ranking features by information gain and keeping the highest scorers is precisely how the ID3 and C4.5 decision-tree algorithms choose which feature to split on at each node — filter-style feature selection is happening inside every decision tree you build, one split at a time.

Wrapper Methods: Let the Model Judge

Wrapper methods fix the blind-spot in filter methods by actually training a model on different feature subsets and comparing real performance, rather than scoring features one at a time. The most direct version, exhaustive search, would try every possible subset — but for p features there are 2^p possible subsets (each feature is independently either in or out; this is the same 2ⁿ "number of subsets of a set" result you meet in Class 11 Permutations and Combinations). For just 20 features, that's 2²⁰ = 1,048,576 models to train — infeasible.

Forward selection avoids this by building the subset greedily, one feature at a time. Start with zero features and add whichever single feature improves the model most; then, keeping that feature fixed, add whichever remaining feature improves it most; repeat until no remaining feature improves the model by more than some small threshold. Here's a worked (illustrative) run predicting an exam score from four candidate features:

Forward selection stops at {Study hours, Attendance}, since neither remaining feature clears the improvement threshold. Backward elimination runs in reverse: start with all features, and repeatedly drop whichever single feature hurts performance the least when removed, stopping once every remaining feature matters. Both are still greedy — they can miss a pair of features that only helps jointly (an XOR-style case) if neither one looks promising alone at the step it would need to be added — but they need only O(p²) model fits instead of 2^p, which is what makes them usable in practice. Scikit-learn implements a related idea, Recursive Feature Elimination (RFE), which trains a model on all features, discards the least important one (by the model's own coefficients or importance scores), and repeats.

Embedded Methods: Lasso and the Geometry of Sparsity

Embedded methods build feature selection directly into model training, rather than as a separate step before or after. The clearest example is Lasso regression (Least Absolute Shrinkage and Selection Operator), which modifies ordinary linear regression's cost function by adding a penalty on the size of the coefficients:

J(w) = MSE(w) + λ · Σ |w_i|          (L1 penalty -- Lasso)
J(w) = MSE(w) + λ · Σ w_i²         (L2 penalty -- Ridge, for comparison)

Both penalties discourage large coefficients, which reduces overfitting. But only the L1 penalty routinely forces coefficients to become exactly zero — which is precisely feature selection, since a zero coefficient means that feature is dropped from the model entirely. Ridge's L2 penalty shrinks coefficients toward zero but essentially never lands exactly on it. The reason is geometric, and it's worth seeing directly, because it explains a result that otherwise looks like a coincidence.

Minimizing MSE(w) + λ·(penalty) is mathematically equivalent to minimizing MSE(w) alone subject to the penalty term staying below some fixed budget t. That budget defines a constraint region: for the L1 penalty Σ|w_i| ≤ t, the region is a diamond (in two dimensions); for the L2 penalty Σw_i² ≤ t, it's a circle. Meanwhile, the unconstrained best-fit solution sits at some point off-center, and the MSE grows in concentric ellipses around it as you move away — so the constrained solution is wherever the smallest ellipse first touches the constraint region.

The diamond has sharp corners that sit exactly on the axes (where one coordinate is zero); the circle has no corners at all. Because the MSE ellipses can approach from any angle, they are far more likely to first touch the diamond precisely at a corner than to touch the smooth circle at a point where one coordinate happens to be exactly zero. That geometric asymmetry — corners versus smoothness — is the entire reason Lasso performs feature selection automatically while Ridge only shrinks. In scikit-learn, increasing Lasso's alpha (which corresponds to shrinking the diamond, i.e. lowering the budget t) pushes more and more coefficients to exactly zero, and past a large enough alpha, every coefficient is guaranteed to become exactly 0.0 — the diamond shrinks to a single point at the origin.

from sklearn.linear_model import Lasso

model = Lasso(alpha=2.0)
model.fit(X_train, y_train)

selected = [name for name, coef in zip(feature_names, model.coef_) if coef != 0]
print(selected)   # only the features Lasso kept

Decision trees and random forests offer a different embedded method: feature importance, computed by summing, across every split in the tree, how much that feature's splits reduced entropy (or the Gini index) weighted by how many samples passed through that split. A feature the tree never bothers to split on gets an importance of exactly zero — the tree has performed feature selection as a side effect of being built.

Choosing a Method in Practice

Filter methods (correlation, information gain) are the cheapest — no model training required — and are the right first pass on a dataset with hundreds of candidate features, to cut the field down before anything expensive runs. But, as the Listing ID and XOR examples showed, they can be fooled by coincidental correlation and blind to interaction effects. Wrapper methods (forward selection, backward elimination, RFE) directly optimize what you actually care about — real model performance — at the cost of training many models, so they suit small-to-medium feature counts where 2^p is out of reach but O(p²) is affordable. Embedded methods (Lasso, tree importance) get selection "for free" during a training run you were going to do anyway, making them the default choice when your model family already supports one. In practice, serious pipelines often chain all three: a filter pass to cut 500 features to 50, then an embedded or wrapper pass to go from 50 to the final handful — because no single method covers every failure mode discussed above.

Exam Focus

The Pearson correlation coefficient computed here is the identical formula from CBSE Class 11 Statistics; expect it tested both as a standalone numerical problem and, in the CBSE Artificial Intelligence curriculum (Code 417), applied to a small dataset exactly like the house-price table above. The subset-counting argument (2^p possible feature subsets) is a direct application of Class 11 Permutations and Combinations, and counting arguments of this shape are common in Olympiad and KVPY-style aptitude sections. Entropy and information gain, along with L1/L2 regularization, appear explicitly in GATE's Data Science and Artificial Intelligence (DA) paper under machine learning and information theory — the exact derivations above (not just the final formulas) are what such papers test, since a memorized formula without the geometric or counting reasoning behind it breaks under a slightly rephrased question.

Check Your Understanding

1. A dataset has 12 candidate features. How many possible feature subsets exist in total (including the empty set)? How many model fits would exhaustive wrapper search require?
2. Two features, "temperature in Celsius" and "temperature in Fahrenheit," are both included in a linear regression. Which problem discussed in this chapter does this create, and why does it matter even though both features are individually meaningful?
3. A feature has a Pearson correlation of exactly 0.02 with the target. Give a concrete reason (from this chapter) why it might still be worth keeping.
4. For a target with a 6-pass, 2-fail split (8 total), compute the entropy H in bits. (Use log&sub2;(0.75) ≈ −0.415 and log&sub2;(0.25) = −2.)
5. Explain, using the diamond-versus-circle picture, why Ridge regression essentially never sets a coefficient to exactly zero.

Answers: (1) 2¹² = 4096 subsets, so exhaustive wrapper search needs 4096 model fits. (2) Multicollinearity — the two columns carry identical information (one is a fixed linear transform of the other), which destabilizes a linear model's coefficient estimates even though each column, alone, is meaningful. (3) It may be part of an interaction effect invisible to a univariate filter score, exactly like the XOR example — low individual correlation doesn't rule out joint predictive power. (4) H = −(0.75)(−0.415) − (0.25)(−2) = 0.311 + 0.5 = 0.811 bits. (5) The L2 constraint region is a smooth circle with no corners, so the ellipse representing the MSE is very unlikely to first touch it at a point where any coordinate is exactly zero; only the L1 diamond has corners sitting exactly on the axes.

Summary

Feature selection chooses which columns of a dataset actually earn a place in a model. The case for doing it is quantitative, not aesthetic: the curse of dimensionality shows that irrelevant features make data sparse exponentially fast (10^p cells for p features), and multicollinearity shows that redundant features destabilize coefficient estimates. Filter methods (Pearson correlation for numeric targets, information gain for categorical ones) rank features cheaply but can be fooled by spurious correlation and miss interaction effects like XOR. Wrapper methods (forward selection, backward elimination, RFE) train real models on real subsets to sidestep that blind spot, at a computational cost that scales with the 2^p subset-counting problem from combinatorics. Embedded methods fold selection into training itself — Lasso's L1 penalty produces exact zeros because its diamond-shaped constraint region has corners on the axes, while Ridge's circular L2 region does not, and decision trees zero out the importance of any feature they never split on. No single method is complete on its own; a correct feature selection pipeline treats correlation, information gain, greedy search, and regularization as complementary tools, each catching failure modes the others miss.