Why Feature Selection?
More features is not always better. Irrelevant features add noise. Redundant features waste computation. Too many features relative to samples → overfitting. Feature selection finds the subset that maximises predictive power while minimising complexity.
Correlation Heatmap
The fastest way to identify redundant (highly correlated) features and strong predictors of the target:
Python
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
df = pd.DataFrame(data.data, columns=data.feature_names)
df["target"] = data.target
corr = df.corr()
# Full heatmap
fig, ax = plt.subplots(figsize=(16, 12))
import numpy as np
mask = np.triu(np.ones_like(corr, dtype=bool))
sns.heatmap(corr, mask=mask, annot=False, cmap="RdYlBu_r", center=0,
square=True, linewidths=0.3, ax=ax)
ax.set_title("Feature Correlation Matrix")
plt.tight_layout(); plt.show()
# Find highly correlated pairs (|r| > 0.9 → redundant)
corr_pairs = (corr.abs()
.where(np.triu(np.ones(corr.shape), k=1).astype(bool))
.stack()
.sort_values(ascending=False)
)
print("Highly correlated pairs (|r| > 0.9):")
print(corr_pairs[corr_pairs > 0.9].head(10))Filter Methods
Score each feature independently of any model. Fast but ignores feature interactions.
Python
from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif
from sklearn.datasets import load_breast_cancer
import pandas as pd
X, y = load_breast_cancer(return_X_y=True)
feature_names = load_breast_cancer().feature_names
# ANOVA F-test (assumes linear relationship)
sel_f = SelectKBest(f_classif, k=10)
sel_f.fit(X, y)
f_scores = pd.Series(sel_f.scores_, index=feature_names).sort_values(ascending=False)
print("Top 10 by F-score:
", f_scores.head(10))
# Mutual Information (non-parametric — detects non-linear relationships)
mi_scores = mutual_info_classif(X, y, random_state=42)
mi_series = pd.Series(mi_scores, index=feature_names).sort_values(ascending=False)
print("
Top 10 by Mutual Information:
", mi_series.head(10))
# Remove near-zero variance features (essentially constants)
from sklearn.feature_selection import VarianceThreshold
vt = VarianceThreshold(threshold=0.01)
X_high_var = vt.fit_transform(X)
print(f"
Features after variance filtering: {X_high_var.shape[1]} / {X.shape[1]}")Wrapper Methods
Evaluate feature subsets using a model. More accurate but computationally expensive.
Python
from sklearn.feature_selection import RFE, RFECV
from sklearn.ensemble import RandomForestClassifier
# RFE — Recursive Feature Elimination
rfe = RFE(estimator=RandomForestClassifier(n_estimators=50, random_state=42),
n_features_to_select=10, step=2)
rfe.fit(X, y)
selected_rfe = feature_names[rfe.support_]
print("RFE selected:", selected_rfe)
# RFECV — RFE with cross-validation to find optimal number automatically
rfecv = RFECV(estimator=RandomForestClassifier(n_estimators=50, random_state=42),
cv=5, scoring="roc_auc", step=1)
rfecv.fit(X, y)
print(f"
Optimal # features (RFECV): {rfecv.n_features_}")Embedded Methods (L1 + Tree Importance)
Feature selection happens as part of model training — more efficient than wrappers.
Python
from sklearn.linear_model import LassoCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
import pandas as pd
X, y = load_breast_cancer(return_X_y=True)
# L1 (Lasso) — drives irrelevant features to exactly zero
lasso_sel = SelectFromModel(LassoCV(cv=5, random_state=42))
lasso_sel.fit(X, y.astype(float))
print(f"Lasso selected {lasso_sel.get_support().sum()} features")
# Random Forest importance — mean decrease in impurity
rf = RandomForestClassifier(n_estimators=200, random_state=42)
rf.fit(X, y)
importances = pd.Series(rf.feature_importances_, index=feature_names).sort_values(ascending=False)
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(10, 8))
importances.head(15).sort_values().plot.barh(ax=ax, color="#4a8fa8")
ax.set_title("Random Forest — Feature Importance (top 15)")
ax.set_xlabel("Mean Decrease in Impurity")
plt.tight_layout(); plt.show()PCA for Dimensionality Reduction
PCA creates new orthogonal features (principal components) that capture maximum variance. It doesn't select original features — it creates linear combinations of them.
Python
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import matplotlib.pyplot as plt
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
# PCA must be applied after scaling
pca_pipe = Pipeline([
("scaler", StandardScaler()),
("pca", PCA()),
])
X_pca = pca_pipe.fit_transform(X)
# How many components do we need?
explained = pca_pipe.named_steps["pca"].explained_variance_ratio_
cumulative = np.cumsum(explained)
n_for_95 = np.searchsorted(cumulative, 0.95) + 1
print(f"Components needed for 95% variance: {n_for_95} / {X.shape[1]}")
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.bar(range(1, 16), explained[:15] * 100, color="#4a8fa8")
ax1.set_title("Scree Plot — Explained Variance per Component")
ax1.set_xlabel("Component"); ax1.set_ylabel("Explained Variance (%)")
ax2.plot(range(1, len(cumulative)+1), cumulative * 100, color="#b85c2a", marker="o", ms=3)
ax2.axhline(95, color="green", ls="--", label="95%")
ax2.set_title("Cumulative Explained Variance"); ax2.legend()
ax2.set_xlabel("Number of Components"); ax2.set_ylabel("Cumulative (%)")
plt.tight_layout(); plt.show()
# Final: keep 10 components (covers 95%+)
pca_final = Pipeline([("scaler", StandardScaler()), ("pca", PCA(n_components=n_for_95))])
X_reduced = pca_final.fit_transform(X)
print(f"Original shape: {X.shape} → Reduced: {X_reduced.shape}")PCA or feature selection? Use feature selection when interpretability
matters (you want to know which original features are important). Use PCA when you just
want to reduce dimensions for speed/storage and don't need to explain individual features.
Summary
- Correlation heatmap: find redundant pairs (|r| > 0.9 → drop one).
- Filter methods: fast, univariate (F-test, mutual information, variance threshold).
- Wrapper methods: RFE uses a model to score subsets. Slow but accurate.
- Embedded methods: L1 (Lasso) → zero out features; RF importance → rank features. Best of both worlds.
- PCA: creates new features as linear combinations. Use when you need maximum compression, not interpretability.