The Fundamental Tradeoff
Every ML model makes two kinds of mistakes. Bias is systematic error from wrong assumptions — the model is too simple to capture the true pattern. Variance is sensitivity to training data fluctuations — the model memorises noise instead of learning the signal.
Bias–Variance Decomposition of MSE$$\mathbb{E}[(y - \hat{f})^2] = \underbrace{\text{Bias}[\hat{f}]^2}_{\text{Underfitting}} + \underbrace{\text{Var}[\hat{f}]}_{\text{Overfitting}} + \underbrace{\sigma^2}_{\text{Irreducible noise}}$$Irreducible noise $\sigma^2$ is the floor — you can never beat it. Your job is to balance bias and variance. Reducing one typically increases the other.
Diagnosing Underfitting vs. Overfitting
Underfitting (High Bias)
- High training error and high validation error.
- Training and validation curves both plateau at a high loss.
- Fix: more complex model, add features, reduce regularisation.
Overfitting (High Variance)
- Low training error but high validation error.
- Large gap between training and validation loss curves.
- Fix: more data, regularisation, simpler model, dropout, early stopping.
Learning Curves
A learning curve plots training and validation performance as a function of training set size. It's one of the most informative diagnostic tools available — you can read the bias-variance situation directly from the shape of the curves.
Python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.datasets import make_regression
X, y = make_regression(n_samples=400, n_features=1, noise=25, random_state=42)
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
for ax, (degree, alpha, label) in zip(axes, [(1, 0.0, "Degree 1 — Underfit"),
(10, 0.01, "Degree 10 — Good Fit")]):
pipe = Pipeline([("poly", PolynomialFeatures(degree=degree)),
("scaler", StandardScaler()),
("model", Ridge(alpha=alpha))])
train_sizes, train_scores, val_scores = learning_curve(
pipe, X, y,
train_sizes=np.linspace(0.05, 1.0, 20),
cv=5, scoring="neg_mean_squared_error", n_jobs=-1
)
train_rmse = np.sqrt(-train_scores.mean(axis=1))
val_rmse = np.sqrt(-val_scores.mean(axis=1))
train_std = np.sqrt(-train_scores).std(axis=1)
val_std = np.sqrt(-val_scores).std(axis=1)
ax.plot(train_sizes, train_rmse, label="Train RMSE", color="#4a8fa8", lw=2)
ax.fill_between(train_sizes, train_rmse - train_std, train_rmse + train_std, alpha=0.15, color="#4a8fa8")
ax.plot(train_sizes, val_rmse, label="Val RMSE", color="#b85c2a", lw=2)
ax.fill_between(train_sizes, val_rmse - val_std, val_rmse + val_std, alpha=0.15, color="#b85c2a")
ax.set_title(label); ax.set_xlabel("Training Set Size"); ax.set_ylabel("RMSE")
ax.legend(); ax.grid(alpha=0.3)
plt.suptitle("Learning Curves — Diagnosing Bias vs. Variance", fontsize=13)
plt.tight_layout(); plt.show()Reading learning curves: If both train and val error are high and converge → high bias (underfitting). If train error is low but val error is much higher → high variance (overfitting). A large gap that shrinks as you add data → adding more data will help.
Validation Curves
A validation curve plots performance vs. a single hyperparameter. Reveals the sweet spot between underfitting and overfitting for that parameter.
Python
from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeRegressor
depths = np.arange(1, 20)
train_scores, val_scores = validation_curve(
DecisionTreeRegressor(random_state=42),
X, y, param_name="max_depth", param_range=depths,
cv=5, scoring="neg_mean_squared_error"
)
train_rmse = np.sqrt(-train_scores.mean(axis=1))
val_rmse = np.sqrt(-val_scores.mean(axis=1))
plt.figure(figsize=(8, 4))
plt.plot(depths, train_rmse, label="Train RMSE", color="#4a8fa8", lw=2, marker="o", ms=4)
plt.plot(depths, val_rmse, label="Val RMSE", color="#b85c2a", lw=2, marker="o", ms=4)
best_depth = depths[np.argmin(val_rmse)]
plt.axvline(best_depth, color="green", ls="--", label=f"Best depth = {best_depth}")
plt.xlabel("max_depth"); plt.ylabel("RMSE")
plt.title("Validation Curve — Decision Tree max_depth")
plt.legend(); plt.grid(alpha=0.3); plt.tight_layout(); plt.show()Cross-Validation
A single train/val split gives a noisy estimate of generalisation performance. K-fold cross-validation uses all data for both training and validation, giving a more reliable estimate.
Python
from sklearn.model_selection import KFold, StratifiedKFold, cross_validate
from sklearn.ensemble import RandomForestRegressor
# K-Fold: splits data into K equal folds
# StratifiedKFold: maintains class proportion in each fold (use for classification)
cv = KFold(n_splits=5, shuffle=True, random_state=42)
results = cross_validate(
RandomForestRegressor(n_estimators=100, random_state=42),
X, y, cv=cv,
scoring=["neg_root_mean_squared_error", "r2"],
return_train_score=True
)
print(f"Train RMSE: {-results['train_neg_root_mean_squared_error'].mean():.3f} "
f"± {results['train_neg_root_mean_squared_error'].std():.3f}")
print(f"Val RMSE: {-results['test_neg_root_mean_squared_error'].mean():.3f} "
f"± {results['test_neg_root_mean_squared_error'].std():.3f}")
print(f"Val R²: {results['test_r2'].mean():.4f}")Fixing High Bias and High Variance
High bias fixesAdd more features; use a more complex model (higher polynomial degree, deeper tree); reduce regularisation strength ($\lambda$); train longer.
High variance fixesGet more training data; increase regularisation ($\lambda$); reduce model complexity; use dropout (neural nets); use ensemble methods (Random Forest averages out variance).
Both highRare but possible. Usually indicates fundamentally wrong model family or corrupted data. Revisit problem formulation.
Summary
- MSE = Bias² + Variance + Noise. You control the first two; noise is irreducible.
- Learning curves: diagnose whether adding data or changing model complexity will help.
- Validation curves: find the optimal value for a single hyperparameter.
- K-fold CV gives a more reliable generalisation estimate than a single split.