Regression Metrics

Overview of Regression Metrics

For regression, our model outputs a continuous number and we measure how far predictions are from the truth. Each metric penalises errors differently — the right choice depends on whether outliers matter, whether you need interpretability, and whether you want a normalised score.

MAE — Mean Absolute Error

MAE$$\text{MAE} = \frac{1}{m}\sum_{i=1}^m |\hat{y}^{(i)} - y^{(i)}|$$Average absolute error. Same units as the target. Robust to outliers (uses absolute value, not squared). Interpretable: "on average, predictions are off by X units."

MSE and RMSE

MSE and RMSE$$\text{MSE} = \frac{1}{m}\sum_{i=1}^m (\hat{y}^{(i)} - y^{(i)})^2 \qquad \text{RMSE} = \sqrt{\text{MSE}}$$MSE penalises large errors quadratically — a prediction off by 10 contributes 100× more than one off by 1. RMSE brings units back to the target scale. RMSE > MAE always (unless all errors are equal).

R² — Coefficient of Determination

R² Score$$R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}} = 1 - \frac{\sum_i(y_i-\hat{y}_i)^2}{\sum_i(y_i-\bar{y})^2}$$$R^2 = 1$: perfect fit. $R^2 = 0$: model no better than always predicting the mean. $R^2 < 0$: model is worse than the mean baseline. Scale-free — comparable across different datasets.

MAPE — Mean Absolute Percentage Error

MAPE$$\text{MAPE} = \frac{100}{m}\sum_{i=1}^m \left|\frac{y_i - \hat{y}_i}{y_i}\right| \%$$Scale-free, expressed as a percentage. Intuitive for business stakeholders. Breaks when $y_i \approx 0$. Use SMAPE as a more robust alternative.

Full Metrics Comparison

Python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import (mean_absolute_error, mean_squared_error,
                              r2_score, mean_absolute_percentage_error)
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

data = fetch_california_housing()
X, y = data.data, data.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

models = {
    "Ridge":             Pipeline([("sc", StandardScaler()), ("m", Ridge())]),
    "Random Forest":     RandomForestRegressor(n_estimators=100, random_state=42),
    "Gradient Boosting": GradientBoostingRegressor(n_estimators=200, learning_rate=0.1, random_state=42),
}

print(f"{'Model':20s} {'MAE':>8s} {'RMSE':>8s} {'R²':>8s} {'MAPE':>8s}")
print("-" * 60)
for name, model in models.items():
    model.fit(X_tr, y_tr)
    y_pred = model.predict(X_te)
    mae  = mean_absolute_error(y_te, y_pred)
    rmse = np.sqrt(mean_squared_error(y_te, y_pred))
    r2   = r2_score(y_te, y_pred)
    mape = mean_absolute_percentage_error(y_te, y_pred) * 100
    print(f"{name:20s} {mae:8.4f} {rmse:8.4f} {r2:8.4f} {mape:7.2f}%")

Residual Plots

Python

import scipy.stats as stats

gb = GradientBoostingRegressor(n_estimators=200, learning_rate=0.1, random_state=42)
gb.fit(X_tr, y_tr)
y_pred = gb.predict(X_te)
residuals = y_te - y_pred

fig, axes = plt.subplots(1, 3, figsize=(15, 4))

# 1. Predicted vs. Actual (should hug the diagonal)
axes[0].scatter(y_pred, y_te, alpha=0.3, s=10, color="#4a8fa8")
lim = [min(y_pred.min(), y_te.min()), max(y_pred.max(), y_te.max())]
axes[0].plot(lim, lim, "r--", lw=1.5)
axes[0].set_xlabel("Predicted"); axes[0].set_ylabel("Actual")
axes[0].set_title(f"Predicted vs. Actual  (R²={r2_score(y_te, y_pred):.4f})")

# 2. Residuals vs. Fitted (should show no pattern — random scatter around 0)
axes[1].scatter(y_pred, residuals, alpha=0.3, s=10, color="#5c8a58")
axes[1].axhline(0, color="red", ls="--", lw=1.5)
axes[1].set_xlabel("Fitted values"); axes[1].set_ylabel("Residuals")
axes[1].set_title("Residuals vs. Fitted")

# 3. Q-Q plot (residuals should be Gaussian)
stats.probplot(residuals, plot=axes[2])
axes[2].set_title("Normal Q-Q Plot")

plt.tight_layout(); plt.show()

Choosing the Right Metric

MAEWhen outliers are real and shouldn't dominate. Robust, interpretable in original units. Use for median prediction.

RMSEWhen large errors are especially bad. Penalises outliers more. The standard optimisation target for regression.

R²When you need a normalised score to compare across datasets. Intuitive: "fraction of variance explained."

MAPEWhen percentage errors are meaningful (e.g., forecasting). Avoid when target can be zero or near-zero.

Summary

MAE: average absolute error. Robust to outliers. Same units as target.
RMSE: square root of mean squared error. Penalises large errors more heavily. Most common.
R²: fraction of variance explained. Scale-free, comparable across datasets. 1.0 is perfect.
Always plot predicted vs. actual and residuals — numbers alone don't reveal patterns.