The Wisdom of Crowds
A single decision tree is high-variance — small changes in training data produce wildly different trees. Ensemble methods combine many weak learners into one strong learner. Two fundamentally different strategies exist: Bagging (parallel, reduces variance) and Boosting (sequential, reduces bias).
Bagging — Bootstrap Aggregating
Train $T$ models on different bootstrap samples (random sampling with replacement) of the training data. Average predictions for regression; majority vote for classification.
Variance Reduction via
Averaging$$\text{Var}\!\left(\frac{1}{T}\sum_{t=1}^T h_t\right) = \frac{\rho\sigma^2 +
(1-\rho)\sigma^2/T}{1} \xrightarrow{T\to\infty} \rho\sigma^2$$$\rho$: correlation
between trees. $\sigma^2$: individual tree variance. More trees + less correlation = lower ensemble
variance. Bias stays constant.
Random Forest
Bagging + one extra trick: at each split, only a random subset of $\sqrt{n}$ features is considered. This de-correlates the trees ($\rho$ decreases), reducing ensemble variance further.
Python
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report
import pandas as pd, numpy as np, matplotlib.pyplot as plt
X, y = make_classification(n_samples=2000, n_features=20, n_informative=12,
n_redundant=4, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)
rf = RandomForestClassifier(
n_estimators=200, # number of trees
max_features="sqrt", # features per split: sqrt(n) for classification
max_depth=None, # grow full trees (RF controls variance via averaging)
min_samples_leaf=1,
bootstrap=True, # use bootstrap samples (default)
oob_score=True, # free validation on out-of-bag samples
n_jobs=-1,
random_state=42,
)
rf.fit(X_train, y_train)
print(f"OOB score: {rf.oob_score_:.4f}") # Free, unbiased estimate of test accuracy
print(f"Test score: {rf.score(X_test, y_test):.4f}")
print(classification_report(y_test, rf.predict(X_test)))
# Feature importance
feat_imp = pd.Series(rf.feature_importances_,
index=[f"feat_{i}" for i in range(20)]).sort_values(ascending=False)
feat_imp.head(10).sort_values().plot.barh(color="#5c8a58", figsize=(8,5))
plt.title("Random Forest Feature Importance (top 10)")
plt.xlabel("Mean Decrease in Impurity"); plt.tight_layout(); plt.show()Boosting — Sequential Error Correction
Build models sequentially. Each new model focuses on examples the previous ensemble got wrong. This directly targets and reduces bias.
XGBoost — Extreme Gradient Boosting
The algorithm that dominated Kaggle tabular competitions for years. It fits each new tree to the negative gradient of the loss (pseudo-residuals), with regularisation terms on the tree structure itself:
XGBoost Objective$$\mathcal{L}^{(t)} = \sum_i
l(y_i,\hat{y}_i^{(t-1)}+f_t(\mathbf{x}_i)) + \Omega(f_t) \qquad \Omega(f)=\gamma T +
\frac{1}{2}\lambda\|\mathbf{w}\|^2$$$T$: number of leaves. $\gamma$: minimum gain
to make a split. $\lambda$: L2 regularisation on leaf weights. Both are tunable.
Python
import xgboost as xgb
import lightgbm as lgb
from catboost import CatBoostClassifier
from sklearn.model_selection import cross_val_score
import numpy as np
# ── XGBoost
xgb_model = xgb.XGBClassifier(
n_estimators=300,
learning_rate=0.05, # shrinkage: smaller = more trees needed, less overfitting
max_depth=5,
subsample=0.8, # row subsampling per tree (like bagging)
colsample_bytree=0.8, # feature subsampling per tree (like Random Forest)
reg_alpha=0.1, # L1 regularisation on leaf weights
reg_lambda=1.0, # L2 regularisation on leaf weights
use_label_encoder=False,
eval_metric="logloss",
random_state=42,
n_jobs=-1,
)
# ── LightGBM — faster, handles categorical natively, leaf-wise splitting
lgb_model = lgb.LGBMClassifier(
n_estimators=300,
learning_rate=0.05,
num_leaves=31, # controls complexity (not max_depth). Rule of thumb: 2^max_depth/2
min_child_samples=20, # regularisation
colsample_bytree=0.8,
subsample=0.8,
random_state=42,
n_jobs=-1,
verbose=-1,
)
# ── CatBoost — handles categoricals natively, great default settings
cat_model = CatBoostClassifier(
iterations=300,
learning_rate=0.05,
depth=6,
random_seed=42,
verbose=0,
)
for name, model in [("XGBoost", xgb_model), ("LightGBM", lgb_model), ("CatBoost", cat_model)]:
cv = cross_val_score(model, X, y, cv=5, scoring="roc_auc", n_jobs=-1)
print(f"{name:10s} AUC: {cv.mean():.4f} ± {cv.std():.4f}")Early Stopping
Python
from sklearn.model_selection import train_test_split
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.1)
xgb_es = xgb.XGBClassifier(n_estimators=1000, learning_rate=0.05, max_depth=5,
eval_metric="auc", random_state=42)
xgb_es.fit(X_tr, y_tr,
eval_set=[(X_val, y_val)],
early_stopping_rounds=30, # stop if no improvement for 30 rounds
verbose=100)
print(f"Best iteration: {xgb_es.best_iteration}")Random Forest vs. Gradient Boosting
Random Forest
- Parallel training — fast on multi-core.
- Very few hyperparameters to tune. Robust defaults.
- Handles noisy data well (averaging).
- Good default for quick baselines.
Gradient Boosting (XGB/LGB)
- Sequential — slower to train.
- More hyperparameters; needs careful tuning.
- Better peak performance on tabular data.
- State-of-the-art for most Kaggle tabular tasks.
Summary
- Bagging: parallel training on bootstrap samples → reduces variance. Random Forest adds feature subsampling to de-correlate trees further.
- Gradient Boosting: sequential residual fitting → reduces bias. XGBoost adds regularisation on tree structure itself.
- LightGBM: leaf-wise splitting, faster training. CatBoost: native categoricals, great defaults.
- Always use early stopping with a validation set for boosting models.