Probability & Statistics

Probability: Reasoning Under Uncertainty

Data is noisy. Labels can be wrong. Models are approximations. Probability gives us a rigorous language to quantify and reason about uncertainty — the foundation of everything from Naive Bayes to Bayesian neural networks.

Fundamentals

Kolmogorov Axioms$$P(A) \geq 0 \qquad P(\Omega) = 1 \qquad P(A \cup B) = P(A)+P(B)\; ext{if}\;A\cap B=\emptyset$$

Conditional$P(A|B) = P(A\cap B)/P(B)$ — probability of $A$ given $B$ occurred.

Independence$P(A\cap B) = P(A)P(B)$ — knowing $B$ tells you nothing about $A$.

Total Probability$P(A) = \sum_i P(A|B_i)P(B_i)$ — decompose by exhaustive partition.

Bayes' Theorem

The single most important equation for probabilistic ML:

Bayes' Theorem$$P(H|D) = rac{P(D|H)\,P(H)}{P(D)}$$Prior $ imes$ Likelihood / Evidence = Posterior. Update beliefs as evidence arrives.

Python

# Spam filter example
# 30% of emails are spam. "Free" appears in 80% of spam, 5% of legit.
# Given "Free", what's P(spam)?

p_spam    = 0.30
p_free_spam  = 0.80   # P("Free" | spam)
p_free_legit = 0.05   # P("Free" | not spam)

# Law of total probability: P("Free")
p_free = p_free_spam * p_spam + p_free_legit * (1 - p_spam)

# Bayes update
p_spam_given_free = (p_free_spam * p_spam) / p_free
print(f"P(spam | 'Free') = {p_spam_given_free:.3f}")    # ≈ 0.829

Key Distributions

Python

import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats

rng = np.random.default_rng(42)
fig, axes = plt.subplots(2, 3, figsize=(14, 7))

# Bernoulli — binary outcome (p=0.3)
x = rng.binomial(1, 0.3, 1000)
axes[0,0].bar([0,1], [(x==0).mean(), (x==1).mean()], color=["#e07070","#70a870"])
axes[0,0].set_title("Bernoulli(p=0.3)")

# Binomial — count of successes in 10 trials
x = rng.binomial(10, 0.4, 1000)
axes[0,1].hist(x, bins=11, density=True, color="#4a8fa8", edgecolor="white")
axes[0,1].set_title("Binomial(n=10, p=0.4)")

# Gaussian — bell curve
x_range = np.linspace(-4, 4, 200)
for mu, sigma, col in [(0,1,"#4a8fa8"),(-1,0.5,"#5c8a58"),(1.5,2,"#b85c2a")]:
    axes[0,2].plot(x_range, stats.norm.pdf(x_range, mu, sigma), label=f"μ={mu},σ={sigma}", color=col)
axes[0,2].set_title("Gaussian PDFs"); axes[0,2].legend(fontsize=7)

# Uniform
x = rng.uniform(0, 1, 1000)
axes[1,0].hist(x, bins=30, density=True, color="#c4891a", edgecolor="white")
axes[1,0].set_title("Uniform(0, 1)")

# Exponential
x = rng.exponential(2, 1000)
axes[1,1].hist(x, bins=40, density=True, color="#5c8a58", edgecolor="white")
axes[1,1].set_title("Exponential(λ=0.5)")

# CLT demo — average of uniform samples → Gaussian
sample_means = [rng.uniform(0,1,30).mean() for _ in range(3000)]
axes[1,2].hist(sample_means, bins=40, density=True, color="#4a8fa8", edgecolor="white")
x_r = np.linspace(0.3, 0.7, 200)
axes[1,2].plot(x_r, stats.norm.pdf(x_r, 0.5, 1/np.sqrt(12*30)), color="#b85c2a", lw=2, label="CLT prediction")
axes[1,2].set_title("CLT: mean of 30 Uniforms"); axes[1,2].legend(fontsize=8)

plt.tight_layout(); plt.savefig("distributions.png", dpi=120); plt.show()

PDF and CDF

PDF $f(x)$Probability Density Function. Continuous. $P(a \le X \le b) = \int_a^b f(x)\,dx$. Not a probability itself — it's a density.

PMF $P(X=k)$Probability Mass Function. Discrete. Gives exact probabilities. Sums to 1.

CDF $F(x)$$F(x) = P(X \le x)$. Always exists (discrete or continuous). Monotonically non-decreasing from 0 to 1.

Python

import scipy.stats as stats, numpy as np

dist = stats.norm(loc=0, scale=1)    # standard Gaussian

# PDF and CDF
x = np.linspace(-4, 4, 200)
pdf_vals = dist.pdf(x)              # f(x)
cdf_vals = dist.cdf(x)              # F(x) = P(X ≤ x)

# Key quantities
print(f"P(-1 ≤ Z ≤ 1) = {dist.cdf(1) - dist.cdf(-1):.4f}")   # ≈ 0.6827
print(f"P(-2 ≤ Z ≤ 2) = {dist.cdf(2) - dist.cdf(-2):.4f}")   # ≈ 0.9545
print(f"95th percentile = {dist.ppf(0.95):.4f}")               # ≈ 1.645

Hypothesis Testing

Hypothesis testing answers: is an observed effect real, or could it arise by chance? The framework:

$H_0$ (null hypothesis): no effect, no difference. Default assumption.
$H_a$ (alternative): there is an effect.
Compute a test statistic from the data.
Calculate the p-value: probability of observing data this extreme if $H_0$ is true.
If p-value $< lpha$ (e.g., 0.05), reject $H_0$.

Python

import scipy.stats as stats
import numpy as np

rng = np.random.default_rng(42)

# Does model A significantly outperform model B? (paired t-test)
scores_A = rng.normal(0.85, 0.04, 50)   # model A CV scores
scores_B = rng.normal(0.82, 0.04, 50)   # model B CV scores

t_stat, p_value = stats.ttest_rel(scores_A, scores_B)
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value:     {p_value:.4f}")
print(f"Conclusion:  {'A is significantly better' if p_value < 0.05 else 'No significant difference'}")

# Are two feature distributions the same? (KS test)
feature_train = rng.normal(0, 1, 500)
feature_prod  = rng.normal(0.3, 1.2, 500)   # slightly shifted (data drift!)
ks_stat, ks_p = stats.ks_2samp(feature_train, feature_prod)
print(f"
KS test (distribution drift): stat={ks_stat:.3f}, p={ks_p:.4f}")
print("Data drift detected!" if ks_p < 0.05 else "Distributions look similar")

⚠️

p-value ≠ effect size. With enough data, even a trivially small difference becomes statistically significant. Always report effect size (e.g., Cohen's d) alongside p-values.

Summary

Bayes: $P(H|D) \propto P(D|H)\cdot P(H)$ — posterior ∝ likelihood × prior.
PDF/PMF describes the distribution. CDF = cumulative probability up to $x$.
Gaussian is ubiquitous (CLT). Bernoulli/Binomial for binary outcomes.
Hypothesis testing: p-value is not effect size. Small p + large data ≠ important effect.