Calculus & Gradient Descent

Why Calculus for ML?

Every learning algorithm is an optimisation algorithm. Training a model means finding parameter values that minimise a loss function — and calculus tells us how to move the parameters in the right direction. You don't need to derive integrals; you need to understand derivatives and gradient descent.

Derivatives: Measuring Change

The derivative $f'(x)$ measures the instantaneous rate of change of $f$ at $x$. Geometrically: the slope of the tangent line.

Derivative Definition$$f'(x) = \lim_{h o 0} rac{f(x+h) - f(x)}{h}$$

Python

import numpy as np
import matplotlib.pyplot as plt

# Numerical derivative (finite difference approximation)
def numerical_grad(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)   # central difference

f = lambda x: x**2 - 3*x + 2
print(f"f'(4) ≈ {numerical_grad(f, 4):.4f}")   # should be 2*4-3 = 5.0

# Visualise f and its derivative
x = np.linspace(-1, 5, 300)
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.plot(x, f(x), color="#4a8fa8", lw=2);  ax1.set_title("f(x) = x² - 3x + 2")
ax1.axhline(0, color="gray", lw=0.5);      ax1.axvline(0, color="gray", lw=0.5)
df = 2*x - 3
ax2.plot(x, df, color="#b85c2a", lw=2); ax2.set_title("f'(x) = 2x - 3")
ax2.axhline(0, color="gray", lw=0.5);   ax2.axvline(1.5, color="green", ls="--", label="minimum at x=1.5")
ax2.legend(); plt.tight_layout(); plt.show()

Partial Derivatives

When a function has multiple inputs (like a loss with many parameters), the partial derivative $\partial L/\partial w_j$ measures how $L$ changes when we nudge $w_j$ while holding all other parameters fixed.

Partial Derivative of MSE$$L(\mathbf{w}) = rac{1}{m}\sum_i(\mathbf{w}^T\mathbf{x}^{(i)} - y^{(i)})^2 \qquad rac{\partial L}{\partial w_j} = rac{2}{m}\sum_i(\hat{y}^{(i)} - y^{(i)})\,x_j^{(i)}$$

The Gradient

The gradient $ abla_\mathbf{w} L$ stacks all partial derivatives into a vector. It points in the direction of steepest ascent. To minimise the loss, we step in the opposite direction.

Gradient of MSE$$ abla_\mathbf{w} L = rac{2}{m}\mathbf{X}^T(\mathbf{X}\mathbf{w} - \mathbf{y})$$In matrix form — one line of NumPy code. The gradient is a vector of the same shape as $\mathbf{w}$.

Chain Rule

When functions are composed — as in $L(\hat{y}(\mathbf{w}))$ — the chain rule computes the overall derivative as a product of local derivatives:

Chain Rule$$ rac{dL}{dw} = rac{dL}{d\hat{y}} \cdot rac{d\hat{y}}{dw}$$Backpropagation in neural networks is just the chain rule applied recursively through layers.

Python

import numpy as np

# Manual backprop through: L = MSE(sigmoid(Xw), y)
def sigmoid(z): return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

X = np.random.randn(100, 3)
y = (np.random.randn(100) > 0).astype(float)
w = np.zeros(3)

# Forward pass
z     = X @ w                              # pre-activation: (100,)
p     = sigmoid(z)                         # prediction: (100,)
loss  = -np.mean(y * np.log(p + 1e-9) + (1-y) * np.log(1-p + 1e-9))

# Backward pass (chain rule)
# dL/dp = -(y/p - (1-y)/(1-p)) / m
# dp/dz = p*(1-p)   [sigmoid derivative]
# dz/dw = X
# Combined:  dL/dw = X.T @ (p - y) / m
grad  = X.T @ (p - y) / len(y)

print(f"Loss: {loss:.4f}, Grad norm: {np.linalg.norm(grad):.4f}")

Gradient Descent

Update parameters iteratively in the direction of negative gradient:

Gradient Descent Update$$\mathbf{w} \leftarrow \mathbf{w} - lpha abla_\mathbf{w} L$$$lpha$ (learning rate): controls step size. Too large: overshoots, diverges. Too small: converges slowly.

Python

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(42)
X = np.c_[np.ones(200), np.random.randn(200, 2)]   # add bias col
w_true = np.array([1.0, 2.5, -1.0])
y = X @ w_true + np.random.randn(200) * 0.5

def mse_grad(X, y, w):
    residual = X @ w - y
    return 2 * X.T @ residual / len(y)

# Batch gradient descent
w = np.zeros(3); lr = 0.05; history = []
for i in range(500):
    loss = np.mean((X @ w - y)**2)
    history.append(loss)
    w -= lr * mse_grad(X, y, w)

print(f"Recovered w: {w.round(3)} (true: {w_true})")

plt.plot(history, color="#4a8fa8", lw=2)
plt.xlabel("Iteration"); plt.ylabel("MSE Loss")
plt.title("Gradient Descent Convergence")
plt.yscale("log"); plt.grid(True, alpha=0.3); plt.show()

SGD, Mini-Batch, and Adam

Batch GDUse all $m$ samples per step. Exact gradient, smooth convergence, slow on large data.

SGDUse 1 sample per step. Noisy, but fast. The noise can help escape local minima.

Mini-batch GDUse $b$ samples (e.g., $b=32$). Best of both worlds — used in practice for all deep learning.

AdamAdaptive moment estimation. Maintains per-parameter learning rates. Default choice for neural networks.

Summary

Derivatives measure instantaneous rate of change — the slope at a point.
The gradient $ abla_\mathbf{w}L$ points in the direction of steepest ascent. We step against it to minimise.
Chain rule: compose derivatives through layers of a computational graph → backpropagation.
Gradient descent: $\mathbf{w} \leftarrow \mathbf{w} - lpha abla L$. Mini-batch GD is used in practice.