Advanced DL: CNNs & RNNs

Beyond Fully-Connected Networks

MLPs treat every input feature equally — a pixel in the top-left corner of an image has no special relationship to its neighbours, and word position in a sentence carries no meaning. Specialised architectures encode these structural priors directly into the model.

Convolutional Neural Networks (CNNs)

CNNs exploit two properties of images: local correlations (nearby pixels are related) and translation invariance (a cat is a cat regardless of where in the image it appears). They achieve this via learnable convolutional filters.

Convolution Operation$$(\mathbf{I} * \mathbf{K})_{ij} = \sum_{m}\sum_{n} \mathbf{I}_{i+m,\, j+n} \cdot \mathbf{K}_{m,n}$$$\mathbf{I}$: input feature map. $\mathbf{K}$: learnable kernel/filter (e.g., 3×3). Each filter slides across the input, computing dot products at every position.

Conv LayerApplies $F$ learnable filters. Each filter detects one feature (edge, curve, texture). Output: $F$ feature maps.

PoolingMax or average pooling: spatially downsamples feature maps. Reduces computation; adds translation invariance.

Flatten + DenseAfter conv/pool layers: flatten to 1D, pass through Dense layers for final classification/regression.

Python

import tensorflow as tf
from tensorflow import keras
import numpy as np

# ── CNN for MNIST digit classification
(X_tr, y_tr), (X_te, y_te) = keras.datasets.mnist.load_data()
X_tr = X_tr[..., np.newaxis] / 255.0    # (60000,28,28,1) float32
X_te = X_te[..., np.newaxis] / 255.0

cnn = keras.Sequential([
    keras.layers.Input(shape=(28, 28, 1)),

    # Block 1: extract low-level features (edges)
    keras.layers.Conv2D(32, kernel_size=3, padding="same", activation="relu"),
    keras.layers.Conv2D(32, kernel_size=3, padding="same", activation="relu"),
    keras.layers.MaxPooling2D(pool_size=2),      # (14,14,32)
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.25),

    # Block 2: extract higher-level features (shapes)
    keras.layers.Conv2D(64, kernel_size=3, padding="same", activation="relu"),
    keras.layers.Conv2D(64, kernel_size=3, padding="same", activation="relu"),
    keras.layers.MaxPooling2D(pool_size=2),      # (7,7,64)
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.25),

    # Classifier head
    keras.layers.Flatten(),
    keras.layers.Dense(256, activation="relu"),
    keras.layers.Dropout(0.5),
    keras.layers.Dense(10, activation="softmax"),   # 10 digit classes
])

cnn.compile(
    optimizer=keras.optimizers.Adam(1e-3),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)
cnn.summary()

cnn.fit(X_tr, y_tr, validation_split=0.1, epochs=10, batch_size=128,
        callbacks=[keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)])

loss, acc = cnn.evaluate(X_te, y_te, verbose=0)
print(f"Test accuracy: {acc:.4f}")    # typically > 99%

Data Augmentation for Images

Python

# Keras built-in augmentation layer (runs on GPU, no extra code at inference)
augment = keras.Sequential([
    keras.layers.RandomFlip("horizontal"),
    keras.layers.RandomRotation(0.1),
    keras.layers.RandomZoom(0.1),
    keras.layers.RandomTranslation(0.1, 0.1),
], name="augmentation")

# Add at the start of the model
cnn_augmented = keras.Sequential([augment] + cnn.layers[1:])

Recurrent Neural Networks (RNNs)

MLPs and CNNs have no memory — each input is processed independently. RNNs process sequences by maintaining a hidden state that propagates information across time steps.

RNN Hidden State Update$$\mathbf{h}_t = \tanh(\mathbf{W}_h \mathbf{h}_{t-1} + \mathbf{W}_x \mathbf{x}_t + \mathbf{b})$$$\mathbf{h}_t$: hidden state at step $t$ (the "memory"). $\mathbf{x}_t$: current input. Problem: gradients vanish through long sequences → simple RNNs can't learn long-range dependencies.

LSTM — Long Short-Term Memory

LSTMs solve the vanishing gradient problem with a cell state $c_t$ — a "conveyor belt" of information that flows through the sequence with only small, controlled modifications at each step, controlled by three gates:

Forget gate $f_t$Decides what to erase from cell state. $f_t = \sigma(W_f[h_{t-1}, x_t] + b_f)$

Input gate $i_t$Decides what new information to write. $i_t = \sigma(W_i[h_{t-1}, x_t] + b_i)$

Output gate $o_t$Decides what to expose as hidden state. $o_t = \sigma(W_o[h_{t-1}, x_t] + b_o)$

Python

import numpy as np
import tensorflow as tf
from tensorflow import keras

# ── Time-series: predict next value from 30-step windows
np.random.seed(42)
t     = np.linspace(0, 200, 2000)
signal = np.sin(t) + 0.3 * np.sin(3*t) + 0.1 * np.random.randn(2000)

# Create sliding windows: (X[t:t+30], y[t+30])
SEQ_LEN = 30
X_seq = np.array([signal[i:i+SEQ_LEN] for i in range(len(signal)-SEQ_LEN)])
y_seq = signal[SEQ_LEN:]
X_seq = X_seq[..., np.newaxis]    # (N, 30, 1)

split = int(0.8 * len(X_seq))
X_tr, X_te = X_seq[:split], X_seq[split:]
y_tr, y_te = y_seq[:split], y_seq[split:]

# ── LSTM model
lstm_model = keras.Sequential([
    keras.layers.Input(shape=(SEQ_LEN, 1)),
    keras.layers.LSTM(64, return_sequences=True),   # return_sequences=True: pass full output to next LSTM
    keras.layers.Dropout(0.2),
    keras.layers.LSTM(32),                          # last LSTM: return final hidden state only
    keras.layers.Dense(16, activation="relu"),
    keras.layers.Dense(1),                          # predict next value
])
lstm_model.compile(optimizer="adam", loss="mse", metrics=["mae"])
lstm_model.fit(X_tr, y_tr, validation_split=0.1, epochs=30,
               batch_size=64, verbose=0,
               callbacks=[keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)])

mse = lstm_model.evaluate(X_te, y_te, verbose=0)[0]
print(f"LSTM Test MSE: {mse:.6f}")

GRU — Gated Recurrent Unit

GRU is a simplified LSTM with just two gates (update + reset). Comparable performance to LSTM on most tasks, faster to train (fewer parameters). Good default when LSTM feels like overkill.

Python

gru_model = keras.Sequential([
    keras.layers.Input(shape=(SEQ_LEN, 1)),
    keras.layers.GRU(64, return_sequences=True),
    keras.layers.GRU(32),
    keras.layers.Dense(1),
])
gru_model.compile(optimizer="adam", loss="mse")
gru_model.fit(X_tr, y_tr, epochs=30, batch_size=64, verbose=0,
              callbacks=[keras.callbacks.EarlyStopping(patience=5)])

Transfer Learning (Practical Tip)

Training CNNs from scratch requires millions of images. Transfer learning uses a model pre-trained on ImageNet (e.g., MobileNetV2, EfficientNet) as a feature extractor — freeze its weights, add your own classification head, and fine-tune on your small dataset.

CopyPython

base = keras.applications.MobileNetV2(weights="imagenet", include_top=False,
                                       input_shape=(128, 128, 3))
base.trainable = False     # freeze pretrained weights

model_tl = keras.Sequential([
    base,
    keras.layers.GlobalAveragePooling2D(),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dropout(0.3),
    keras.layers.Dense(num_classes, activation="softmax"),
])

Summary

CNNs: convolutional filters learn local features. Pooling adds translation invariance. Default for images.
RNNs: hidden state carries memory across time steps. Vanishing gradient limits range.
LSTMs: cell state + 3 gates solve vanishing gradients. Default for sequences (text, time-series).
GRUs: lighter LSTM variant. Good default when training speed matters.
Transfer learning: fine-tune ImageNet models for new vision tasks. Works well with as little as 1000 images.