Network Architecture

From Perceptron to Multi-Layer Network

A single neuron can only draw a straight line. A Multi-Layer Perceptron (MLP) stacks layers of neurons, each applying a non-linear transformation. With just one hidden layer and enough neurons, an MLP can approximate any continuous function — the Universal Approximation Theorem.

Anatomy of a Network

Input LayerOne node per feature in $\mathbf{x}$. No computation — just passes inputs forward. Size = number of features.

Hidden LayersOne or more layers of neurons with activation functions. Each layer learns increasingly abstract features. The depth of the network = number of hidden layers.

Output Layer1 neuron (regression or binary classification), $K$ neurons (multiclass). Activation: linear (regression), sigmoid (binary), softmax (multiclass).

Forward Pass (L-layer network)$$\mathbf{Z}^{[l]} = \mathbf{W}^{[l]}\mathbf{A}^{[l-1]} + \mathbf{b}^{[l]} \qquad \mathbf{A}^{[l]} = g^{[l]}(\mathbf{Z}^{[l]})$$$\mathbf{W}^{[l]}$: weight matrix of layer $l$. $\mathbf{A}^{[0]} = \mathbf{X}$. The output prediction is $\hat{\mathbf{y}} = \mathbf{A}^{[L]}$.

Building MLPs with Keras

Python

import tensorflow as tf
from tensorflow import keras
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
import numpy as np

# Data
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler()
X_tr = scaler.fit_transform(X_tr)
X_te = scaler.transform(X_te)

# ── Sequential API — clean, readable, sufficient for most MLPs
model = keras.Sequential([
    keras.layers.Input(shape=(X_tr.shape[1],)),                # 30 features
    keras.layers.Dense(128, activation="relu",
                       kernel_initializer="he_normal",
                       kernel_regularizer=keras.regularizers.l2(1e-4)),
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.3),                                  # drop 30% during training
    keras.layers.Dense(64, activation="relu",
                       kernel_initializer="he_normal"),
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(1, activation="sigmoid"),                # binary output
])

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss="binary_crossentropy",
    metrics=["accuracy", keras.metrics.AUC(name="auc")]
)
model.summary()

# ── Train with callbacks
callbacks = [
    keras.callbacks.EarlyStopping(monitor="val_auc", patience=15, restore_best_weights=True, mode="max"),
    keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5, patience=7, verbose=1),
]
history = model.fit(
    X_tr, y_tr,
    validation_split=0.15,
    epochs=200,
    batch_size=32,
    callbacks=callbacks,
    verbose=0,
)
loss, acc, auc = model.evaluate(X_te, y_te, verbose=0)
print(f"
Test — Loss: {loss:.4f}  Accuracy: {acc:.4f}  AUC: {auc:.4f}")

Plotting Training History

Python

fig, axes = plt.subplots(1, 2, figsize=(12, 4))
for ax, metric, title in zip(axes, ["loss","auc"], ["Loss","AUC"]):
    ax.plot(history.history[metric],     label=f"Train {title}", color="#4a8fa8", lw=2)
    ax.plot(history.history[f"val_{metric}"], label=f"Val {title}",   color="#b85c2a", lw=2)
    ax.set_title(f"Training History — {title}")
    ax.set_xlabel("Epoch"); ax.legend(); ax.grid(alpha=0.3)
plt.tight_layout(); plt.show()
# Gap between train and val curves → overfitting
# Both high and flat → underfitting / too low learning rate

Neural Network for Regression

Python

from sklearn.datasets import fetch_california_housing
X, y = fetch_california_housing(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
sc = StandardScaler(); X_tr = sc.fit_transform(X_tr); X_te = sc.transform(X_te)

reg_model = keras.Sequential([
    keras.layers.Input(shape=(8,)),
    keras.layers.Dense(256, activation="relu", kernel_initializer="he_normal"),
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dense(64,  activation="relu"),
    keras.layers.Dense(1),                     # linear output for regression
])
reg_model.compile(optimizer="adam", loss="mse", metrics=["mae"])
reg_model.fit(X_tr, y_tr, validation_split=0.1, epochs=100,
              callbacks=[keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)],
              verbose=0)
_, mae = reg_model.evaluate(X_te, y_te, verbose=0)
print(f"Test MAE: {mae:.4f}")

Architecture Design Decisions

Width (neurons/layer)Wider = more capacity per layer. Start with 64–256. Double or halve between layers.

Depth (hidden layers)Deeper = more abstract representations. 1–3 layers for tabular data. More for images/sequences.

Batch NormalisationNormalises layer inputs within each mini-batch. Stabilises training, allows higher learning rates. Use after Dense, before activation.

DropoutRandomly zeros activations during training. Prevents co-adaptation of neurons. Typical rates: 0.2–0.5 for hidden layers.

Summary

MLP = stacked layers of neurons with non-linear activations. Can approximate any continuous function.
Input: one node/feature. Hidden: ReLU + He init + BatchNorm + Dropout. Output: depends on task.
Keras Sequential API: clean way to build standard feed-forward networks.
EarlyStopping + ReduceLROnPlateau: the two most important training callbacks.