10 - Feedforward Neural Networks: From Perceptron to Deep Learning

A single artificial neuron draws a straight line, and a line cannot solve XOR, the function that is 1 when the two inputs are different: no line separates (0,1) and (1,0) from (0,0) and (1,1). Add an intermediate layer and the problem is solved with eight parameters.

A feedforward neural network, or Multi-Layer Perceptron (MLP), is a stack of layers in which each layer computes a linear combination of its input and then applies a non-linear activation function; «feedforward» means that information always flows forward, without cycles. The weights are found with backpropagation, which computes for every weight the gradient (how much the error changes if the weight changes) by applying the chain rule from the output toward the input.

This article, the tenth in the ML Fundamentals series, shows each of these pieces with a computation you can redo: the perceptron that fails, XOR solved by hand, a gradient checked in three ways, and then two experiments with PyTorch on why deep networks are hard to train and what really reduces overfitting.

The more extended versions of these topics are in the Machine Learning Engineer series, which devotes one article to the perceptron and activation functions, one to backpropagation derived in full and one to a network from scratch in NumPy and then in PyTorch. Here the goal is narrower: to understand where the leap from the linear model to the deep one lies, in a way you can verify in a minute of computation. You need to have seen logistic regression, which is a single neuron with a sigmoid activation.

The perceptron and the problem it cannot solve

Rosenblatt's perceptron computes $\hat{y} = 1$ if $w \cdot x + b > 0$, otherwise 0. The learning rule is a single line: if the prediction is wrong, move the weight vector toward the example, $w \leftarrow w + (y - \hat{y})\, x$, and do the same with $b$.

For data that a line can separate the rule converges in a finite number of steps (the perceptron convergence theorem, which I do not prove here). The listing applies it to two logical functions on the four points of the square: AND, which is separable, and XOR, which is not. At the end it also tries 15,625 different lines by brute force on a grid of weights and biases, to see what the maximum number of XOR points a line can classify correctly is.

# Rosenblatt's perceptron: it learns AND, it cannot learn XOR
import itertools
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
functions = {"AND": np.array([0, 0, 0, 1]), "XOR": np.array([0, 1, 1, 0])}

def perceptron(X, y, max_epochs=100):
    """Rule: if you are wrong, w <- w + (y - y_pred) * x. Stop when you are no longer wrong."""
    w, b = np.zeros(2), 0.0
    for epoch in range(1, max_epochs + 1):
        errors = 0
        for xi, yi in zip(X, y):
            pred = 1 if w @ xi + b > 0 else 0
            if pred != yi:
                w, b = w + (yi - pred) * xi, b + (yi - pred)
                errors += 1
        if errors == 0:
            return w, b, epoch
    return w, b, None

for name, y in functions.items():
    w, b, ep = perceptron(X, y)
    pred = (X @ w + b > 0).astype(int)
    outcome = f"converges at epoch {ep}" if ep else "does NOT converge in 100 epochs"
    print(f"{name}: {outcome}; w = {w.tolist()}, b = {b}; correct points = {(pred == y).sum()}/4")

# Exhaustive test: no line w1*x1 + w2*x2 + b > 0 classifies all 4 XOR points correctly
grid = np.round(np.arange(-3, 3.01, 0.25), 2)
best = 0
for w1, w2, b in itertools.product(grid, grid, grid):
    pred = (X @ np.array([w1, w2]) + b > 0).astype(int)
    best = max(best, int((pred == functions["XOR"]).sum()))
print(f"XOR: maximum of correct points over {len(grid) ** 3} lines tried = {best}/4")

# Output:
# AND: converges at epoch 6; w = [2.0, 1.0], b = -2.0; correct points = 4/4
# XOR: does NOT converge in 100 epochs; w = [-1.0, 0.0], b = 1.0; correct points = 2/4
# XOR: maximum of correct points over 15625 lines tried = 3/4

On AND the perceptron converges at the sixth epoch, with $w = (2,1)$ and $b = -2$: the line passes between (1,1) and the other three points. On XOR after 100 epochs it has not converged, and it stops with 2 points right out of 4; the exhaustive search confirms that this is not a limit of the rule, but of the model: the maximum obtainable with a line over 15,625 attempts is 3 points out of 4. This is the historical reason why more layers are needed: a combination of lines can draw regions that a single line cannot.

One hidden layer, the forward pass and backpropagation

With a hidden layer of two ReLU units ($\max(0, z)$) XOR is solved with hand-written weights, those that the book by Goodfellow, Bengio and Courville uses in chapter 6: $h = \max(0, Wx + c)$ with $W = \begin{pmatrix}1&1\\1&1\end{pmatrix}$ and $c = (0,-1)$, then $y = w \cdot h$ with $w = (1,-2)$.

For the input (1,1): $Wx + c = (2, 1)$, so $h = (2,1)$ and $y = 2 - 2 = 0$; for (0,1): $h = (1, 0)$ and $y = 1$. The first hidden neuron counts how many inputs are on, the second turns on only if both are on and has twice its value subtracted: that way the case (1,1) goes back to zero.

To find the weights without guessing them you need the gradient. For a 3-4-1 network with $\tanh$ activation and loss $L = \tfrac{1}{2N}\sum (y - t)^2$, the chain rule gives, from output to input: $\delta_y = (y - t)/N$, then $\partial L/\partial W_2 = h^\top \delta_y$, then you propagate $\delta_h = \delta_y W_2^\top$, you go through the $\tanh$ by multiplying by its derivative $1 - h^2$, and finally $\partial L/\partial W_1 = x^\top \delta_a$.

The second half of the listing implements these lines and compares them with two independent references: PyTorch's autograd, in double precision, and the finite differences, that is $(L(w+\epsilon) - L(w-\epsilon))/2\epsilon$ computed weight by weight.

# 1) The 2-2-1 ReLU network that solves XOR by hand; 2) hand-written backpropagation against autograd and finite differences
import numpy as np
import torch

torch.set_num_threads(2)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)

# Hand-chosen weights: h = ReLU(W x + c), y = w . h + b
W = np.array([[1.0, 1.0], [1.0, 1.0]]); c = np.array([0.0, -1.0]); w = np.array([1.0, -2.0]); b = 0.0
H = np.maximum(0, X @ W.T + c)
print("hidden activations for the 4 inputs:\n", H.tolist())
print("outputs:", (H @ w + b).tolist(), "  (expected XOR: [0, 1, 1, 0])")

# --- Hand backpropagation on a 3 -> 4 (tanh) -> 1 (linear) network, MSE loss, in float64 ---
rng = np.random.default_rng(0)
x = rng.normal(size=(5, 3)); t = rng.normal(size=(5, 1))
W1, b1 = rng.normal(size=(3, 4)) * 0.5, np.zeros(4)
W2, b2 = rng.normal(size=(4, 1)) * 0.5, np.zeros(1)

def forward(W1, b1, W2, b2):
    a = x @ W1 + b1; h = np.tanh(a); y = h @ W2 + b2
    return a, h, y, 0.5 * ((y - t) ** 2).mean()

a, h, y, L = forward(W1, b1, W2, b2)
dy = (y - t) / len(x)                       # dL/dy
gW2, gb2 = h.T @ dy, dy.sum(axis=0)         # output layer
dh = dy @ W2.T                              # propagated backwards
da = dh * (1 - h ** 2)                      # derivative of tanh: 1 - tanh^2
gW1, gb1 = x.T @ da, da.sum(axis=0)

# PyTorch autograd
P = [torch.tensor(v, dtype=torch.float64, requires_grad=True) for v in (W1, b1, W2, b2)]
yy = torch.tanh(torch.tensor(x) @ P[0] + P[1]) @ P[2] + P[3]
(0.5 * ((yy - torch.tensor(t)) ** 2).mean()).backward()
print("max |hand gradient - autograd|:", max(float(np.abs(g - p.grad.numpy()).max()) for g, p in zip((gW1, gb1, gW2, gb2), P)))

# finite differences on the W1 weights: (L(w+e) - L(w-e)) / 2e
e = 1e-6; num = np.zeros_like(W1)
for i in range(W1.shape[0]):
    for j in range(W1.shape[1]):
        Wp, Wm = W1.copy(), W1.copy(); Wp[i, j] += e; Wm[i, j] -= e
        num[i, j] = (forward(Wp, b1, W2, b2)[3] - forward(Wm, b1, W2, b2)[3]) / (2 * e)
print("max |hand gradient - finite differences| on W1:", f"{np.abs(gW1 - num).max():.1e}")

# Output:
# hidden activations for the 4 inputs:
#  [[0.0, 0.0], [1.0, 0.0], [1.0, 0.0], [2.0, 1.0]]
# outputs: [0.0, 1.0, 1.0, 0.0]   (expected XOR: [0, 1, 1, 0])
# max |hand gradient - autograd|: 5.551115123125783e-17
# max |hand gradient - finite differences| on W1: 4.4e-11

The four outputs are 0, 1, 1, 0 as XOR requires, and the hidden activations coincide with the hand computation (for (1,1) they are (2,1)). The hand gradient differs from PyTorch's by 5.6e-17, that is by rounding, and from the finite differences by 4.4e-11, an error of the order of 1e-10, which is the limit imposed by rounding with $\epsilon = 10^{-6}$.

Finite differences are too slow to train a real network (one loss computation per weight) but they are the right tool to check a hand-written backpropagation: if the comparison does not match, the bug is in your code and not in the method.

Why a deep network does not get started: the gradients

In the gradient just seen there appear products of weights and of activation derivatives, one for each layer crossed. With many layers, if those factors are on average smaller than 1 the gradient of the first layers dies out (vanishing gradient), if they are larger than 1 it explodes. The sigmoid has a derivative of at most 0.25, so in a deep stack each layer cuts the gradient by at least a factor of four.

Glorot and Bengio (2010) find that the logistic sigmoid is not suited to deep networks with random initialization because of its mean value, which can drive the top layer into saturation, and they propose an initialization that preserves the variance of the signals. He and colleagues (2015) correct it for ReLUs: since the ReLU zeroes half of the values, the condition becomes $\tfrac{1}{2} n\, \mathrm{Var}[w] = 1$, that is a standard deviation of $\sqrt{2/n}$.

The listing builds a network of 10 layers of 64 units and measures, at initialization and with a single random batch, the mean magnitude of the weight gradient in the first and in the tenth layer, for five combinations of activation and initialization; then it measures the standard deviation of the activation at the tenth layer for three initializations of the ReLU. The initializations are those of torch.nn.init: xavier_uniform_ (Glorot), kaiming_normal_ with nonlinearity="relu" (He) and a Gaussian with a standard deviation of 0.01.

# Vanishing gradient: 10-layer network with 64 units, mean gradient per layer, at initialization
import numpy as np
import torch
from torch import nn

torch.set_num_threads(2)

def network(activation, init, depth=10, width=64):
    layers = []
    for i in range(depth):
        lin = nn.Linear(width if i else 20, width)
        init(lin.weight); nn.init.zeros_(lin.bias)
        layers += [lin, activation()]
    return nn.Sequential(*layers, nn.Linear(width, 2))

def measure(name, activation, init):
    torch.manual_seed(0)
    net = network(activation, init)
    x = torch.randn(256, 20); y = torch.randint(0, 2, (256,))
    nn.functional.cross_entropy(net(x), y).backward()
    lin = [m for m in net if isinstance(m, nn.Linear)]
    g = [float(m.weight.grad.abs().mean()) for m in lin]       # mean magnitude of the weight gradient
    print(f"{name:34s} layer 1: {g[0]:.2e}  layer 10: {g[-2]:.2e}  ratio 1/10: {g[0] / g[-2]:.2e}")

xavier = nn.init.xavier_uniform_
he = lambda w: nn.init.kaiming_normal_(w, nonlinearity="relu")
small = lambda w: nn.init.normal_(w, std=0.01)
measure("sigmoid + Xavier", nn.Sigmoid, xavier)
measure("tanh + Xavier", nn.Tanh, xavier)
measure("ReLU + He (std = sqrt(2/n))", nn.ReLU, he)
measure("ReLU + weights N(0, 0.01^2)", nn.ReLU, small)
measure("ReLU + Xavier", nn.ReLU, xavier)

# Variance propagated forward: the standard deviation of the last activation, ReLU, 10 layers
for name, init in [("He", he), ("Xavier", xavier), ("N(0, 0.01^2)", small)]:
    torch.manual_seed(0)
    net = network(nn.ReLU, init)
    h = torch.randn(256, 20)
    for m in list(net)[:-1]:
        h = m(h)
    print(f"standard deviation of the activation at the tenth layer, {name:12s}: {float(h.detach().std()):.3e}")

# Output:
# sigmoid + Xavier                   layer 1: 1.41e-09  layer 10: 3.95e-04  ratio 1/10: 3.56e-06
# tanh + Xavier                      layer 1: 9.23e-04  layer 10: 5.19e-04  ratio 1/10: 1.78e+00
# ReLU + He (std = sqrt(2/n))        layer 1: 1.27e-03  layer 10: 1.83e-03  ratio 1/10: 6.91e-01
# ReLU + weights N(0, 0.01^2)        layer 1: 6.79e-15  layer 10: 1.45e-15  ratio 1/10: 4.69e+00
# ReLU + Xavier                      layer 1: 7.65e-05  layer 10: 1.22e-05  ratio 1/10: 6.29e+00
# standard deviation of the activation at the tenth layer, He          : 5.978e-01
# standard deviation of the activation at the tenth layer, Xavier      : 1.406e-02
# standard deviation of the activation at the tenth layer, N(0, 0.01^2): 1.121e-13

Sigmoid with Xavier shows the vanishing gradient in its pure form: the gradient of the first layer is 1.4e-09, against 4.0e-04 for the tenth, a ratio of 3.6e-06. With tanh and Xavier, for which Glorot and Bengio had designed the initialization, the two layers have similar gradients (9.2e-04 and 5.2e-04). ReLU with He preserves both the gradient (1.3e-03 against 1.8e-03) and the activation: the standard deviation at the tenth layer is 0.598.

With ReLU and too small weights (0.01) everything is of the order of 1e-15 and 1e-13: the ratio between the two layers is 4.7, of the same order, but the network cannot learn because the signal has vanished from the start. ReLU with Xavier sits in the middle, with an activation of 0.014 at the tenth layer: the choice of initialization must go with the activation, not be a constant. This is a measurement of one network and one batch, which illustrates the mechanism without claiming to quantify it in general.

Training a network: three optimizers on the same digits

The optimizer decides how to use the gradient. Stochastic gradient descent (SGD) moves the weights by $-\eta\, g$, where $\eta$ is the learning rate. With momentum it keeps a moving average of past gradients, and damps oscillations. Adam (Kingma and Ba, 2014) adapts the step for every weight using moving averages of the gradient and of its square.

The listing trains the same 64-64-10 network with ReLU on the 1,797 8x8 images of load_digits (1,257 for training, 540 for testing, pixels rescaled to 0..1) with the same initialization and the same sequence of mini-batches of 64, for 40 epochs, and changes only the optimizer. As a reference it prints the accuracy of a logistic regression.

# An MLP 64-64-10 on digits with three optimizers, same initialization, same 40 epochs
import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

torch.set_num_threads(2)
X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X / 16.0, y, test_size=0.3, random_state=0, stratify=y)
Xtr_t, ytr_t = torch.tensor(Xtr, dtype=torch.float32), torch.tensor(ytr)
Xte_t, yte_t = torch.tensor(Xte, dtype=torch.float32), torch.tensor(yte)

def train(name, opt_fn, epochs=40, batch=64):
    torch.manual_seed(0)
    net = nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 10))
    opt = opt_fn(net.parameters())
    g = torch.Generator().manual_seed(0)
    history = []
    for ep in range(epochs):
        perm = torch.randperm(len(Xtr_t), generator=g)
        for i in range(0, len(perm), batch):
            idx = perm[i:i + batch]
            opt.zero_grad()
            nn.functional.cross_entropy(net(Xtr_t[idx]), ytr_t[idx]).backward()
            opt.step()
        if ep + 1 in (1, 5, 10, 20, 40):
            with torch.no_grad():
                history.append(float(nn.functional.cross_entropy(net(Xtr_t), ytr_t)))
    with torch.no_grad():
        acc = (net(Xte_t).argmax(1) == yte_t).float().mean().item()
    print(f"{name:28s} training loss at epochs 1/5/10/20/40: " + " ".join(f"{v:.3f}" for v in history) + f"   test accuracy {acc:.3f}")

train("SGD lr=0.05", lambda p: torch.optim.SGD(p, lr=0.05))
train("SGD lr=0.05 momentum=0.9", lambda p: torch.optim.SGD(p, lr=0.05, momentum=0.9))
train("Adam lr=0.003", lambda p: torch.optim.Adam(p, lr=0.003))

lr = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
print(f"reference, logistic regression: test accuracy {lr.score(Xte, yte):.3f} (540 test images)")

# Output:
# SGD lr=0.05                  training loss at epochs 1/5/10/20/40: 2.250 1.901 1.138 0.458 0.217   test accuracy 0.950
# SGD lr=0.05 momentum=0.9     training loss at epochs 1/5/10/20/40: 1.914 0.195 0.105 0.077 0.024   test accuracy 0.978
# Adam lr=0.003                training loss at epochs 1/5/10/20/40: 1.807 0.334 0.159 0.086 0.034   test accuracy 0.978
# reference, logistic regression: test accuracy 0.970 (540 test images)

Plain descent with $\eta = 0.05$ is still at loss 0.217 after 40 epochs, with test accuracy 0.950: it has not finished descending yet. Momentum, at the same learning rate, brings the loss to 0.024 and the accuracy to 0.978; Adam with $\eta = 0.003$ reaches 0.034 and the same 0.978.

Logistic regression, a single layer, obtains 0.970 on the same test set: the network beats it by 0.8 points, that is four images out of 540. So for these digits a layered network is not a leap in quality, and the difference between optimizers is much larger than that between network and linear model. With a single seed you cannot say that SGD with momentum and Adam are different: they are closer than the uncertainty that a single training leaves you with.

Overfitting: what really reduces it

A network with thousands of weights can memorize the training set: the training accuracy reaches 1.000 and the test accuracy stays lower, and the difference is overfitting. The two most used regularization techniques are dropout, which at every step randomly switches off a fraction of the neurons, and weight decay, which penalizes large weights (in AdamW it is a term separate from the gradient).

The listing has two parts. In the first it trains a 64-256-256-10 network on 200 images, with and without these techniques and with two seeds, and compares the accuracy on 500 test images never touched. In the second, without regularization, it increases the amount of training data from 50 to 1,297 images.

# Overfitting: the 64-256-256-10 network memorizes 200 images. What really reduces it?
import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

torch.set_num_threads(2)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32); y = torch.tensor(y)
idx_tr_all, idx_te = train_test_split(np.arange(len(X)), test_size=500, random_state=0, stratify=y.numpy())
Xte, yte = X[idx_te], y[idx_te]                       # 500 test images, never touched

def trial(n, dropout=0.0, wd=0.0, seed=0, epochs=100):
    idx = idx_tr_all[:n]                              # the first n (already shuffled by the split)
    Xtr, ytr = X[idx], y[idx]
    torch.manual_seed(seed)
    net = nn.Sequential(nn.Linear(64, 256), nn.ReLU(), nn.Dropout(dropout),
                        nn.Linear(256, 256), nn.ReLU(), nn.Dropout(dropout), nn.Linear(256, 10))
    opt = torch.optim.AdamW(net.parameters(), lr=0.003, weight_decay=wd)
    g = torch.Generator().manual_seed(seed)
    for ep in range(epochs):
        net.train()
        perm = torch.randperm(n, generator=g)
        for i in range(0, n, 50):
            b = perm[i:i + 50]
            opt.zero_grad()
            nn.functional.cross_entropy(net(Xtr[b]), ytr[b]).backward()
            opt.step()
    net.eval()
    with torch.no_grad():
        return (net(Xtr).argmax(1) == ytr).float().mean().item(), (net(Xte).argmax(1) == yte).float().mean().item()

print("A) with 200 training images, 2 seeds (test accuracy, mean and range)")
for name, kw in [("no regularization", {}), ("dropout 0.5", {"dropout": 0.5}),
                 ("weight decay 1.0 (AdamW)", {"wd": 1.0}), ("dropout 0.5 + weight decay 1.0", {"dropout": 0.5, "wd": 1.0})]:
    r = np.array([trial(200, seed=s, **kw) for s in range(2)])
    print(f"   {name:32s} train {r[:, 0].mean():.3f}  test {r[:, 1].mean():.3f} ({r[:, 1].min():.3f}-{r[:, 1].max():.3f})")

print("B) without regularization, as the training data grow (1 seed)")
for n in (50, 100, 200, 400, 800, 1297):
    tr, te = trial(n)
    print(f"   n = {n:5d}  train {tr:.3f}  test {te:.3f}  gap {tr - te:+.3f}")

# Output:
# A) with 200 training images, 2 seeds (test accuracy, mean and range)
#    no regularization                train 1.000  test 0.935 (0.932-0.938)
#    dropout 0.5                      train 1.000  test 0.938 (0.936-0.940)
#    weight decay 1.0 (AdamW)         train 1.000  test 0.937 (0.936-0.938)
#    dropout 0.5 + weight decay 1.0   train 1.000  test 0.932 (0.926-0.938)
# B) without regularization, as the training data grow (1 seed)
#    n =    50  train 1.000  test 0.684  gap +0.316
#    n =   100  train 1.000  test 0.882  gap +0.118
#    n =   200  train 1.000  test 0.932  gap +0.068
#    n =   400  train 1.000  test 0.958  gap +0.042
#    n =   800  train 1.000  test 0.966  gap +0.034
#    n =  1297  train 1.000  test 0.980  gap +0.020

In part A the training is always at 1.000 and the test differences range from 0.932 to 0.938 (ranges of two seeds, which overlap): on these data and with this network, neither dropout 0.5 nor weight decay 1.0 moves the accuracy beyond the oscillation between seeds. In part B the effect of the data is instead enormous: 0.684 with 50 images, 0.882 with 100, 0.932 with 200, 0.980 with 1,297, and the gap between training and test falls from 0.316 to 0.020.

You cannot conclude that dropout «is useless»: the experiment concerns one network, one dataset and one setup, and the techniques should matter more with larger networks or noisy data. The result that holds is another: before adding a regularization it is worth knowing how much you would gain with more data, and this is usually measured with a learning curve like part B. The article on overfitting and regularization and the one on dropout and batch normalization of the Machine Learning Engineer series go deeper into both techniques.

Worked exercise

A 2-2-1 network has $W = \begin{pmatrix}1&-1\\2&0\end{pmatrix}$, $c = (0,-1)$, $w = (1,3)$, $b = 0.5$, ReLU activation, input $x = (1,2)$. Compute by hand $z$, $h$, $y$; then the gradients of $y$ with respect to $w$, $W$ and $c$. There is one neuron that does not learn: which one, and why? Then check with the listing.

# Exercise: forward pass and gradients of a 2-2-1 network with ReLU
import torch

W = torch.tensor([[1.0, -1.0], [2.0, 0.0]], requires_grad=True)
c = torch.tensor([0.0, -1.0], requires_grad=True)
w = torch.tensor([1.0, 3.0], requires_grad=True)
b = torch.tensor(0.5, requires_grad=True)
x = torch.tensor([1.0, 2.0])

z = W @ x + c
h = torch.relu(z)
y = w @ h + b
y.backward()
print("z =", z.tolist(), " h =", h.tolist(), " y =", y.item())
print("dy/dw =", w.grad.tolist(), " dy/db =", b.grad.item())
print("dy/dW =", W.grad.tolist())
print("dy/dc =", c.grad.tolist())

# Output:
# z = [-1.0, 1.0]  h = [0.0, 1.0]  y = 3.5
# dy/dw = [0.0, 1.0]  dy/db = 1.0
# dy/dW = [[0.0, 0.0], [3.0, 6.0]]
# dy/dc = [0.0, 3.0]

What you can do now

After this article you can do five things, all with a check at hand:

• explain why a single neuron does not solve XOR and build by hand the network that solves it;
• write the forward pass and the backpropagation of a small network and check them with autograd and with finite differences;
• recognize the vanishing gradient and choose initialization (Xavier or He) and activation consistently;
• train a network with SGD, momentum or Adam knowing what changes;
• tell the effect of regularization from that of the data, with a learning curve.

These five points connect in a single sentence: a network is a composite function, the gradient is the chain rule applied to that composition, and almost all training problems are problems of the scale of the numbers that cross the composition. In the next article we move from training a model to judging it: the evaluation metrics. To put it into operation there is then deployment; earlier in the series there were PCA and K-Means.

The same series also includes: 01 - Introduction to Machine Learning: Taxonomy and Workflow and 02 - Linear Regression from Scratch in Python.