53 - Dropout and Batch Normalization: regularizing and stabilizing a CNN

Two problems accompany every training of a deep network. The first you know: overfitting, the network that memorizes the training examples instead of learning what generalizes. The second is more subtle: training can be unstable, because at every update the weights of a layer change and so does the distribution of the values that layer passes to the next one, which finds itself learning on a moving target; with learning rates that are not just right the descent stalls or diverges. The two techniques of this lesson are cheap and very widespread remedies: dropout against overfitting and batch normalization against instability.

Prerequisites: you know how to train a network with PyTorch and how to read the norm of a gradient, as in the lesson on residual networks. You will implement the two techniques by hand, check them against PyTorch and measure them. One of the two has a spectacular effect here, the other none that is measurable: understanding why helps you avoid adopting a technique «because that is what one does». The listings were run with PyTorch 2.12.0 and scikit-learn 1.9.0 on CPU, twice in a row, with identical output (measurement of 2026-11-12).

The vocabulary

• Co-adaptation: neurons that learn to work only in combination with each other (one corrects the mistakes of the other), instead of each being useful on its own. It makes the model fragile and favors overfitting.
• Mask: a vector of 0s and 1s that selects which elements to keep.
• Normalizing: subtracting the mean and dividing by the standard deviation, to obtain values with mean 0 and deviation 1.
• Batch statistics: the mean and variance computed on the examples of the current mini-batch only.
• Running statistics: an exponential moving average of the statistics seen in the last batches, used at inference, when there is no batch to compute them from.
• Buffer (in PyTorch): a tensor that is part of a module's state but is not a parameter trained by the gradient (for example the running statistics).

The six items tell three different things. Co-adaptation and mask belong to dropout: the first is the defect to cure, the second is the tool that cures it. Normalizing and batch statistics describe what batch normalization does while training. Running statistics and buffer describe how it behaves afterwards, at inference, and it is the pair that trips people up most often: they are state the model carries around without the gradient updating it.

Dropout: switching neurons off at random

Imagine a study group in which, at every session, some classmates chosen at random stay at home. Each one learns that they cannot rely on «somebody else knowing the hard part»: they must know enough on their own. Dropout (N. Srivastava, G. Hinton et al., 2014) does the same with neurons: at every training step, each neuron of a layer is switched off (its output is zeroed) with probability p, independently of the others. The network that computes the step is therefore a «thinned» version of the full one, different at every mini-batch.

The drawing shows the difference: in training (on the left) some neurons are switched off (red crosses) and the connections to and from them disappear; in evaluation (on the right) all neurons are active. There is a detail not to be missed. If in training a fraction p of the neurons is switched off, the weighted sum reaching the next layer is on average smaller by a factor (1 − p) than the one with all neurons on; if in evaluation all of them were switched on, the next layer would see values larger than those it was trained for. This is solved with the method called inverted dropout: during training the surviving neurons are multiplied by 1/(1 − p), so that the expected value of the output stays the same, and in evaluation nothing is done.

Formally, for every element a of the layer we draw m ∈ {0, 1} with P(m = 1) = 1 − p, and the output is ã = a·m/(1 − p). The expected value is computed in one line:

E[ã] = a · E[m] / (1 − p) = a · (1 − p) / (1 − p) = a                (dropout does not change the expected value)
Var[ã] = a² · Var[m] / (1 − p)² = a² · p(1 − p) / (1 − p)² = a² · p / (1 − p)    (but it adds noise)

With p = 0.5 the added variance is exactly a²: the noise has the same size as the signal. This noise is the mechanism of the regularization. It can also be read as an ensemble of networks: a network with n «switchable» neurons contains 2ⁿ subnetworks, which share the weights; dropout trains, one after the other, a random sample of them, and the full network in evaluation approximates the average of their predictions. It is the same principle as the random forest of the lesson on classical machine learning, obtained with a single model.

Hand implementation and statistical check

import numpy as np
import torch
import torch.nn as nn

class ManualDropout:
    def __init__(self, p, seed=0):
        self.p, self.rng, self.mask = p, np.random.default_rng(seed), None

    def forward(self, x, training):
        if not training or self.p == 0:                                    # in evaluation: identity
            self.mask = None
            return x
        self.mask = (self.rng.random(x.shape) >= self.p) / (1.0 - self.p)    # 0 with prob. p, otherwise 1/(1-p)
        return x * self.mask

    def backward(self, d_out):
        return d_out if self.mask is None else d_out * self.mask           # switched-off neurons receive no gradient

x = np.ones(100000)
for p in (0.2, 0.5, 0.8):
    drop = ManualDropout(p)
    y = drop.forward(x, training=True)
    print("p=%.1f   mean %.3f (expected 1)   variance %.3f (expected p/(1-p) = %.3f)   fraction of zeros %.3f" % (p, y.mean(), y.var(), p / (1 - p), (y == 0).mean()))

print("evaluation: output identical to input:", np.array_equal(ManualDropout(0.5).forward(x, training=False), x))

p=0.2   mean 0.998 (expected 1)   variance 0.251 (expected p/(1-p) = 0.250)   fraction of zeros 0.201
p=0.5   mean 0.998 (expected 1)   variance 1.000 (expected p/(1-p) = 1.000)   fraction of zeros 0.501
p=0.8   mean 0.991 (expected 1)   variance 3.972 (expected p/(1-p) = 4.000)   fraction of zeros 0.802
evaluation: output identical to input: True

What the code does. forward builds the mask by comparing uniform random numbers with p (true with probability 1 − p) and dividing by 1 − p: the surviving elements are worth 1/(1 − p), the switched-off ones 0. backward applies the same mask to the incoming gradient: a switched-off neuron did not influence the output, so it receives no gradient (like a ReLU with zero derivative), and the survivors carry the amplification factor. The experiment on 100,000 ones verifies the formulas: the mean stays around 1 for every p, the variance matches p/(1 − p) (0.25, 1 and 4) and the fraction of zeros matches p. In evaluation the function is the identity.

The experiment: does dropout improve generalization?

We put dropout to the test in the regime where we would expect it to be most useful: a wide network (512 neurons per layer) and little data (100 training examples of Digits), trained for 150 epochs, three dropout levels, eight different seeds. We measure training, validation and test accuracy, and the validation loss.

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X_tmp, X_te, y_tmp, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
X_pool, X_va, y_pool, y_va = train_test_split(X_tmp, y_tmp, test_size=0.5, stratify=y_tmp, random_state=0)

def experiment(p, n_train=100, width=512, seeds=8, epochs=150):
    rows = []
    for seed in range(seeds):
        X_tr, _, y_tr, _ = train_test_split(X_pool, y_pool, train_size=n_train, stratify=y_pool, random_state=seed)
        mean, std = X_tr.mean(), X_tr.std()
        to_t = lambda a: torch.tensor((a - mean) / std, dtype=torch.float32)
        T_tr, T_va, T_te = to_t(X_tr), to_t(X_va), to_t(X_te)
        L_tr, L_va, L_te = [torch.tensor(a) for a in (y_tr, y_va, y_te)]
        torch.manual_seed(seed)
        model = nn.Sequential(nn.Linear(64, width), nn.ReLU(), nn.Dropout(p),
                              nn.Linear(width, width), nn.ReLU(), nn.Dropout(p), nn.Linear(width, 10))
        opt = torch.optim.Adam(model.parameters(), lr=1e-3)
        loss_fn = nn.CrossEntropyLoss()
        loader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(T_tr, L_tr), batch_size=16, shuffle=True)
        for _ in range(epochs):
            model.train()
            for xb, yb in loader:
                opt.zero_grad(); loss_fn(model(xb), yb).backward(); opt.step()
        model.eval()
        with torch.no_grad():
            rows.append([(model(a).argmax(1) == b).float().mean().item() for a, b in ((T_tr, L_tr), (T_va, L_va), (T_te, L_te))]
                        + [loss_fn(model(T_va), L_va).item()])
    return np.array(rows)

print("p     train acc.       validation acc.    test acc.         validation loss")
for p in (0.0, 0.2, 0.5):
    r = experiment(p)
    print("%.1f   %.3f +- %.3f    %.3f +- %.3f      %.3f +- %.3f    %.3f +- %.3f" % ((p,) + tuple(v for k in range(4) for v in (r[:, k].mean(), r[:, k].std()))))

p     train acc.       validation acc.    test acc.         validation loss
0.0   1.000 +- 0.000    0.915 +- 0.011      0.906 +- 0.015    0.368 +- 0.064
0.2   1.000 +- 0.000    0.917 +- 0.010      0.913 +- 0.009    0.367 +- 0.063
0.5   1.000 +- 0.000    0.918 +- 0.012      0.909 +- 0.006    0.378 +- 0.059

What the code does. The experiment function repeats the training with eight different seeds, which change both the 100 examples chosen and the initialization; nn.Dropout(p) is the PyTorch layer that implements inverted dropout (active only in train()). The table says something unwelcome but instructive: in this experiment dropout has no measurable effect at all. With p = 0, 0.2 and 0.5 the test accuracy is 0.906, 0.913 and 0.909, with deviations between seeds of 0.006-0.015; the validation loss is 0.37-0.38 in all three cases. The differences are within the noise. All the networks reach 100% on the training set regardless of dropout.

Why? Digits is an easy, low-dimensional problem, and a 150-epoch training with Adam leads all the versions to memorize the training set; the noise of dropout is not enough to prevent it with 512 neurons per layer. The original paper evaluates it on larger networks and datasets (including convolutional networks on CIFAR and ImageNet), and it is in that regime that it is worth measuring; here the fact is that on the 8×8 digits it is not needed. The rule you take away is more useful than the result: a regularization technique is a hypothesis to measure on your problem, not an obligation.

The most common trap: forgetting eval()

torch.manual_seed(0)
X_tr, _, y_tr, _ = train_test_split(X_pool, y_pool, train_size=250, stratify=y_pool, random_state=0)
mean, std = X_tr.mean(), X_tr.std()
T_tr, T_te = [torch.tensor((a - mean) / std, dtype=torch.float32) for a in (X_tr, X_te)]
L_tr, L_te = torch.tensor(y_tr), torch.tensor(y_te)

model = nn.Sequential(nn.Linear(64, 256), nn.ReLU(), nn.Dropout(0.5), nn.Linear(256, 256), nn.ReLU(), nn.Dropout(0.5), nn.Linear(256, 10))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(T_tr, L_tr), batch_size=32, shuffle=True)
for _ in range(100):
    model.train()
    for xb, yb in loader:
        opt.zero_grad(); nn.CrossEntropyLoss()(model(xb), yb).backward(); opt.step()

def test_accuracy():
    with torch.no_grad():
        return (model(T_te).argmax(1) == L_te).float().mean().item()

model.train()                                                      # eval() FORGOTTEN: dropout is still active
print("train mode (wrong), 5 measurements on the same test set:", np.round([test_accuracy() for _ in range(5)], 3))
model.eval()
print("eval mode  (right), 5 measurements on the same test set:", np.round([test_accuracy() for _ in range(5)], 3))

train mode (wrong), 5 measurements on the same test set: [0.911 0.9   0.908 0.908 0.903]
eval mode  (right), 5 measurements on the same test set: [0.922 0.922 0.922 0.922 0.922]

What the code does. It trains a network with dropout 0.5 and then measures the accuracy on the same test set five times, first leaving the model in train() mode and then in eval() mode. In train mode dropout switches neurons off at random even in evaluation: the five measurements differ from one another (from 0.900 to 0.911) and are on average lower; in eval mode they are identical (0.922). Two unmistakable symptoms: a result that changes from one evaluation to the next on the same data, and an accuracy worse than that of training plus validation. It is the most frequent mistake with dropout, and the cure is the habit: model.eval() before every evaluation, model.train() before every training.

Batch Normalization: normalizing inside the network

In the lesson on regression you saw that standardizing the features makes gradient descent much easier. Batch normalization (S. Ioffe and C. Szegedy, 2015) applies the same idea not only to the input but in the middle of the network: before a nonlinearity, it normalizes each activation using the mean and standard deviation of the mini-batch, and then lets the network learn the scale and shift it prefers. For every feature (or channel) j and a batch of N examples:

μ_j = (1/N) Σ_i x_ij                          batch mean
σ²_j = (1/N) Σ_i (x_ij − μ_j)²                batch variance
x̂_ij = (x_ij − μ_j) / √(σ²_j + ε)              normalization (ε ≈ 10⁻⁵ avoids division by zero)
y_ij = γ_j · x̂_ij + β_j                        LEARNED scale and shift

The two parameters γ (scale) and β (shift) are trained by gradient descent like any weight. They are needed because forcing every activation to mean 0 and deviation 1 would remove expressiveness (for example, a sigmoid in its linear zone would no longer have saturation): with γ and β the network can, if it pays off, cancel the normalization (γ = √(σ² + ε) and β = μ give back the identity).

The drawing shows the three phases: an activation with mean 5 and deviation 3 (on the left), the same after normalization, mean 0 and deviation 1 (in the middle), and after the learned scale and shift (on the right).

Forward and backward by hand, checked against PyTorch

The forward pass follows the formulas. The backward pass requires a little patience, because μ and σ² depend on all the examples of the batch: every x_ij influences all the outputs of its column. Applying the chain rule on this graph gives, with dx̂ = dy·γ:

dγ_j = Σ_i dy_ij · x̂_ij              dβ_j = Σ_i dy_ij
dx_ij = (1/N) · (1/√(σ²_j + ε)) · ( N·dx̂_ij − Σ_k dx̂_kj − x̂_ij · Σ_k dx̂_kj · x̂_kj )

The three terms in the parentheses have a meaning: the first is the «direct» gradient; the second removes the component common to the whole batch (because shifting all the x_ij by the same amount changes nothing after the mean is subtracted); the third removes the component along x̂ (because scaling the whole batch does not change the normalized output). We verify the formula by comparing it with automatic differentiation:

rng = np.random.default_rng(0)
N, C, eps = 32, 4, 1e-5
X = rng.normal(5, 3, size=(N, C))                                    # activations with mean 5 and standard deviation 3
gamma = np.array([1.0, 2.0, 0.5, 1.5]); beta = np.array([0.0, 1.0, -1.0, 0.5])

def bn_forward(X, gamma, beta):
    mu, var = X.mean(axis=0), X.var(axis=0)                          # batch statistics, per column
    xh = (X - mu) / np.sqrt(var + eps)
    return gamma * xh + beta, (xh, var)

def bn_backward(dY, gamma, cache):
    xh, var = cache
    dxh = dY * gamma
    dX = (1.0 / N) / np.sqrt(var + eps) * (N * dxh - dxh.sum(axis=0) - xh * (dxh * xh).sum(axis=0))
    return dX, (dY * xh).sum(axis=0), dY.sum(axis=0)

Y, cache = bn_forward(X, gamma, beta)
print("after BN: mean per column", np.round(Y.mean(axis=0), 3), "  standard deviation per column", np.round(Y.std(axis=0), 3), "(= beta and gamma)")

dY = rng.normal(size=X.shape)
dX, dgamma, dbeta = bn_backward(dY, gamma, cache)

Xt = torch.tensor(X, requires_grad=True)
bn_torch = nn.BatchNorm1d(C, eps=eps).double()
bn_torch.weight.data, bn_torch.bias.data = torch.tensor(gamma), torch.tensor(beta)
out = bn_torch(Xt)
out.backward(torch.tensor(dY))
print("forward matches PyTorch:", np.allclose(Y, out.detach().numpy()))
print("maximum gradient difference (dX, dgamma, dbeta): %.1e  %.1e  %.1e" % (
    np.abs(dX - Xt.grad.numpy()).max(), np.abs(dgamma - bn_torch.weight.grad.numpy()).max(), np.abs(dbeta - bn_torch.bias.grad.numpy()).max()))

after BN: mean per column [ 0.   1.  -1.   0.5]   standard deviation per column [1.  2.  0.5 1.5] (= beta and gamma)
forward matches PyTorch: True
maximum gradient difference (dX, dgamma, dbeta): 2.2e-16  6.2e-15  8.9e-16

What the code does. bn_forward computes the mean and variance along the sample axis (axis=0), normalizes and applies γ and β; it saves x̂ and the variance in the cache for the way back. bn_backward transcribes the gradient formula. After normalization each column has exactly mean β_j and deviation γ_j (the first result), and the forward pass and the gradients coincide with those computed by PyTorch up to rounding errors (10⁻¹⁵). Having derived and verified the formula gives you complete confidence that backpropagation through an operation that involves the whole batch works the way you expect.

Training versus inference: the running statistics

A detail that trips many people. In training BN uses the mean and variance of the current batch. At inference you may have to predict a single example, and a batch of one has no sensible statistics. So during training BN maintains an exponential moving average of μ and σ² (with momentum 0.1: new = 0.9·old + 0.1·batch), and in eval() mode it uses those. The behavior is different in the two modes, and the difference is a trap:

bn = nn.BatchNorm1d(4)
for _ in range(200):                                                         # 200 training batches, data with mean 5 and std. 3
    bn(torch.tensor(rng.normal(5, 3, size=(32, 4)), dtype=torch.float32))
print("running statistics learned: mean", bn.running_mean.numpy().round(2), " variance", bn.running_var.numpy().round(2), "(true: 5 and 9)")

x = torch.tensor(rng.normal(5, 3, size=(2, 4)), dtype=torch.float32)         # a mini-batch of 2 examples
bn.eval()
print("\neval mode (uses the running statistics):\n", bn(x).detach().numpy().round(2))
bn.train()
print("train mode with a batch of 2 (uses the statistics of those 2 examples):\n", bn(x).detach().numpy().round(2))

conv_bn = nn.BatchNorm2d(3)                                                  # for images: one mean per channel over (N, H, W)
img = torch.randn(8, 3, 5, 5) * torch.tensor([1., 5., 10.])[None, :, None, None] + torch.tensor([0., 3., -2.])[None, :, None, None]
y = conv_bn(img)
print("\nBatchNorm2d: per-channel means after normalization", y.mean(dim=(0, 2, 3)).detach().numpy().round(3),
      " trainable parameters:", sum(p.numel() for p in conv_bn.parameters()), "(gamma and beta for 3 channels)",
      " buffers:", [n for n, _ in conv_bn.named_buffers()])

running statistics learned: mean [5.14 4.94 4.91 4.97]  variance [8.98 8.43 9.07 8.58] (true: 5 and 9)

eval mode (uses the running statistics):
 [[-1.47 -1.1  -0.79 -1.43]
 [ 0.21 -1.18  0.32 -0.64]]
train mode with a batch of 2 (uses the statistics of those 2 examples):
 [[-1.  1. -1. -1.]
 [ 1. -1.  1.  1.]]

BatchNorm2d: per-channel means after normalization [-0.  0.  0.]  trainable parameters: 6 (gamma and beta for 3 channels)  buffers: ['running_mean', 'running_var', 'num_batches_tracked']

What the code does. After 200 batches, the running statistics converge to the true values of the data (mean 5, variance 9). Then the comparison: in eval mode an example is normalized with the accumulated statistics, and produces reasonable values (between −1.5 and 0.3); in train mode with a batch of only two examples, each column has two values, which normalization brings to −1 and +1 (approximately) whatever their content: the information about the size of the differences is erased. This is why BN suffers with very small batches (the statistics are noisy) and why, at inference, statistics computed on the batch cannot be used. For images, BatchNorm2d computes one mean and one variance per channel, averaging over all the examples and all the positions (N, H, W): the three per-channel means return to zero although starting from different scales. The trainable parameters are 2C (γ and β for each channel); the running statistics are buffers, not parameters: they are part of the state_dict but receive no gradient.

The effect: sensitivity to the learning rate

We put BN to the test where we expect it to be needed: a deep network (7 dense layers of 64) trained with SGD, with different learning rates, with and without BN before every ReLU. We measure the final training loss and the test accuracy.

X_all, X_te2 = X_tmp, X_te
mean, std = X_all.mean(), X_all.std()
T_all, T_test = [torch.tensor((a - mean) / std, dtype=torch.float32) for a in (X_all, X_te2)]
L_all, L_test = torch.tensor(y_tmp), torch.tensor(y_te)

def deep_net(with_bn, depth=6, width=64):
    layers = [nn.Linear(64, width)]
    for _ in range(depth):
        if with_bn:
            layers.append(nn.BatchNorm1d(width))
        layers += [nn.ReLU(), nn.Linear(width, width)]
    if with_bn:
        layers.append(nn.BatchNorm1d(width))
    layers += [nn.ReLU(), nn.Linear(width, 10)]
    return nn.Sequential(*layers)

def fit(model, lr, epochs=20):
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    loss_fn = nn.CrossEntropyLoss()
    loader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(T_all, L_all), batch_size=32, shuffle=True)
    for _ in range(epochs):
        model.train()
        for xb, yb in loader:
            opt.zero_grad(); loss_fn(model(xb), yb).backward(); opt.step()
    model.eval()
    with torch.no_grad():
        return loss_fn(model(T_all), L_all).item(), (model(T_test).argmax(1) == L_test).float().mean().item()

print("learning rate   without BN: loss  test acc.      with BN: loss  test acc.")
for lr in (0.01, 0.05, 0.2, 0.5):
    torch.manual_seed(0); without = fit(deep_net(False), lr)
    torch.manual_seed(0); with_ = fit(deep_net(True), lr)
    print("%13g   %18.4f  %8.3f   %19.4f  %8.3f" % ((lr,) + without + with_))

learning rate   without BN: loss  test acc.      with BN: loss  test acc.
         0.01               2.3022     0.100                0.0348     0.978
         0.05               2.2955     0.236                0.0314     0.961
          0.2                  nan     0.100                0.0650     0.956
          0.5               1.6249     0.283                0.1007     0.922

What the code does. deep_net builds 7 dense layers, with or without BatchNorm1d before every ReLU. fit uses plain SGD (no momentum and no Adam, to isolate the effect) for 20 epochs and returns the loss on the training set and the accuracy on the test set. The effect is spectacular. Without BN, with learning rate 0.01 and 0.05 the network does not take off: the loss stays at 2.30 (that is ln 10, chance) and the test accuracy is 0.10 and 0.24; with 0.2 the descent diverges (the loss becomes nan, test 0.10) and with 0.5 it is unstable (loss 1.62, test 0.28).

With BN the same network learns at every learning rate: training loss between 0.031 and 0.101 and test accuracy between 0.922 and 0.978; the quality drops a little as the learning rate grows (0.978 with 0.01, 0.922 with 0.5), but no cell failed. In this experiment BN makes training insensitive to the choice of learning rate over a 50-fold range (from 0.01 to 0.5). Each cell is a single run with seed 0, so read the trend and not the decimals; Ioffe and Szegedy report the same advantage: much higher learning rates.

Why does it work?

The explanation in the original paper was that BN reduces the «internal covariate shift», that is the change in the distribution of the inputs of each layer during training. Later work, in particular S. Santurkar et al. (2018), showed that the effect does not depend on that mechanism: BN makes the landscape of the loss function smoother (the gradients vary less abruptly from one step to the next), and this allows larger steps without instability. A second effect is regularizing: since the statistics depend on the batch, each example is normalized slightly differently at every step, a noise similar to that of dropout. In practice honesty requires saying «it works, and the why is still debated»: the technique is used in almost all CNNs, but the theoretical explanation remains a subject of research.

Where to put them, and how to combine them

• BN: after the convolution (or dense layer) and before the ReLU in the original scheme; many also use the arrangement after the ReLU, with similar results. With BN the bias of the previous layer is redundant (the subtraction of the mean cancels it), so you can use bias=False.
• Dropout: typically after the final dense layers, with p between 0.2 and 0.5 as in the listings. Srivastava et al. also apply it to convolutional layers, with lower switch-off probabilities in the early layers; mind the direction: in their text p is the probability of keeping the neuron, in PyTorch nn.Dropout(p) is the probability of switching it off.
• Together: in CNNs with BN dropout is often reduced or omitted, because BN already has a regularizing effect. You decide by measuring.
• Alternatives: layer normalization normalizes per example, over all the features, and does not depend on the batch: it is the standard choice in transformers and when batches are small or variable.

The four items answer a single question: where does each technique sit, and when can one replace the other. Batch normalization goes inside the network, in front of the nonlinearity; dropout sits at the end, where the dense layers are large. When you use the first, you often remove or reduce the second; the replacement for small batches is layer normalization. None of the four is a rule: they are starting points to confirm on the validation set, as the dropout experiment showed.

Trade-offs and real traps

• Forgetting model.eval(): with dropout it produces noisy results; with BN it uses the batch statistics instead of the running ones. It is the number one mistake for both.
• Batches that are too small with BN: noisy statistics, different behavior between training and inference. With very small batches a normalization that does not depend on the batch (layer, group) is better.
• Non-representative batches: a batch of only «7» digits has different statistics from a mixed one. Shuffle.
• Stale running statistics: if you change the data (fine-tuning on a different domain) the running statistics must be recomputed or BN must be frozen consciously.
• Dropout on the last layer or p too high: it removes information needed for the prediction.
• Adopting dropout «because that is what one does»: in the experiment above it gave no measurable advantage. If it does not improve the validation, remove it: every extra component is something that can break.
• Redundant bias before BN: not wrong, just useless (wasted parameters).
• Saving only the parameters and forgetting the buffers: state_dict includes them, but if you save parameters() by hand you lose the running statistics and the loaded model behaves differently.
• Evaluating with the whole batch instead of small batches in train mode: the statistics of an unbalanced test set contaminate the evaluation. At inference always use eval().

The nine traps almost all stem from the same root: dropout and batch normalization behave differently in training and at inference, and the code does not remind you. The first four concern that double behavior (eval(), small batches, non-representative batches, stale statistics); the others are choices to measure or wasted parameters. A symptom to recognize immediately is a different result at every evaluation on the same data: in the listing of the dropout section it went from 0.900 to 0.911, against a stable 0.922 in eval() mode. If you save and reload the model and the numbers change, check that the buffers are in the state_dict.

Frequently asked questions

Does dropout slow down training? Yes, usually more epochs are needed, because the noise makes the gradient signal less clean. The cost per step is negligible.

Is BN a form of regularization? It has a regularizing effect (the noise due to the batch statistics), but its main purpose is to stabilize and speed up the optimization.

Why are γ and β per channel and not per position? Because the convolution treats all positions in the same way (weight sharing); normalization per channel respects this symmetry.

Can I use BN with batch size 1 at inference? Yes, in eval() mode: it uses the running statistics. It is in training that a batch of 1 does not work.

Three exercises

Exercise 1. Repeat the dropout experiment with a smaller network (32 neurons per layer) and with more data (400 examples): does the effect change? Also try p = 0.8. What happens to the training accuracy?

Exercise 2. In the deep network with BN, try again with batches of 2 examples instead of 32 and compare the test accuracy in eval() with that of batch 32. How much does BN suffer?

Exercise 3. Add BatchNorm2d to the CNN of the previous lesson and measure whether it improves robustness to shifts. (Probably little: BN acts on the optimization, not on the invariance.)

What you can do now

You can write inverted dropout by hand and check that the mean stays 1 and the variance equals p/(1 − p). You can recognize its classic mistake, a forgotten eval(), from the result that changes at every measurement. You have seen a technique that is not needed here (dropout on Digits: test 0.906, 0.913 and 0.909 with p = 0, 0.2 and 0.5, within the noise between seeds) and one that changes the outcome (BN: the deep network without BN does not take off with learning rate 0.01 and 0.05, with BN it learns at all four values tried). You can derive the gradient of batch normalization and check it against PyTorch (maximum difference of the order of 10⁻¹⁵), and you know that at inference it uses the running statistics, which are buffers and not parameters. In the next lesson, feature maps, global pooling and U-Net, you look inside a CNN; to reuse one that is already trained, go to transfer learning, where the running statistics of BN count again.

Other lessons in the series that link to this one: Training a LeNet-5 style CNN with PyTorch.