50 - Stride, padding, pooling and counting parameters in a CNN

Imagine scanning a page with a magnifying glass. You can move the lens a millimetre at a time and read every zone several times, or a centimetre at a time, faster but skipping details: this is the stride. You can lay the page on a larger sheet, with a white margin all around, so that even the words on the edge are looked at in the centre of the lens: this is the padding. And when you have finished you can summarize each zone with its most striking detail, throwing away the rest, to get a smaller and more manageable map: this is the pooling. Three simple operations that, together with convolution, make up the architecture of almost every convolutional network.

This lesson assumes convolution with filters and kernels and the receptive-field computation found there. The everyday work that becomes precise here is predicting the shape of the tensors at every point of a network: it is where mistakes are made most often, and a shape error is only discovered when the network runs. We derive the formula, count by hand the parameters of a LeNet-5-style network, the ancestor of modern convolutional networks, and see how much computation they cost.

A note on the code, because it changes the way you read the listings. The machine on which they were run has no PyTorch, so nothing that follows uses it: the shapes and the parameters are verified with a small NumPy implementation (convolution, ReLU, max pooling, flatten and dense layer, a few lines per class). The formulas you derive hold identically in PyTorch, where the documentation of Conv2d gives the same expression for the output; but the numbers and the error messages you read below are those of NumPy 2.3 with Python 3.14, not of PyTorch.

Stride and padding, with the formula derived

In the previous lesson the window slid one pixel at a time, without leaving the image, and the output shrank by k − 1 pixels. Two knobs change this behaviour:

• Stride S: by how many pixels the window moves between one position and the next. With S = 1 all the positions are looked at; with S = 2 one position out of two is skipped and the output has about half the size per side.
• Padding P: how many pixels of “border” are added on each side of the image before convolving. The most common value is zero (zero padding), and it widens the image from H to H + 2P.

The two knobs act on different things: the stride decides how many window positions exist, the padding decides how big the image is on which it slides. The stride reduces the work and the resolution of the output, the padding preserves the borders and the size; neither changes the number of parameters of the filter, which depends only on kernel and channels.

Let us derive the size of the output. After the padding the image has side H + 2P. The top-left corner of the window can sit at positions 0, S, 2S, ... as long as the window stays inside, that is, as long as its position p satisfies p + K ≤ H + 2P, hence p ≤ H + 2P − K. The valid positions are 0, S, 2S, ..., up to the largest multiple of S not exceeding H + 2P − K. How many are there? They are ⌊(H + 2P − K)/S⌋ + 1 (the +1 counts position 0):

H_out = ⌊ (H + 2P − K) / S ⌋ + 1

Notable cases:
  P = 0, S = 1:             H_out = H − K + 1             (the “valid” convolution of the previous lesson)
  K odd, P = (K−1)/2, S = 1:   H_out = H                  (“same” padding: the output has the same size as the input)
  K = 2, S = 2, P = 0:      H_out = ⌊(H − 2)/2⌋ + 1 = ⌊H/2⌋        (halves: typical of pooling)

The symbol ⌊·⌋ is the floor: if the division is not exact, the pixels that “are left over” at the end are simply ignored. So as not to trust the derivation, we put it to the test with an independent count: we build the image with the padding, extract all the K×K windows with NumPy's sliding_window_view, keep one window every S and count how many there are, without ever using the formula.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

def dimensione_uscita(H, K, S=1, P=0):
    return (H + 2 * P - K) // S + 1

def posizioni_finestra(H, K, S, P):
    """Counts the window positions on an image with zero padding, without using the formula."""
    img = np.pad(np.zeros((H, H)), P)
    return sliding_window_view(img, (K, K))[::S, ::S].shape[0]

controlli, corretti = 0, 0
for H in range(5, 20):
    for K in (1, 2, 3, 5):
        for S in (1, 2, 3):
            for P in (0, 1, 2):
                if H + 2 * P < K:
                    continue                                              # the kernel does not even fit
                controlli += 1
                corretti += (posizioni_finestra(H, K, S, P) == dimensione_uscita(H, K, S, P))
print("combinations checked: %d   formula correct in %d" % (controlli, corretti))

print("\nH   K  S  P   output")
for H, K, S, P in ((28, 5, 1, 0), (28, 5, 1, 2), (32, 3, 2, 1), (224, 7, 2, 3), (7, 2, 2, 0)):
    print("%3d %2d %2d %2d   %d" % (H, K, S, P, dimensione_uscita(H, K, S, P)))

combinations checked: 540   formula correct in 540

H   K  S  P   output
 28  5  1  0   24
 28  5  1  2   28
 32  3  2  1   16
224  7  2  3   112
  7  2  2  0   3

What the code does. posizioni_finestra builds an image of zeros with np.pad, derives from it all the overlapping windows with sliding_window_view and with [::S, ::S] keeps those at a distance of S; the length of the positions axis is the size of the output. The quadruple loop goes through 540 valid combinations of size, kernel, stride and padding, excluding those in which the kernel does not even fit in the image, and the formula coincides with the count in all 540.

The final lines are examples you will meet: 28 with kernel 5 and no padding gives 24; with padding 2 it stays 28 (“same” padding); 32 with kernel 3, stride 2 and padding 1 gives 16; 224 with kernel 7, stride 2 and padding 3 (the first layer of many networks for images) gives 112; and a 2×2 window with stride 2 on a 7×7 map gives 3, discarding the last row and the last column. The check is on a single dimension because the image is square: for a rectangular image the formula is applied separately to height and width.

Why padding? Two reasons. The first is to keep the dimensions: without padding the image shrinks at every layer and after many layers nothing is left. The second is to treat the borders like the rest: a corner pixel, without padding, takes part in a single window, a central one in nine (with a 3×3 kernel); the padding rebalances the contribution. Zero padding does, however, introduce artificial borders (a black frame) that the filter can mistake for real edges; there are variants (reflection, replication) that reduce the problem, at the cost of a little complexity.

Pooling: summarizing a window

Pooling reduces the size of a map by summarizing each window with a single number, with no parameters to learn. The two variants:

• Max pooling: the maximum of the window. It says “the pattern is present somewhere in here”, without saying exactly where.
• Average pooling: the mean of the window. It says “how much, on average, the pattern is present”.

The two operators answer different questions: the maximum keeps the strongest signal and throws away its exact position inside the window, the mean keeps the overall intensity and smooths the peaks. Neither has weights to learn, so pooling adds zero parameters and serves only to shrink the map.

The most common choice is a 2×2 window with stride 2, which halves height and width (and therefore divides by four the number of values of each map).

In the drawing, each of the four coloured 2×2 windows is replaced by its maximum (in red): 4, 5, 3, 7. Pooling works on each channel independently, so it does not change the number of channels.

What happens to the gradients

Max pooling has no parameters, but it must let backpropagation through. The rule follows from the definition: the output is the maximum of four values, so it depends (locally) only on the value that is the maximum; nudging the other three slightly leaves the output unchanged. So the gradient arriving at the output is sent entirely to the pixel that won, and the other three get zero. In average pooling the output depends on all four values equally (derivative 1/4 each): the gradient is split into equal parts. Without an automatic-differentiation framework at hand, we check the rule in the most direct way: we estimate the derivative with finite differences (one pixel at a time is moved by ±10^-6 and we see how much the sum of the outputs changes) and compare it with the rule. In the same listing we implement max pooling by hand and measure an often-cited effect, invariance to small translations.

import numpy as np
def max_pooling(I, k=2, s=2):
    H = (I.shape[0] - k) // s + 1
    W = (I.shape[1] - k) // s + 1
    return np.array([[I[i * s:i * s + k, j * s:j * s + k].max() for j in range(W)] for i in range(H)])

invariati, cambiati_prima, prove = 0, 0, 0
for r in range(8):
    for c in range(7):
        a = np.zeros((8, 8)); a[r, c] = 1.0
        b = np.zeros((8, 8)); b[r, c + 1] = 1.0
        prove += 1
        cambiati_prima += not np.array_equal(a, b)
        invariati += np.array_equal(max_pooling(a), max_pooling(b))
print("shifting the pattern by 1 pixel: %d trials out of %d change the image, only %d out of %d change the pooling output" % (
    cambiati_prima, prove, prove - invariati, prove))

x = np.array([[1., 3, 2, 0], [4, 2, 1, 5], [0, 1, 7, 2], [3, 2, 1, 1]])

def gradiente_max_pooling(x, k=2):
    """Derivative of the sum of the outputs with respect to x: 1 on the winning pixel, 0 elsewhere."""
    g = np.zeros_like(x)
    for i in range(0, x.shape[0], k):
        for j in range(0, x.shape[1], k):
            blocco = x[i:i + k, j:j + k]
            r, c = np.unravel_index(blocco.argmax(), blocco.shape)
            g[i + r, j + c] = 1.0
    return g

def media_pooling(I, k=2, s=2):
    H = (I.shape[0] - k) // s + 1
    W = (I.shape[1] - k) // s + 1
    return np.array([[I[i * s:i * s + k, j * s:j * s + k].mean() for j in range(W)] for i in range(H)])

def somma_pooling(x, f=max_pooling):
    return f(x).sum()

def gradiente_numerico(x, f, eps=1e-6):
    g = np.zeros_like(x)
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            piu = x.copy(); piu[i, j] += eps
            meno = x.copy(); meno[i, j] -= eps
            g[i, j] = (somma_pooling(piu, f) - somma_pooling(meno, f)) / (2 * eps)
    return g

# finite-difference check: the analytic gradient must coincide
numerico = gradiente_numerico(x, max_pooling)
numerico_media = gradiente_numerico(x, media_pooling)

print("\nmax pooling output:\n", max_pooling(x))
print("gradient with respect to the input (1 only where the maximum was):\n", gradiente_max_pooling(x))
print("matches the finite differences:", np.allclose(gradiente_max_pooling(x), numerico, atol=1e-6))
print("gradient with average pooling (finite differences, rounded to 2 decimals):\n", numerico_media.round(2))

shifting the pattern by 1 pixel: 56 trials out of 56 change the image, only 24 out of 56 change the pooling output

max pooling output:
 [[4. 5.]
 [3. 7.]]
gradient with respect to the input (1 only where the maximum was):
 [[0. 0. 0. 0.]
 [1. 0. 0. 1.]
 [0. 0. 1. 0.]
 [1. 0. 0. 0.]]
matches the finite differences: True
gradient with average pooling (finite differences, rounded to 2 decimals):
 [[0.25 0.25 0.25 0.25]
 [0.25 0.25 0.25 0.25]
 [0.25 0.25 0.25 0.25]
 [0.25 0.25 0.25 0.25]]

What the code does. max_pooling slides 2×2 windows with stride 2 and takes the maximum of each. The first experiment puts a single lit pixel in all 56 possible positions and shifts it one column to the right: the image always changes (56 out of 56), but the pooling output changes in only 24 cases out of 56 (the cases in which the shift crosses the boundary between two windows), that is, it stays identical in 32 cases (57%). Pooling therefore gives a partial invariance to small translations: what falls inside the same window is “absorbed”. It is not a complete invariance, and a shift larger than the window will change the output anyway. In the second part the gradient of max pooling is 1 only at the position of the maxima (4, 5, 3 and 7) and zero everywhere else, and it coincides with the finite differences; with average pooling the finite differences give 0.25 everywhere.

A caution about the gradient: if in a window two values are equal to the maximum, the function is not differentiable at that point and the frameworks choose a convention (here argmax takes the first); in real data it rarely happens, but with ReLU, which produces many zeros, an all-zero window does occur.

A complete CNN: shapes and parameters, layer by layer

Let us put the ingredients together in a classic LeNet-5-style architecture for 28×28 handwritten digits in greyscale: two “convolution, ReLU, pooling” blocks followed by three dense layers. The passage from the spatial world (image) to that of vectors is called flatten: the tensor (N, C, H, W) becomes a matrix (N, C·H·W).

The drawing gives the shapes for a single example: 1@28×28 means 1 channel of 28×28 pixels. Each convolution changes the number of channels (1 → 6 → 16) and each pooling halves the spatial resolution; at the end 16 channels of 5×5 become a vector of 400 numbers, which feeds the three dense layers. To verify every shape and every parameter count we need real layers, so the listing that follows builds the small NumPy library that we will use in the two following examples too (if you run them, put them in the same file). The convolution is an einsum on the windows extracted with sliding_window_view; the weights are random with a fixed seed, because for shapes and counts the values do not matter; parametri() returns the number of elements of weights and biases, which is exactly what numel() counts in PyTorch.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

rng = np.random.default_rng(0)

class Conv2d:
    def __init__(self, c_in, c_out, k, padding=0):
        self.W = rng.normal(0, 0.1, (c_out, c_in, k, k)); self.b = np.zeros(c_out); self.p = padding
    def __call__(self, x):                                    # x: (N, C, H, W)
        x = np.pad(x, ((0, 0), (0, 0), (self.p, self.p), (self.p, self.p)))
        k = self.W.shape[-1]
        finestre = sliding_window_view(x, (k, k), axis=(2, 3))   # (N, C, H', W', k, k)
        return np.einsum("nchwij,ocij->nohw", finestre, self.W) + self.b[None, :, None, None]
    def parametri(self): return self.W.size + self.b.size

class ReLU:
    def __call__(self, x): return np.maximum(x, 0)
    def parametri(self): return 0

class MaxPool2d:
    def __init__(self, k): self.k = k
    def __call__(self, x):
        n, c, h, w = x.shape; k = self.k
        return x[:, :, :h // k * k, :w // k * k].reshape(n, c, h // k, k, w // k, k).max(axis=(3, 5))
    def parametri(self): return 0

class Flatten:
    def __call__(self, x): return x.reshape(x.shape[0], -1)
    def parametri(self): return 0

class Linear:
    def __init__(self, n_in, n_out):
        self.W = rng.normal(0, 0.1, (n_in, n_out)); self.b = np.zeros(n_out)
    def __call__(self, x): return x @ self.W + self.b
    def parametri(self): return self.W.size + self.b.size

def Sequential(*strati): return list(strati)

rete = Sequential(
    Conv2d(1, 6, 5, padding=2), ReLU(), MaxPool2d(2),            # 1x28x28 -> 6x28x28 -> 6x14x14
    Conv2d(6, 16, 5),           ReLU(), MaxPool2d(2),            # -> 16x10x10 -> 16x5x5
    Flatten(),                                                    # -> 400
    Linear(400, 120), ReLU(),
    Linear(120, 84),  ReLU(),
    Linear(84, 10))

z = np.zeros((2, 1, 28, 28))                                      # a batch of 2 images
totale = 0
print("layer                 output shape (batch of 2)       parameters")
for strato in rete:
    z = strato(z)
    n = strato.parametri()
    totale += n
    print("%-20s  %-28s  %9d" % (type(strato).__name__, tuple(z.shape), n))
print("total parameters: %d" % totale)

fc1 = 400 * 120 + 120
print("share of the parameters in the first dense layer: %.1f%%   in the two convolutions: %.1f%%" % (100 * fc1 / totale, 100 * (156 + 2416) / totale))

layer                 output shape (batch of 2)       parameters
Conv2d                (2, 6, 28, 28)                      156
ReLU                  (2, 6, 28, 28)                        0
MaxPool2d             (2, 6, 14, 14)                        0
Conv2d                (2, 16, 10, 10)                    2416
ReLU                  (2, 16, 10, 10)                       0
MaxPool2d             (2, 16, 5, 5)                         0
Flatten               (2, 400)                              0
Linear                (2, 120)                          48120
ReLU                  (2, 120)                              0
Linear                (2, 84)                           10164
ReLU                  (2, 84)                               0
Linear                (2, 10)                             850
total parameters: 61706
share of the parameters in the first dense layer: 78.0%   in the two convolutions: 4.2%

What the code does. Sequential is here a simple list of layers; the loop goes through it layer by layer with a batch of zeros, printing the shape of each output and how many parameters the layer has. The counts can be rebuilt by hand with the formulas of the previous lesson: the first convolution has 6·(1·5·5 + 1) = 156 parameters; the second 16·(6·5·5 + 1) = 2,416; the first dense layer 400·120 + 120 = 48,120; the second 120·84 + 84 = 10,164; the last 84·10 + 10 = 850. Total: 61,706. Pooling, ReLU and flatten have no parameters.

The final line brings out a structural fact: 78% of the parameters are in the first dense layer, and the two convolutions together have only 4.2%. The convolutional layers are frugal thanks to weight sharing; it is the dense connection after the flatten, which has to connect every value to every neuron, that weighs. This is why modern architectures avoid bulky dense layers and use a global pooling (you find it in the lesson on feature maps, global pooling and U-Net).

Parameters and computation are not the same thing

Another measure of the “cost” of a network is the number of multiply-accumulates (MAC) for one image. A convolution applies the same kernel at many positions: it has few parameters but many operations. For a convolutional layer, the MACs are H_out · W_out · C_out · (C_in · k · k); for a dense one, n_in · n_out.

The listing is pure Python: it applies the two formulas to the five parts with parameters.

mac = {
    "conv1 (1 -> 6, 5x5, output 28x28)":  28 * 28 * 6 * (1 * 5 * 5),
    "conv2 (6 -> 16, 5x5, output 10x10)": 10 * 10 * 16 * (6 * 5 * 5),
    "dense 400 -> 120":                   400 * 120,
    "dense 120 -> 84":                    120 * 84,
    "dense 84 -> 10":                     84 * 10,
}
totale_mac = sum(mac.values())
for nome, valore in mac.items():
    print("%-38s %9d MAC  (%4.1f%% of the computation)" % (nome, valore, 100 * valore / totale_mac))
print("total: %d MAC per image  ->  %.1f MAC per parameter" % (totale_mac, totale_mac / 61706))

conv1 (1 -> 6, 5x5, output 28x28)         117600 MAC  (28.2% of the computation)
conv2 (6 -> 16, 5x5, output 10x10)        240000 MAC  (57.6% of the computation)
dense 400 -> 120                           48000 MAC  (11.5% of the computation)
dense 120 -> 84                            10080 MAC  ( 2.4% of the computation)
dense 84 -> 10                               840 MAC  ( 0.2% of the computation)
total: 416520 MAC per image  ->  6.8 MAC per parameter

What the code does. The two convolutions contain 4% of the parameters but 86% of the computation (about 358,000 MAC out of 417,000), while the first dense layer contains 78% of the parameters but 11% of the computation. It is an important duality in convolutional networks: the memory of the weights is dominated by the dense layers, the computing time by the convolutional layers, and the two optimizations (compressing the model, speeding it up) act on different parts of the network.

The classic trap: the size after the flatten

The number 400 in Linear(400, 120) is not arbitrary: it is 16·5·5, and it depends on the whole chain of convolutions, padding and pooling that precedes it. If you change the image resolution, the padding or a kernel, that number changes, and the network stops working with a shape error. A convenient way to compute it without mistakes is to let the network do it, with a trial pass on a tensor of zeros. The listing reuses the classes of the previous block.

import numpy as np
exec(open("c.py").read().split("rete = Sequential(")[0])

parte_conv = Sequential(Conv2d(1, 6, 5, padding=2), ReLU(), MaxPool2d(2),
                        Conv2d(6, 16, 5), ReLU(), MaxPool2d(2), Flatten())

def esegui(strati, x):
    for s in strati:
        x = s(x)
    return x

for lato in (28, 32, 40):
    n_flatten = esegui(parte_conv, np.zeros((1, 1, lato, lato))).shape[1]
    r = ((lato + 4 - 5 + 1) // 2 - 4) // 2
    print("image %dx%d  ->  after the flatten: %d values  (formula: 16 * %d * %d = %d)" % (lato, lato, n_flatten, r, r, 16 * r ** 2))

try:
    esegui(parte_conv + [Linear(400, 10)], np.zeros((1, 1, 32, 32)))         # the 400 holds only for 28x28
except ValueError as e:
    print("\nerror with 32x32 images and Linear(400, ...):", e)

image 28x28  ->  after the flatten: 400 values  (formula: 16 * 5 * 5 = 400)
image 32x32  ->  after the flatten: 576 values  (formula: 16 * 6 * 6 = 576)
image 40x40  ->  after the flatten: 1024 values  (formula: 16 * 8 * 8 = 1024)

error with 32x32 images and Linear(400, ...): matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 400 is different from 576)

What the code does. The first loop sends images of side 28, 32 and 40 through the convolutional part and reads how many values come out of the flatten: 400, 576 and 1024. Compare them with the formula: with the pooling that halves twice and the convolutions that remove 4 pixels, the final side is ((side + 4 − 5 + 1)/2 − 4)/2. The last test adds Linear(400, 10) and applies it to 32×32 images: NumPy stops with an error on the dimensions of matmul (“size 400 is different from 576”), which says clearly that the flatten produces 576 values, not 400. The same mistake in PyTorch has a differently worded message (I have not reproduced it, for the reason given at the start); the framework also offers nn.LazyLinear, which infers the input size at the first pass; alternatively you compute it with a trial pass as above, instead of trusting a mental calculation.

A second worked example: 32×32 colour images

Let us practise the method on a network for colour images of 32×32 pixels and 10 classes, with three “3×3 convolution with padding 1, ReLU, 2×2 pooling” blocks that double the channels (32, 64, 128) while halving the resolution. The procedure is always the same: for each layer the size formula and the parameter formula are applied.

• Input: 3 channels, 32×32.
• Conv 1 (3 → 32, k=3, P=1): side (32 + 2 − 3)/1 + 1 = 32, output 32@32×32. Parameters: 32·(3·9 + 1) = 896. Pool: 32@16×16.
• Conv 2 (32 → 64): side 16, output 64@16×16. Parameters: 64·(32·9 + 1) = 18,496. Pool: 64@8×8.
• Conv 3 (64 → 128): side 8, output 128@8×8. Parameters: 128·(64·9 + 1) = 73,856. Pool: 128@4×4.
• Flatten: 128·4·4 = 2,048 values. Dense 2,048 → 10: 2,048·10 + 10 = 20,490 parameters.
• Total: 896 + 18,496 + 73,856 + 20,490 = 113,738 parameters.

Read the count by columns: the spatial size does not change in the convolutions (padding 1 with kernel 3 preserves the side) and is halved only in the poolings, while the parameters grow with the product of input and output channels. The fact that matters is in the last item: after three halvings 4×4 positions remain, and the flatten brings 2,048 values into the dense layer, which with its 20,490 parameters weighs less than the last convolution (73,856).

Let us verify the count with the code (same classes as the previous block), and observe the practical rule that this architecture embodies: as the spatial resolution is halved (reducing by a factor of 4 the number of positions), the channels are doubled, so that the computation per layer stays of the same order of magnitude and the network can represent ever more numerous and abstract features.

import numpy as np
exec(open("c.py").read().split("rete = Sequential(")[0])

cifar_style = Sequential(
    Conv2d(3, 32, 3, padding=1),   ReLU(), MaxPool2d(2),
    Conv2d(32, 64, 3, padding=1),  ReLU(), MaxPool2d(2),
    Conv2d(64, 128, 3, padding=1), ReLU(), MaxPool2d(2),
    Flatten(), Linear(128 * 4 * 4, 10))

z = np.zeros((1, 3, 32, 32))
for strato in cifar_style:
    z = strato(z)
    if isinstance(strato, (Conv2d, MaxPool2d, Flatten, Linear)):
        print("%-10s -> %-18s parameters %7d" % (type(strato).__name__, tuple(z.shape[1:]), strato.parametri()))
print("total: %d   (by hand: 896 + 18496 + 73856 + 20490 = %d)" % (sum(s.parametri() for s in cifar_style), 896 + 18496 + 73856 + 20490))

Conv2d     -> (32, 32, 32)       parameters     896
MaxPool2d  -> (32, 16, 16)       parameters       0
Conv2d     -> (64, 16, 16)       parameters   18496
MaxPool2d  -> (64, 8, 8)         parameters       0
Conv2d     -> (128, 8, 8)        parameters   73856
MaxPool2d  -> (128, 4, 4)        parameters       0
Flatten    -> (2048,)            parameters       0
Linear     -> (10,)              parameters   20490
total: 113738   (by hand: 896 + 18496 + 73856 + 20490 = 113738)

What the code does. It goes through the network with an image of zeros and prints, for each layer with an interesting shape or parameters, the shape of the output (without the batch dimension) and the number of parameters. The numbers coincide with the hand computation line by line. Note also that the last convolutional block (73,856 parameters) is the most “expensive” in parameters among the convolutions because it multiplies the number of input channels (64) by that of output channels (128): the cost of a convolution grows with the product C_in·C_out, and this is why in deep layers with many channels cheaper structures are sought (the 1×1 convolutions of the lesson on residual networks and Inception).

Practical rules for designing an architecture

• Start with 3×3 kernels and “same” padding: they are the most common choice and the one that makes the shape computation simplest (the dimensions change only with poolings or strides).
• Halve the resolution, double the channels: the number of values per layer (channels × height × width) grows or falls in a controlled way, and the network builds more abstract features as the receptive field widens.
• Reduce before flattening: bring the map down to a small size (4×4, 7×7) before the flatten, so as not to have a dense layer with millions of parameters.
• Compute the receptive field: in the last convolutional layer it must be comparable to the size of the objects to be recognized. With the three blocks of the example the receptive field at the last pooling is 22 pixels (you can verify it with the campo_ricettivo function of the previous lesson), about two thirds of the side of a 32×32 image: enough for medium-sized objects; for large images more layers or larger strides are needed.
• Always verify with a trial pass before training: shapes, number of parameters, time of one pass.

The five rules are heuristics, not theorems: they come from observing what works in classic architectures and have counterexamples (networks with 5×5 kernels in the first layer, channels that do not double at every step). Use them as a starting point for the first attempt and correct them with a measurement: the rule on the receptive field, for example, you verify with the function of the previous lesson (campo_ricettivo([(3, 1), (2, 2)] * 3) gives 22, computed for this article), while the rules on the flatten and on the trial pass you verify with the listings above.

Three exercises to fix the ideas

Exercise 1. A 64×64 image goes through a 5×5 convolution with stride 1 and padding 0, then a 2×2 max pooling with stride 2. What is the size of the output? (Convolution: 64 − 5 + 1 = 60; pooling: 60/2 = 30.)

Exercise 2. How many parameters does a Conv2d(64, 128, kernel_size=3) layer have? And a Conv2d(64, 128, kernel_size=1)? (128·(64·9 + 1) = 73,856 and 128·(64·1 + 1) = 8,320: the 1×1 kernel costs about a ninth, but it does not look at neighbouring pixels; it is used to change the number of channels.)

Exercise 3. Modify the colour network above by removing the last pooling: how many values come out of the flatten and how many parameters does the new dense layer have? (128·8·8 = 8,192 values; 8,192·10 + 10 = 81,930 parameters: almost four times those of before, just for one pooling fewer.)

Other ways to reduce the resolution

• Convolution with stride 2: it does the size reduction with a learned filter instead of a fixed rule like the maximum. Many modern architectures give up pooling in favour of strided convolutions.
• Dilated convolution (dilation): it spaces out the weights of the kernel (for example a 3×3 with a hole between the weights) to widen the receptive field without more parameters. The effective size of the kernel becomes K + (K − 1)(d − 1) with dilation d, and the output formula is updated accordingly.
• Global pooling: a pooling with a window as large as the whole map; it reduces each channel to a single number. You will see it in the lesson on feature maps and global pooling.

They are three different ways of answering a different problem: stride 2 replaces pooling with a filter that learns how to summarize, dilation widens the receptive field without reducing the resolution, global pooling eliminates the spatial dimension altogether. None is “better”; the choice depends on what must survive the reduction. If the position is needed (segmentation) you avoid throwing it away; if only one decision per image is needed you can throw it away without regret.

Trade-offs and real-world traps

• Pooling and position: pooling discards information about the precise position. It is an advantage if you care about “there is an eye” and a disadvantage if you care about “where the eye is” (segmentation, localization), where different strategies are used (lesson on U-Net).
• Stride larger than the kernel: if S > K some pixels are never seen by any window.
• Odd dimensions and floor: with a 7×7 image and 2×2 pooling the last row and column are lost. A choice of dimensions that halve exactly (28 → 14 → 7) avoids this effect, while 7 → 3 (and then 3 → 1) discards information.
• Missing padding: in PyTorch the default of Conv2d is padding=0 (read in the module's source, see Sources). If you expect an output of the same size you have to ask for it: with an odd kernel and stride 1 padding=(K-1)//2 is enough, and Conv2d also accepts the string 'same' (only with stride 1).
• Giant dense layers after the flatten: they dominate the parameters (78% in the example) and favour overfitting.
• Confusing parameters with computation cost: convolutions have few parameters but a lot of computation.
• Computing dimensions in your head instead of verifying them: a trial pass with tensors of zeros costs a second and saves hours of debugging.
• Max pooling and ReLU: the order between the two does not change the result, because both are monotone (the maximum of values with ReLU is the ReLU of the maximum); by convention the ReLU goes first.

The traps have a common origin: the shape of a tensor is a consequence of five numbers (size, kernel, stride, padding, pooling) and changing one is enough for everything downstream to change. This is why it is better to have the network compute the shapes with a trial pass rather than carry them in your head; what remains to be judged by eye are the trade-offs, that is, how much information about position and how many parameters you are willing to pay.

Frequently asked questions

Is max pooling or the mean better? Max is the most common choice in the intermediate layers, because it preserves strong signals (the presence of a feature). The mean, applied to the whole map (global average pooling), is the one used before the last layer in modern architectures.

Why is the pooling window almost always 2×2? Because it is the minimum that reduces the resolution and keeps enough information. Larger windows reduce too quickly.

How many convolutional layers are needed? It depends on the complexity of the task: LeNet-5 has 2 for digits; networks for natural images have dozens (following lessons).

Is zero padding always a good idea? It is the default choice and works well, but it introduces an artificial frame. For applications sensitive to borders, reflection or replication are evaluated.

What you can do now

You can compute the output size of a convolution, ⌊(H + 2P − K)/S⌋ + 1, and you have seen it coincide with a direct count of the windows on 540 combinations. You know that pooling summarizes each window without parameters, that in max pooling the gradient goes only to the maximum (verified with finite differences) and that the translation invariance it offers is partial: 24 cases out of 56 change the output. You can count the shapes and parameters of a LeNet-5-style CNN (61,706, of which 78% in the first dense layer, while 86% of the multiplications are in the convolutions) and have the network compute the size after the flatten. In the next lesson, training a LeNet-5-style CNN with PyTorch, that same network learns from real images; if you want to review how the receptive field is obtained, go back to the lesson on convolution.