49 - Convolution: filters, kernels and why look locally

To find a word written in small print on a page, you do not read the whole sheet at once: you take a magnifying glass and slide it along the text. At each position the lens shows you a small window, and you check whether what you are looking for is in there. A convolution does exactly this on an image: a small window (the filter or kernel) slides over the whole image and at each position answers with a number, high if the pattern the filter is looking for is there (an edge, a corner, a blob), low if it is not. The set of all the answers is a new image, the feature map.

Convolutional networks are the answer to the question: how do you make a neural network see? In the previous lessons you trained a dense network on 8×8 pixel digits, and it worked (about 96% accuracy). But with larger images the dense strategy does not scale and, above all, it ignores the most important thing about an image: that neighboring pixels are correlated. In this lesson you understand why, you define the convolution precisely, you compute it by hand and in code, you discover that a filter is not to be designed but learned with the same gradient you already know, and you find all the calculations (shapes, parameters, receptive field) that you will use in the next lessons.

Why a dense layer is not enough for images

A 224×224 pixel color photograph has 224·224·3 = 150 528 numbers. If you connect these inputs to a dense layer of 1000 neurons, you need 150 528·1000 + 1000 = 150 529 000 parameters, about 150 million, in a single layer. The problem is not only cost. There are two deeper defects.

The first is that a dense layer ignores proximity: for it the pixels are a list of numbers with no order, and swapping the columns of X in the same way across all the examples would change nothing for learning, whereas in an image the local patterns matter (an edge is a difference between adjacent pixels). The second is that it does not reuse what it learns: if the network learns to recognize an eye at the top left, it cannot recognize it at the bottom right, and it would have to learn it again, with different weights, for every position.

Convolutional networks solve both defects with three ideas worth naming and remembering. The three ideas are read together: the first narrows the gaze of each neuron, the second reuses it everywhere, and the third is the consequence you observe at the output.

1. Local connectivity (local receptive field): each neuron looks only at a small region of the input, for example 3×3 pixels, not at the whole image.
2. Weight sharing: the same set of weights (the kernel) is used at all positions. A single vertical-edge detector, applied everywhere.
3. Translation equivariance: if you move the pattern in the image, the response moves by the same amount. An edge in another place produces the same signal in another place.

The ideas were not born with deep learning: local connectivity and the hierarchy of detectors appeared in K. Fukushima's "neocognitron" (1980), and the backpropagation training of a convolutional network on handwritten digits is by Y. LeCun's group (1989, then LeNet-5, 1998), the primary sources from which this module starts.

The image as a tensor

A grayscale image is a matrix of shape (H, W): H rows of pixels, W columns, each value the intensity (generally an integer from 0 to 255 if it is stored with 8 bits). A color image has three channels (red, green, blue): it is a tensor with three stacked matrices. For a batch of N images a dimension is added at the front. There are two conventions on the order, and knowing them avoids recurring mistakes.

• Channels first (the convention I use in this lesson, and the one of PyTorch according to its documentation): (N, C, H, W), that is batch, channels, height, width.
• Channels last (TensorFlow/Keras by default, and most image libraries such as PIL and matplotlib): (N, H, W, C).

Another habit: pixels are integers from 0 to 255, but the network wants floating-point numbers of moderate scale. They are divided by 255 (values in [0, 1]) and often standardized. Further on you will see a concrete trap of this kind.

What a convolution is

Let I be an image (H × W) and K a kernel (k × k, with k odd, for example 3). The output at position (i, j) is the sum of the products between the kernel and the k × k window of the image that starts at (i, j):

O[i, j] = Σ_{u=0..k−1} Σ_{v=0..k−1}  K[u, v] · I[i + u, j + v]

A point of terminology that often confuses. This operation is called, in mathematics, cross-correlation; the rigorous mathematical convolution flips the kernel (it uses I[i − u, j − v]). Deep learning libraries implement the former and call it "convolution". The difference does not matter: the kernel is a set of numbers that the network learns, and if the mathematics required a flip the network would learn the flipped kernel. We follow the convention of the libraries.

The diagram shows the calculation: the yellow 3×3 window at the top left of the 5×5 image is multiplied, element by element, by the kernel (which here looks for a vertical edge: weights +1 on the left, 0 in the middle, −1 on the right) and the nine products are summed. The result 0 is the first cell of the output; then the window slides one position to the right, and so on. With a 5×5 image and a 3×3 kernel there are 3×3 = 9 possible positions: the output is 3×3. In general, without padding at the borders, the output has dimensions (H − k + 1) × (W − k + 1): the image "shrinks" by k − 1 pixels per side. In the next lesson you will see how to control this with padding and stride.

Hand implementation: a vertical-edge detector

We build a synthetic 8×8 image with a bright 4×4 square in the center on a black background, and apply the vertical Sobel filter, a classic edge detector that compares the right side with the left side of a window, giving more weight to the middle row. Then we verify the result with SciPy (correlate2d) and, right after, with PyTorch: two independent implementations.

import numpy as np
from scipy.signal import correlate2d

img = np.zeros((8, 8))
img[2:6, 2:6] = 1.0                                          # bright square in the centre

sobel_v = np.array([[-1, 0, 1],
                    [-2, 0, 2],
                    [-1, 0, 1]], dtype=float)                # right minus left, middle row weighted double

def correlazione(I, K):
    k = K.shape[0]
    out = np.zeros((I.shape[0] - k + 1, I.shape[1] - k + 1))
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            out[i, j] = (I[i:i + k, j:j + k] * K).sum()      # sum of the products in the window
    return out

uscita = correlazione(img, sobel_v)
print("input shape", img.shape, " output shape", uscita.shape, " (8 - 3 + 1 = 6)")
print(uscita)
print("matches scipy.signal.correlate2d:", np.allclose(uscita, correlate2d(img, sobel_v, mode="valid")))

input shape (8, 8)  output shape (6, 6)  (8 - 3 + 1 = 6)
[[ 1.  1.  0.  0. -1. -1.]
 [ 3.  3.  0.  0. -3. -3.]
 [ 4.  4.  0.  0. -4. -4.]
 [ 4.  4.  0.  0. -4. -4.]
 [ 3.  3.  0.  0. -3. -3.]
 [ 1.  1.  0.  0. -1. -1.]]
matches scipy.signal.correlate2d: True

What the code does. correlazione runs through the positions (i, j) and for each one extracts the window I[i:i+k, j:j+k], multiplies it by the kernel element by element and sums: it is the formula, transcribed. The output has shape 6×6, as predicted by 8 − 3 + 1. Read the numbers: the two columns on the left have positive values (1, 3, 4, 4, 3, 1) in the rows where the square begins, where the window sees black on the left and white on the right (right minus left is positive); the central part is zero, where the window sees uniform brightness; the columns on the right are negative (−1, −3, −4, ...), where the window sees white on the left and black on the right.

The filter responds strongly only at the vertical edges of the square and signals the polarity of the edge with the sign. The values change in intensity along the rows (1, 3, 4, 4, 3, 1) because near the corners of the square the window sees the edge only in part. The check with SciPy confirms that our function does what correlate2d does in valid mode (the output contains only the elements that do not depend on zero padding, SciPy documentation in the sources). The next listing does the same check with PyTorch.

The following block reuses img, sobel_v and uscita from the previous listing. It was run with PyTorch 2.12.0 on CPU (CUDA_VISIBLE_DEVICES="", 2 threads), twice, with identical output. There is nothing random, so no seed is needed.

import torch
t = torch.nn.functional.conv2d(torch.tensor(img)[None, None], torch.tensor(sobel_v)[None, None])     # (N=1, C=1, H, W)
print("matches torch conv2d:", np.allclose(t[0, 0].numpy(), uscita))

matches torch conv2d: True

PyTorch's conv2d function is therefore exactly this correlation, with the dimensions in the order (N, C, H, W): that is why the image and the kernel were wrapped in two dimensions of length 1 (batch and channel).

Changing the kernel changes the characteristic that the output measures. The blur filter (all weights 1/9) computes the mean of the window, and smooths out the details. The transposed Sobel filter (the same matrix rotated) detects horizontal edges. A "sharpening" filter emphasizes the difference between the central pixel and its neighbors. For decades these filters were designed by hand by computer-vision engineers; the breakthrough of convolutional networks is that it is no longer necessary to do so.

A convolution is a dense layer with many constraints

How does the convolution connect to the dense layers you know? In a very instructive way: a convolution is a dense layer, with a weight matrix that has a particular structure. Let us verify it on a small image. We build a matrix M such that M · vec(I) (that is, the image "unrolled" into a vector) gives the same output as the convolution, and count how many elements of M are nonzero and how many distinct values there are.

rng = np.random.default_rng(0)
H = W = 5
k = 3
I = rng.normal(size=(H, W))
K = rng.normal(size=(k, k))
Ho = H - k + 1                                              # side of the output

M = np.zeros((Ho * Ho, H * W))                              # one row per output cell, one column per input pixel
for i in range(Ho):
    for j in range(Ho):
        for u in range(k):
            for v in range(k):
                M[i * Ho + j, (i + u) * W + (j + v)] = K[u, v]

print("M @ vec(I) matches the convolution:", np.allclose(M @ I.ravel(), correlazione(I, K).ravel()))
print("shape of M:", M.shape, "  total elements:", M.size, "  non-zero:", int((M != 0).sum()), "  distinct values:", len(np.unique(M[M != 0])))
print("parameters of an equivalent dense layer: %d   kernel parameters: %d" % (M.size, K.size))

M @ vec(I) matches the convolution: True
shape of M: (9, 25)   total elements: 225   non-zero: 81   distinct values: 9
parameters of an equivalent dense layer: 225   kernel parameters: 9

What the code does. For each cell (i, j) of the output it writes into the corresponding row of M the nine values of the kernel in the nine columns that correspond to the pixels of the window, and leaves everything else at zero. The matrix-vector product reproduces the convolution. The matrix is 9×25 = 225 elements, of which only 81 are nonzero (9 rows × 9 values), and those 81 elements take only 9 distinct values: those of the kernel, repeated at all positions.

This is weight sharing seen in matrix form: a dense layer with the same function would have 225 free parameters; the convolution has 9, because 64% of the matrix is zero because of local connectivity and the nonzero values are constrained to be equal. This explains two facts: why convolutions are much cheaper than a dense layer, and why they are a more restrictive model, with a strong assumption (the structure is local and translation-invariant) that is true for images and therefore helps generalization.

Filters are learned: backpropagation through a convolution

The kernel is nothing but a set of weights, and weights are learned with gradient descent. To verify it we run a controlled experiment: we generate 20 random 12×12 images and compute the outputs of the true Sobel filter. Then we start from a random kernel and make it learn to produce those outputs, minimizing the mean squared error. If everything works, the learned kernel must be the Sobel.

We need the gradient with respect to the kernel. For an output O[i, j] = Σ K[u, v]·I[i+u, j+v] and an error δ[i, j] = ∂J/∂O[i, j] (the derivative of the loss with respect to the output at position (i, j)), the chain rule gives, summing over all the positions where the weight K[u, v] was used (weight sharing means that each weight affects the loss through many outputs):

∂J/∂K[u, v] = Σ_{i, j}  δ[i, j] · I[i + u, j + v]

The gradient of the kernel is therefore the correlation between the input image and the error map δ. The sign and the form are those you know: "error of the output" times "input that contributed", summed over all positions.

The code that follows uses random images (uniform values in [0, 1)) because a target obtained with the true Sobel makes the result verifiable: if the learned kernel coincides with the Sobel, the gradient is right. It uses full-batch gradient descent rather than mini-batch to eliminate the noise of sampling, and a learning rate of 0.3, chosen to make the problem converge in a few hundred steps; in a separate trial, with learning rate 1.0 and the same problem, the loss exploded instead of going down (it is not in the listing).

rng = np.random.default_rng(0)
immagini = [rng.random((12, 12)) for _ in range(20)]
bersagli = [correlazione(I, sobel_v) for I in immagini]             # the outputs we want to obtain

kernel = rng.normal(size=(3, 3)) * 0.1                              # small random initialization
for passo in range(401):
    grad, perdita = np.zeros((3, 3)), 0.0
    for I, bersaglio in zip(immagini, bersagli):
        uscita = correlazione(I, kernel)
        delta = uscita - bersaglio                                   # error at every position
        perdita += 0.5 * (delta ** 2).mean()
        for u in range(3):
            for v in range(3):
                grad[u, v] += (delta * I[u:u + delta.shape[0], v:v + delta.shape[1]]).mean()   # correlation of delta with the input
    kernel -= 0.3 * grad / len(immagini)                             # gradient descent
    if passo in (0, 10, 50, 100, 400):
        print("step %3d   loss %.2e" % (passo, perdita / len(immagini)))

print("learned kernel:\n", np.round(kernel, 3))
print("true Sobel kernel:\n", sobel_v)

step   0   loss 4.77e-01
step  10   loss 2.89e-01
step  50   loss 3.93e-02
step 100   loss 3.29e-03
step 400   loss 1.39e-09
learned kernel:
 [[-1. -0.  1.]
 [-2. -0.  2.]
 [-1.  0.  1.]]
true Sobel kernel:
 [[-1.  0.  1.]
 [-2.  0.  2.]
 [-1.  0.  1.]]

What the code does. The outer loop is full-batch gradient descent (the 20 images). For each image it computes the output with the current kernel, the error δ with respect to the target, and accumulates the gradient with the formula above: for each pair (u, v) it multiplies δ by the window of the image shifted by (u, v) (I[u:u+h, v:v+w]) and takes the mean. Then it moves the kernel by 0.3 times the mean gradient.

The loss falls from about 0.48 to about 1.4·10⁻⁹ in 400 steps, and the learned kernel coincides with the Sobel: [[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]] up to rounding. Nobody told the model what to look for: it discovered the edge detector on its own by looking only at input and output examples. This, on a much larger scale (millions of images, dozens of layers, hundreds of filters per layer), is what a trained convolutional network does: it discovers the filters useful for the task.

More filters and more channels: the shapes of a real convolutional layer

A single filter looks for a single pattern. A convolutional layer has many in parallel: each filter produces a feature map (an output "channel"), and the maps are stacked. If the input has several channels (for example 3, the colors), each filter has a kernel for each input channel, and the response is obtained by summing the correlations over all channels. The shapes, which you must learn by heart, are:

input:               (N, C_in, H, W)
layer weights:       (C_out, C_in, k, k)      one k×k kernel for each pair (input channel, filter)
bias:                (C_out,)                  one per filter
output:              (N, C_out, H − k + 1, W − k + 1)

O[n, o, i, j] = b[o] + Σ_c Σ_u Σ_v  K[o, c, u, v] · I[n, c, i + u, j + v]

parameters = C_out · (C_in · k · k + 1)

The number of parameters does not depend on the image size (H, W): this is the reason for their economy. For an RGB input and 16 filters of 3×3 that is 16·(3·9 + 1) = 448 parameters, against the 150 million from before. Let us verify the formula, and rebuild the multichannel convolution by hand to understand how the channels combine:

def conv_multicanale(x, pesi, bias):
    N, C_in, H, W = x.shape
    C_out, _, k, _ = pesi.shape
    out = np.zeros((N, C_out, H - k + 1, W - k + 1))
    for n in range(N):
        for o in range(C_out):
            out[n, o] = bias[o] + sum(correlazione(x[n, c], pesi[o, c]) for c in range(C_in))   # sum over the input channels
    return out

rng = np.random.default_rng(0)
pesi = rng.normal(size=(16, 3, 3, 3)) * 0.1
bias = rng.normal(size=16) * 0.1
print("weight shape:", pesi.shape, "(C_out, C_in, k, k)    bias shape:", bias.shape)
print("parameters:", pesi.size + bias.size, "  formula 16 * (3*3*3 + 1) =", 16 * (3 * 3 * 3 + 1))

x = rng.normal(size=(2, 3, 10, 10))                                   # a batch of 2 RGB images, 10x10
y = conv_multicanale(x, pesi, bias)
print("input shape", x.shape, " -> output shape", y.shape, " (10 - 3 + 1 = 8)")

# one output cell rebuilt by hand: sum over channels and window, plus the bias
n, o, i, j = 1, 5, 2, 3
a_mano = bias[o] + sum((pesi[o, c] * x[n, c, i:i + 3, j:j + 3]).sum() for c in range(3))
con_scipy = bias[o] + sum(correlate2d(x[n, c], pesi[o, c], mode="valid") for c in range(3))
print("cell (n=1, filter=5, i=2, j=3): by hand %.6f   function %.6f   scipy %.6f" % (a_mano, y[n, o, i, j], con_scipy[i, j]))

weight shape: (16, 3, 3, 3) (C_out, C_in, k, k)    bias shape: (16,)
parameters: 448   formula 16 * (3*3*3 + 1) = 448
input shape (2, 3, 10, 10)  -> output shape (2, 16, 8, 8)  (10 - 3 + 1 = 8)
cell (n=1, filter=5, i=2, j=3): by hand 1.253561   function 1.253561   scipy 1.253561

What the code does. conv_multicanale is the formula with the four summations: for each image of the batch and for each filter it sums the correlations of the individual channels and adds the bias. The weights have shape (16, 3, 3, 3), that is 16 filters, each with a 3×3 kernel for each of the 3 channels, and the parameters are 16·27 + 16 = 448, as from the formula. The output for a batch of 2 images of 10×10 is (2, 16, 8, 8): 16 feature maps of side 8.

The last part computes a single cell in two independent ways (with explicit sums and with SciPy's correlate2d) and the values coincide with those of the function. The weights are random and small: the lesson is about shapes and calculations, not values. The same layer in PyTorch is Conv2d. The next listing verifies the shape of its weights and the number of parameters, then copies in the weights and bias of our function (in float64, because our arrays are) and compares the two outputs. It reuses torch imported in the previous block, pesi, bias, x, y and the indices n, o, i, j of the previous listing; it was run with PyTorch 2.12.0 on CPU, twice, with identical output.

torch.manual_seed(0)
conv = torch.nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3)
print("weight shape:", tuple(conv.weight.shape), "(C_out, C_in, k, k)    bias shape:", tuple(conv.bias.shape))
print("parameters:", sum(p.numel() for p in conv.parameters()), "  formula 16 * (3*3*3 + 1) =", 16 * (3 * 3 * 3 + 1))

with torch.no_grad():                                                 # same weights as the NumPy version, in float64
    conv.double()
    conv.weight.copy_(torch.tensor(pesi)); conv.bias.copy_(torch.tensor(bias))
    yt = conv(torch.tensor(x))
print("output shape", tuple(yt.shape), "  maximum difference from conv_multicanale: %.2e" % np.abs(yt.numpy() - y).max())
print("cell (n=1, filter=5, i=2, j=3): conv2d %.6f   function %.6f" % (yt[n, o, i, j].item(), y[n, o, i, j]))

weight shape: (16, 3, 3, 3) (C_out, C_in, k, k)    bias shape: (16,)
parameters: 448   formula 16 * (3*3*3 + 1) = 448
output shape (2, 16, 8, 8)   maximum difference from conv_multicanale: 6.66e-16
cell (n=1, filter=5, i=2, j=3): conv2d 1.253561   function 1.253561

What the code does. Conv2d(3, 16, 3) creates the 16 filters with weight shape (16, 3, 3, 3), as in our function, and 448 parameters. After copying the weights, its output differs from ours by at most 6.66·10⁻¹⁶, that is by a rounding: our conv_multicanale computes the same thing as the library layer. That difference depends on the order of the sums, and may change with the machine and the version.

More layers: the receptive field and the hierarchy

A single neuron of a 3×3 layer sees 9 pixels. But if you stack a second 3×3 layer, each of its neurons sees a 3×3 window of neurons of the first layer, and each of these sees 3×3 pixels: the neuron of the second layer depends on a 5×5 region of the original image. This region is called the receptive field. With kernel k and stride 1 each new layer widens it by k − 1 pixels: after L layers of 3×3 the receptive field has side 1 + 2L. With three 3×3 layers you see 7×7 pixels, with a fraction of the parameters of a single 7×7 filter (3 layers: 3·9 = 27 weights per channel against 49), and with two more nonlinearities, which increase expressiveness.

This is the hierarchical principle that makes convolutional networks powerful: the early layers, with a small receptive field, detect simple characteristics (edges, corners, color blobs); the intermediate layers combine them into more complex structures (textures, parts of objects); the final layers, with a wide receptive field, recognize whole objects. Nobody programs this hierarchy: it emerges from training, and the visualization of the filters learned in the first layers of networks trained on natural images often shows oriented edge and color detectors, very similar to the Sobel you have just made learn by hand.

def campo_ricettivo(strati):                         # strati: list of (kernel, stride); recursive formula
    rf, salto = 1, 1                                 # 'salto' = distance in pixels between two adjacent neurons of the current layer
    for k, stride in strati:
        rf += (k - 1) * salto
        salto *= stride
    return rf

print("receptive field, 1 layer 3x3:  ", campo_ricettivo([(3, 1)]))
print("receptive field, 2 layers 3x3: ", campo_ricettivo([(3, 1)] * 2))
print("receptive field, 3 layers 3x3: ", campo_ricettivo([(3, 1)] * 3), "(= 1 + 2*3)")
print("weights per channel: 3 layers 3x3 = %d  versus a single 7x7 filter = %d" % (3 * 9, 49))
print("with stride 2 in the first layer: ", campo_ricettivo([(3, 2), (3, 1), (3, 1)]), "(stride widens the field faster)")

receptive field, 1 layer 3x3:   3
receptive field, 2 layers 3x3:  5
receptive field, 3 layers 3x3:  7 (= 1 + 2*3)
weights per channel: 3 layers 3x3 = 27  versus a single 7x7 filter = 49
with stride 2 in the first layer:  11 (stride widens the field faster)

What the code does. The function applies a formula that keeps track of the receptive field (rf) and of the distance in input pixels between two adjacent neurons of the current layer (salto): each layer adds (k − 1)·salto to the receptive field, and a stride greater than 1 multiplies the jump by the stride. With stride 1 you find the result 1 + 2L. With stride 2 in the first layer, the three layers see 11 pixels instead of 7: stride widens the field faster, at the expense of the resolution of the maps (next lesson).

Trade-offs and real-world pitfalls

• Convolution or cross-correlation: libraries do not flip the kernel. If you compare with a "textbook" convolution (for example scipy.signal.convolve2d), the result is flipped; that is why above we used correlate2d.
• Order of the dimensions: (N, C, H, W) in PyTorch, (N, H, W, C) in Keras and in most image libraries. Passing a tensor with the wrong order does not always raise an error: the network treats the height as if it were a channel.
• Integer pixels that overflow: an image loaded as uint8 (0-255) cannot bear arithmetic operations between 8-bit integers; the example that was run is below the list. Convert to float and normalize before computing anything.
• Unnormalized pixels: values from 0 to 255 fed to a network produce huge pre-activations, disproportionate gradients and unstable training. Divide by 255 and standardize with the training statistics.
• Shrinking at the borders: every "valid" convolution removes k − 1 pixels. After many layers the image could vanish, and border pixels contribute to fewer outputs than central ones. Padding is used (next lesson).
• Forgetting the nonlinearity between convolutions: two consecutive convolutions without activation are equivalent to a single convolution with a larger kernel, for the same reason that two linear layers collapse into one.
• Thinking that convolution is translation-invariant: it is equivariant (the response moves with the pattern); invariance (the final response does not change) comes from pooling and the final layers, which we discuss in the next lesson.
• Too many filters or channels: parameters scale with C_in·C_out·k². A network with 512 input and output channels with 3×3 kernels has 512·512·9 ≈ 2.4 million parameters in a single layer.

The eight entries do not carry the same weight. The first two and the one on integer pixels concern the preparation of data and shapes: they are the ones that produce wrong results without any error. The entries on shrinking, nonlinearity and invariance are conceptual mistakes, avoided by understanding what the convolution does and does not do; the last one recalls that the number of parameters grows with the product of the channels, and therefore is paid for mostly in the deep layers. The case of integer pixels can be checked in two lines:

a, b = np.array([200], dtype=np.uint8), np.array([100], dtype=np.uint8)
print("mean in uint8:", (a + b) // 2, "  mean in float:", (a.astype(float) + b) / 2)

mean in uint8: [22]   mean in float: [150.]

The sum 200 + 100 = 300 does not fit in 8 bits and "wraps around" to 300 − 256 = 44, so the mean comes out as 22 instead of 150. In the NumPy run it issued no warning: the absence of an error does not mean the calculation is right, and the only protection is to convert to float first.

Frequently asked questions

Must the kernel be 3×3? No, but 3×3 is the most common choice: stacking several 3×3 layers gives wide receptive fields with fewer parameters, and more nonlinearity. Larger kernels (5×5, 7×7) appear mostly in the first layer of some networks, and 1×1 kernels have a specific role (lesson on residual blocks and Inception).

How many filters are needed? It is a hyperparameter: typically a few dozen in the early layers and hundreds in the last ones, with more channels as the spatial resolution decreases. It is chosen by looking at the validation set.

Does a convolution work only on images? No. It works on any data with local and regular structure: time series (1D convolution), audio, text (1D convolutions on word vectors), medical volumes (3D convolutions). The question to ask is whether it makes sense for local patterns to hold everywhere.

Why "feature map"? Because each output channel is a spatial map of the presence of a certain characteristic: high values indicate where, in the image, the filter "found" its pattern.

What you can do now

You can define the convolution (in practice a cross-correlation) and compute it by hand and in code, show that it is a dense layer with a highly structured matrix (9 parameters instead of 225 in the example) and make a random kernel learn an edge detector with gradient descent. You know the shapes of a convolutional layer, the parameter formula C_out·(C_in·k² + 1) and the receptive field of stacked layers. The lesson builds on the network trained by hand on digits and on backpropagation; in the next one, stride, padding and pooling, you complete the vocabulary and the counting of shapes and parameters of a network.