54 - Reading a CNN: feature maps, global pooling and U-Net

A neural network has a reputation as a black box: you feed in an image, a class comes out, and in between sit thousands of incomprehensible numbers. The reputation is partly deserved and partly not. Convolutional networks have a peculiarity: their intermediate layers are images (feature maps), and images can be looked at. What lies between input and output can therefore be inspected with PyTorch hooks, summarised with a per-channel average (global pooling), used to derive a map of what the network looked at (CAM) and, with the U-shaped architecture, carried from the classification of an image to that of every pixel (segmentation).

Prerequisites: you know stride, padding and pooling (lesson on counting parameters), you have trained a small CNN with early stopping (lesson on LeNet) and you know residual networks (lesson on residual networks). All the listings were run with PyTorch 2.12.0 and scikit-learn 1.9.0 on CPU, twice in a row, with identical output (measured on 2026-11-12). One result will surprise you in an instructive way: a model that segments nothing can reach 85% pixel accuracy.

The vocabulary

• Feature map: the output of a convolutional layer, a tensor (channels, height, width) in which each channel is a spatial map of the presence of a certain pattern.
• Hook: a function that PyTorch calls automatically when a module runs its forward pass, and that lets you "intercept" intermediate activations without modifying the model.
• Global average pooling (GAP): a pooling whose window is as large as the whole map, reducing each channel to its mean: from (C, H, W) to a vector of C numbers.
• Class Activation Map (CAM): a spatial map showing which regions of the image contributed most to the prediction of a given class.
• Semantic segmentation: assigning a class to every pixel (for example "object" or "background"), instead of one class to the whole image.
• IoU (Intersection over Union): the standard segmentation metric, the ratio between the area shared by the predicted and true masks and the area of their union. It is 1 if they coincide, 0 if they do not touch.
• Transposed convolution: the operation that increases spatial resolution, learning the weights of an "enlargement".

The seven entries follow the three topics of the lesson. Feature maps and hooks let you look inside; global average pooling and CAM are the pair that cuts parameters and then uses that reduction to explain the prediction; segmentation, IoU and transposed convolution belong to the U-Net part. IoU is the entry to remember in any case: further on you will see a useless model with 85% accuracy and an IoU unlike that of a model that really works.

Looking inside: extracting the feature maps

We build a small CNN in two versions (one with the classic dense head at the end, the other with the global pooling we will explain shortly) and train it on the digits. We use the same data prepared in the previous lessons: a stratified 60/20/20 split, normalisation with the training statistics.

A note on the code: the Python blocks in this article are meant to be run in order, in a single script or notebook. Each reuses the imports and variables defined in the previous blocks.

import numpy as np
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X_tmp, X_te, y_tmp, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
X_tr, X_va, y_tr, y_va = train_test_split(X_tmp, y_tmp, test_size=0.25, stratify=y_tmp, random_state=0)
mean, std = X_tr.mean(), X_tr.std()
to_t = lambda a: torch.tensor(((a - mean) / std).reshape(-1, 1, 8, 8), dtype=torch.float32)
T_tr, T_va, T_te = to_t(X_tr), to_t(X_va), to_t(X_te)
L_tr, L_va, L_te = [torch.tensor(a, dtype=torch.long) for a in (y_tr, y_va, y_te)]

class Net(nn.Module):
    def __init__(self, gap):
        super().__init__()
        self.gap = gap
        self.conv1 = nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2))     # (1,8,8)  -> (8,4,4)
        self.conv2 = nn.Sequential(nn.Conv2d(8, 16, 3, padding=1), nn.ReLU())                      # (8,4,4)  -> (16,4,4): the last map stays 4x4
        self.head = nn.Linear(16, 10) if gap else nn.Sequential(nn.Flatten(), nn.Linear(16 * 4 * 4, 10))

    def maps(self, x):                                        # the maps of the last convolutional layer
        return self.conv2(self.conv1(x))

    def forward(self, x):
        f = self.maps(x)
        return self.head(f.mean(dim=(2, 3))) if self.gap else self.head(f)       # GAP: mean over height and width

def train(model, seed, epochs=200, patience=10):
    torch.manual_seed(seed)
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    loss_fn = nn.CrossEntropyLoss()
    loader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(T_tr, L_tr), batch_size=32, shuffle=True)
    best = (np.inf, None, 0)
    for epoch in range(epochs):
        model.train()
        for xb, yb in loader:
            opt.zero_grad(); loss_fn(model(xb), yb).backward(); opt.step()
        model.eval()
        with torch.no_grad():
            vl = loss_fn(model(T_va), L_va).item()
        if vl < best[0] - 1e-4:
            best = (vl, {k: v.clone() for k, v in model.state_dict().items()}, epoch)
        elif epoch - best[2] >= patience:
            break
    model.load_state_dict(best[1]); model.eval()
    with torch.no_grad():
        return (model(T_te).argmax(1) == L_te).float().mean().item()

torch.manual_seed(0)
net = Net(gap=True)
print("test accuracy of the GAP network (seed 0): %.3f" % train(net, 0))

# a "hook" intercepts the output of conv1 without modifying the model
captured = {}
hook = net.conv1[0].register_forward_hook(lambda module, inp, out: captured.update(maps=out.detach()))
digit = T_te[:1]                                              # the first test image
prediction = net(digit).argmax(1).item()
hook.remove()
print("true label %d   prediction %d   shape of the captured maps: %s" % (L_te[0].item(), prediction, tuple(captured["maps"].shape)))

def ascii_map(m, levels=" .:-=+*#%@"):
    m = (m - m.min()) / (m.max() - m.min() + 1e-9)
    return ["".join(levels[int(v * (len(levels) - 1))] * 2 for v in row) for row in m]

print("\ninput digit (8x8):")
print("\n".join(ascii_map(digit[0, 0].numpy())))
strength = captured["maps"][0].abs().mean(dim=(1, 2))
best_ch = strength.argsort(descending=True)[:2].tolist()
for k in best_ch:
    print("\nconv1 channel %d (mean activation %.2f):" % (k, strength[k]))
    print("\n".join(ascii_map(captured["maps"][0, k].numpy())))
    print("3x3 kernel of channel %d:\n%s" % (k, np.round(net.conv1[0].weight[k, 0].detach().numpy(), 2)))

test accuracy of the GAP network (seed 0): 0.961
true label 7   prediction 7   shape of the captured maps: (1, 8, 8, 8)

input digit (8x8):
    ..##@@++    
    ****--@@..  
    ##..  @@::  
      ::++@@--  
      ##@@@@++  
      ::##      
      **++      
      @@::      

conv1 channel 3 (mean activation 3.56):
****==::..--==**
##**==..  --==**
##**++--  ::--**
####++--    --++
##@@**--....++**
##@@++::..--****
##%%++----**%%##
**##======**##**
3x3 kernel of channel 3:
[[-0.82 -0.84 -1.12]
 [-0.77 -0.54 -1.02]
 [-1.36 -0.91 -1.15]]

conv1 channel 1 (mean activation 3.36):
++##**====++++**
####==....==++**
**==--====++==**
****####++====**
**%%**==    ::++
**%%**==....++**
**%%##++==++####
##%%++====##@@##
3x3 kernel of channel 1:
[[-1.49 -1.21 -1.68]
 [-1.45 -1.31 -1.56]
 [-0.03  1.08  1.36]]

What the code does. The Net class has two variants: with gap=True the final part computes the mean of each of the 16 maps and feeds it to a linear layer; with gap=False it flattens the 16×4×4 maps into 256 values and uses a wider linear layer. train is the early-stopping loop you have already seen. The central block uses a hook: register_forward_hook registers on a module a function that receives module, input and output every time the module runs; here it saves the output of conv1[0] in a dictionary, then the hook is removed (remove()), because otherwise it would stay active forever. It is the standard technique for inspecting a model without rewriting it. The ascii_map function normalises a map to [0, 1] and draws it with ever "denser" characters (space, dot, colon, ... up to the at sign): a way to "see" activations without graphics libraries.

How to read the result. The 8×8 input image is the test digit; the two maps printed are the two channels of conv1 with the highest mean response, and under each one is the 3×3 kernel that channel has learned. Compare each map with the digit. Channel 3 has a kernel with all negative weights: after normalisation the background is about −0.8 and the stroke is positive, so that filter responds where the image is dark, and it acts as a "negative" of the digit (in the map the dense areas are the background). Channel 1 has negative weights in the top row of the kernel and positive ones in the bottom row: it responds to vertical changes, that is, to horizontal edges.

These are two recognisable behaviours, but do not expect "clean" kernels like the Sobel one in the lesson on convolution: with 8 filters and 1078 examples (60% of 1797), the filters are noisy and often hard to interpret by eye. The point is not that every filter has a readable meaning: it is that you can look at what happens, and that for natural images and networks trained on large datasets this inspection has revealed filters sensitive to edges and colours in the first layers and, deeper down, to textures and parts of objects (M. Zeiler and R. Fergus, 2014; see Sources).

Global average pooling

In the lesson on stride, padding and pooling you saw that in a classic CNN the dense connection after the flatten holds most of the parameters (78% in LeNet-5). Global average pooling (M. Lin et al., 2014) removes this problem: each map (H × W) is reduced to its mean, and the vector of C means goes straight into a single linear layer towards the classes. For C channels and K classes the head has C·K + K parameters, independent of H and W.

head input: (N, C, H, W)  →  mean over (H, W)  →  (N, C)  →  Linear(C, K)  →  (N, K)

Three advantages: fewer parameters (and so less risk of overfitting); independence from resolution (since the mean works with any H and W, the network accepts images of sizes different from those used in training); and interpretability (each channel is directly tied to a per-class weight: we will use this shortly). The price is that the mean throws away the information on position: the network knows "how much" of each pattern there is, not "where". Let us measure parameters and accuracy over five seeds, and check the independence from resolution:

print("head            parameters  test accuracy (5 seeds)")
for gap in (False, True):
    results = []
    for seed in range(5):
        torch.manual_seed(seed)
        m = Net(gap)
        results.append(train(m, seed))
    print("%-15s  %9d   %.3f +- %.3f" % ("GAP" if gap else "flatten+dense", sum(p.numel() for p in m.parameters()), np.mean(results), np.std(results)))

gap_net, flat_net = Net(True), Net(False)
print("\n16x16 image with the GAP network:      output", tuple(gap_net(torch.zeros(1, 1, 16, 16)).shape))
try:
    flat_net(torch.zeros(1, 1, 16, 16))
except RuntimeError as e:
    print("16x16 image with flatten+dense:        error -", str(e)[:75])

head            parameters  test accuracy (5 seeds)
flatten+dense         3818   0.971 +- 0.002
GAP                   1418   0.958 +- 0.006

16x16 image with the GAP network:      output (1, 10)
16x16 image with flatten+dense:        error - mat1 and mat2 shapes cannot be multiplied (1x1024 and 256x10)

What the code does. For each head it trains five models with different seeds and reports the number of parameters and the mean test accuracy. The network with flatten and dense layer has 3,818 parameters and scores 0.971 ± 0.002; the one with GAP has 1,418 (63% fewer) and scores 0.958 ± 0.006. With these numbers the choice is a real compromise: global pooling costs a little over one percentage point on Digits, where the images are tiny and the position of the stroke matters, but it removes almost two thirds of the parameters. On the matter of sizes: the GAP network happily accepts a 16×16 input (the mean works at any resolution), while the flatten one raises an error because the number of values after the flatten is no longer the one the dense layer expects (here 1024 instead of 256). On large natural images, where the dense layer would have millions of parameters, GAP is the standard of modern architectures.

Class Activation Map: where did the network look?

With GAP there is an elegant observation. The logit of class c is a single weighted sum: logit_c = Σ_k w_{c,k} · mean_{i,j}(f_k(i,j)) + b_c, where f_k is the map of channel k and w_{c,k} the weight the head assigns to channel k for class c. Since the mean is linear, it can be moved inside the sum:

logit_c = mean_{i,j} [ Σ_k w_{c,k} · f_k(i,j) ] + b_c  =  mean_{i,j} CAM_c(i,j) + b_c

CAM_c(i,j) = Σ_k w_{c,k} · f_k(i,j)

The map CAM_c (Class Activation Map, B. Zhou et al., 2016) is a spatial map in which each position contributes to the logit of class c with its own share: where it is high, the network found evidence for that class. And by construction the mean of the CAM plus the bias is exactly the logit: it is an identity, not an approximation. We verify it:

model = Net(True)
train(model, 0)
x = T_te[:1]
f = model.maps(x)[0]                                         # (16, 4, 4): the maps of the last convolutional layer
w, b = model.head.weight, model.head.bias
logit = model(x)[0]
c = logit.argmax().item()
cam = (w[c][:, None, None] * f).sum(dim=0)                   # CAM_c(i,j) = sum_k w_{c,k} f_k(i,j)

print("predicted class: %d   (true: %d)" % (c, L_te[0].item()))
print("logit of the predicted class: %.5f" % logit[c].item())
print("mean of the CAM + bias:       %.5f   -> identity verified: %s" % ((cam.mean() + b[c]).item(), bool(torch.allclose(logit[c], cam.mean() + b[c], atol=1e-4))))
print("\nCAM of the predicted class (4x4), high values = evidence in favour:")
print(np.round(cam.detach().numpy(), 1))
print("\nas a character map (original 8x8 digit on the right):")
cam_up = torch.nn.functional.interpolate(cam[None, None], size=(8, 8), mode="bilinear", align_corners=False)[0, 0]
for r1, r2 in zip(ascii_map(cam_up.detach().numpy()), ascii_map(x[0, 0].numpy())):
    print(r1, "   ", r2)

predicted class: 7   (true: 7)
logit of the predicted class: 6.30565
mean of the CAM + bias:       6.30565   -> identity verified: True

CAM of the predicted class (4x4), high values = evidence in favour:
[[ -6.   -4.4 -55.9 -19.8]
 [ 18.2   9.3  17.6  11.9]
 [ 26.7  23.1  20.6  11.9]
 [ 12.7  13.3  22.3  -2.4]]

as a character map (original 8x8 digit on the right):
++++++--    ::--         ..##@@++    
++++++==::..--==         ****--@@..  
####******++****         ##..  @@::  
%%%%####%%%%####           ::++@@--  
@@%%%%%%%%%%####           ##@@@@++  
%%%%%%%%%%%%##**           ::##      
######%%%%%%****           **++      
########%%##**++           @@::      

What the code does. It takes the last map (16 channels of 4×4), the head weights for the predicted class and computes the weighted sum over the channels: that is the CAM, a 4×4 grid. The identity is verified to five decimals: the mean of the CAM plus the bias coincides with the logit. Then the CAM is enlarged to 8×8 with bilinear interpolation (interpolate) so it can be overlaid on the original image, and printed next to the digit with the same character function. The "denser" zones of the CAM are those where the evidence for the class is greatest. Here the map is very coarse because the last layer is 4×4, and not always intuitive: the top row carries negative evidence and the central ones almost all the positive evidence, without this translating into a "human" reading of the digit 7.

With large images, and wider final maps (7×7, 14×14), the result is a readable heat map showing which part of the object the network based its decision on. It is a valuable tool for checking that the model looks where it should: a CNN that classifies X-rays by looking at the text labels at the edges of the image rather than at the lung would have excellent metrics and no clinical use; the CAM would expose it.

From classification to segmentation: the U-Net

So far the network returned one class per image. For segmentation you need a class for every pixel: the output is a map with the same resolution as the input. A network that lowers the resolution with pooling and then restores it, however, does not have enough information to rebuild precise contours: in the deepest layer (say 4×4) the fine details are gone. The U-Net (O. Ronneberger, P. Fischer, T. Brox, 2015, born for biomedical image segmentation) solves the problem with a U-shaped structure:

• Descending path (encoder): blocks of convolutions followed by pooling; the resolution halves and the channels double. It captures the "what" (context, large shapes).
• Ascending path (decoder): brings the resolution back to the original with transposed convolutions (or interpolation), halving the channels.
• Skip connections: at each level, the encoder maps are concatenated with those of the decoder at the same resolution. They carry the "where" (fine details and contours) that the encoder had before reducing them.
• Output: a 1×1 convolution producing one logit per pixel (or one logit per class and pixel).

Read the four entries from top to bottom as a journey of information. The encoder spends resolution to gain context, the decoder gives it back but without the lost details, the skip connections bring those details back halfway, and the 1×1 output turns channels into one logit per pixel. If you remove the third entry you have a plain encoder-decoder: the experiment that follows measures how much that single entry is worth.

The drawing shows the shape. The orange arrows are the skip connections: they carry the encoder maps (at 16×16 and at 8×8) straight to the decoder. Before the implementation you need to understand how a map grows. A transposed convolution with kernel K, stride S and padding P takes a side H to (H − 1)·S − 2P + K: with K = 2, S = 2, P = 0 it doubles exactly (H → 2H), and it is the typical choice for the decoder. We verify it:

print("input side H, kernel K, stride S, padding P  ->  output side (formula and PyTorch)")
for H, K, S, P in ((4, 2, 2, 0), (4, 3, 2, 1), (8, 4, 2, 1)):
    actual = nn.ConvTranspose2d(1, 1, K, S, P)(torch.zeros(1, 1, H, H)).shape[-1]
    print("H=%d K=%d S=%d P=%d   ->   (H-1)*S - 2P + K = %d   PyTorch: %d" % (H, K, S, P, (H - 1) * S - 2 * P + K, actual))

input side H, kernel K, stride S, padding P  ->  output side (formula and PyTorch)
H=4 K=2 S=2 P=0   ->   (H-1)*S - 2P + K = 8   PyTorch: 8
H=4 K=3 S=2 P=1   ->   (H-1)*S - 2P + K = 7   PyTorch: 7
H=8 K=4 S=2 P=1   ->   (H-1)*S - 2P + K = 16   PyTorch: 16

What the code does. It applies the formula and compares it with the actual size produced by ConvTranspose2d; in all three cases they coincide (8, 7 and 16). The transposed convolution can be seen as the "shape" inverse of convolution: where a convolution with stride 2 halves, the transposed one with stride 2 doubles, and by learning its weights it chooses how to "fill in" the new pixels.

A segmentation experiment: U-Net against an encoder-decoder without skips

To measure the contribution of the skip connections we build a controlled synthetic task: 16×16 images with Gaussian noise, containing 1 to 3 bright squares of side 3-6 (the object to segment) and, in 70% of cases, a bright horizontal line that is a distractor (it is not part of the mask). The true masks are known, so we can measure IoU and pixel accuracy.

def generate(n, seed):
    r = np.random.default_rng(seed)
    X = np.zeros((n, 1, 16, 16), np.float32); Y = np.zeros((n, 1, 16, 16), np.float32)
    for i in range(n):
        img = r.normal(0, 0.35, (16, 16)).astype(np.float32)               # background noise
        mask = np.zeros((16, 16), np.float32)
        for _ in range(r.integers(1, 4)):                                   # 1 to 3 squares
            side = r.integers(3, 7)
            a, b = r.integers(0, 16 - side + 1, 2)
            mask[a:a + side, b:b + side] = 1
        if r.random() < 0.7:
            img[r.integers(0, 16), :] += 0.6                                # bright line: distractor
        X[i, 0] = img + 0.9 * mask; Y[i, 0] = mask
    return torch.tensor(X), torch.tensor(Y)

X_seg_tr, Y_seg_tr = generate(600, 0)
X_seg_te, Y_seg_te = generate(200, 1)

def block(c_in, c_out):
    return nn.Sequential(nn.Conv2d(c_in, c_out, 3, padding=1), nn.ReLU(), nn.Conv2d(c_out, c_out, 3, padding=1), nn.ReLU())

class UNet(nn.Module):
    def __init__(self, skips=True):
        super().__init__()
        self.skips = skips
        self.e1, self.e2, self.bottom = block(1, 8), block(8, 16), block(16, 32)
        self.up2 = nn.ConvTranspose2d(32, 16, 2, 2); self.d2 = block(32 if skips else 16, 16)
        self.up1 = nn.ConvTranspose2d(16, 8, 2, 2);  self.d1 = block(16 if skips else 8, 8)
        self.out = nn.Conv2d(8, 1, 1)                                       # 1x1: one logit per pixel
        self.pool = nn.MaxPool2d(2)

    def forward(self, x):
        e1 = self.e1(x)                                                     # 16x16, 8 channels
        e2 = self.e2(self.pool(e1))                                         # 8x8, 16 channels
        bottom = self.bottom(self.pool(e2))                                 # 4x4, 32 channels
        d2 = self.up2(bottom)                                               # 8x8, 16 channels
        d2 = self.d2(torch.cat([d2, e2], dim=1) if self.skips else d2)      # concatenation = skip connection
        d1 = self.up1(d2)                                                   # 16x16, 8 channels
        d1 = self.d1(torch.cat([d1, e1], dim=1) if self.skips else d1)
        return self.out(d1)

def evaluate_segmentation(model, X, Y):
    model.eval()
    with torch.no_grad():
        p = (model(X) > 0).float()                                         # logit > 0  <=>  probability > 0.5
    inter, union = (p * Y).sum(), ((p + Y) > 0).float().sum()
    return (inter / union).item(), (p == Y).float().mean().item()

print("accuracy of the model that always says 'background': %.3f" % (1 - Y_seg_te.mean().item()))
print("model                      parameters  IoU               pixel accuracy          (3 seeds)")
for skips in (True, False):
    rows = []
    for seed in range(3):
        torch.manual_seed(seed)
        m = UNet(skips)
        opt = torch.optim.Adam(m.parameters(), lr=3e-3)
        loss_fn = nn.BCEWithLogitsLoss()
        for _ in range(40):
            m.train()
            order = torch.randperm(len(X_seg_tr))
            for i in range(0, len(order), 20):
                idx = order[i:i + 20]
                opt.zero_grad(); loss_fn(m(X_seg_tr[idx]), Y_seg_tr[idx]).backward(); opt.step()
        rows.append(evaluate_segmentation(m, X_seg_te, Y_seg_te))
    r = np.array(rows)
    print("%-25s  %9d   %.3f +- %.3f      %.3f +- %.3f" % ("U-Net with skips" if skips else "no skips (enc-dec)", sum(p.numel() for p in m.parameters()), r[:, 0].mean(), r[:, 0].std(), r[:, 1].mean(), r[:, 1].std()))

accuracy of the model that always says 'background': 0.845
model                      parameters  IoU               pixel accuracy          (3 seeds)
U-Net with skips               29321   0.948 +- 0.004      0.992 +- 0.001
no skips (enc-dec)             26441   0.880 +- 0.004      0.980 +- 0.001

What the code does. generate builds the images and the true masks. The UNet class has a descending path (e1 at 16×16, e2 at 8×8, bottom at 4×4) and an ascending one with transposed convolutions (up2, up1); where skips=True it concatenates the encoder maps (torch.cat(..., dim=1), along the channels) before the decoder blocks, and that is why those blocks have twice the input channels (32 instead of 16 and 16 instead of 8). The output is a 1×1 convolution producing one logit per pixel, trained with BCEWithLogitsLoss (binary cross-entropy with a built-in sigmoid). evaluate_segmentation computes IoU and pixel accuracy after thresholding the logits at 0 (equivalent to probability 0.5).

Read the table carefully. The first number printed is a lesson on metrics: a model that always says "background", without looking at the image, has a pixel accuracy of 0.845, because the object occupies a minority of the pixels. An accuracy of 0.98 is therefore not "98% skill": it is the same trap as the imbalanced classes of the first lesson of the module. IoU is not fooled (the "always background" model has IoU 0) and it is the metric to use. With the skip connections the U-Net reaches IoU 0.948 ± 0.004 (accuracy 0.992); without them, with the same structure (the only difference is the concatenation, with fewer parameters: 26,441 against 29,321), IoU 0.880 ± 0.004 (accuracy 0.980). Almost seven points of IoU more for 2,880 parameters, because the information on contours that the encoder has at 16×16 does not have to be rebuilt from a 4×4 bottleneck. The differences are many times larger than the deviations between seeds: no doubt that the skips matter.

Trade-offs and real-world traps

• Pixel accuracy in segmentation: misleading with small objects (0.845 for the model that does nothing). Use IoU, the Dice coefficient or per-class metrics.
• Forgotten hooks: a hook that is registered and not removed keeps running (and occupying memory) on every pass. Call remove().
• Over-interpreting feature maps: the individual filters of small networks almost never have a sharp meaning. Visualisations are an inspection and debugging tool, not proof of understanding the model.
• Low-resolution CAM: the map has the resolution of the last convolutional layer (4×4 here). It must be enlarged to overlay it, and it gives coarse indications, not precise contours.
• GAP and position information: the mean removes it. For tasks where position matters (letters in text, order of parts) it can hurt.
• Shapes in concatenations: the U-Net concatenation requires the same spatial resolution in encoder and decoder. With images whose side is not divisible by 2 at each pooling (for example 15) the sizes do not match. Choose sizes that are multiples of 2^(number of poolings) or pad.
• Transposed convolution and the "checkerboard": with kernels that are not multiples of the stride it can produce checkerboard artefacts; the alternative is interpolation (nearest or bilinear) followed by an ordinary convolution.
• Imbalance between object and background pixels: besides the choice of metric, weight the loss or use IoU/Dice-based losses (for example the Dice loss).
• Evaluating segmentation only on clean images: as in the lesson on CNNs and augmentation, a robustness test (noise, contrast) shows more than a figure on the original test set.

The nine traps fall into three groups. Four concern reading the results (pixel accuracy, over-interpreted filters, coarse CAM, testing only on clean images): the program runs, but the conclusion is wrong. Three concern the shape of the data (unremoved hook, GAP erasing position, concatenation with incompatible resolutions) and two the training of the segmentation (checkerboard artefacts, imbalance between object and background). The first is the costliest: in the listing a model that does nothing has 0.845 pixel accuracy, and only IoU unmasks it.

Frequently asked questions

Why does the U-Net concatenate while the ResNet adds? They are two choices with different purposes. The ResNet sum helps the gradient (lesson on residual blocks) and requires identical shapes. The U-Net concatenation preserves the information of both branches whole and lets the next block decide how to combine it, at the cost of more channels.

Does GAP always replace the dense layer? In modern architectures for large images, almost always. For small images and tasks where position is informative, as in Digits, it can cost a few points.

Does the CAM explain the network? It shows where the network found evidence for a class, not why that pattern leads it to the decision, nor whether the model reasons the right way. It is a verification tool (is it looking where it should?), not a complete explanation.

Can you do segmentation with a ResNet? Yes: many modern segmentation architectures use a (pretrained) ResNet encoder with a U-Net-style decoder.

Three exercises

Exercise 1. Print the conv2 maps (16 channels of 4×4) for two different digits, for example a 0 and a 1. Which channels light up differently? Compute the correlation between the channel vectors (after the GAP) of the two digits.

Exercise 2. Verify the CAM identity on all 360 test images: the maximum difference between the logit and the mean of the CAM plus bias should be of the order of 10⁻⁵.

Exercise 3. In the segmentation experiment, increase the noise (standard deviation 0.6) and compare U-Net with and without skips again. Does the advantage of the skips grow or shrink?

Measuring segmentation without being fooled

The obvious metric for segmentation, the fraction of correctly classified pixels, is also the most misleading. If the object occupies a small part of the image, a model that labels everything as background gets a very high pixel accuracy without having segmented anything: it is the same trap as the imbalanced classes seen in the first module, except that here the imbalance is inside every single image. That is why IoU (intersection over union) is used, per class: the area where prediction and truth coincide divided by the area covered by at least one of the two. IoU ignores the pixels both declare background, so it rewards only whoever finds the object and delimits it well, and it is 0 for a model that says "nothing" when there is something.

Two more cautions. First, report the IoU per class and not just the mean: a mean can hide a rare class that is completely wrong. Second, check the predictions by eye on a few examples with the mask overlaid on the image, because metrics do not tell a blurred contour from a shifted one, and the cause (and the remedy) differ.

As for the architecture, the reason for the U-Net skip connections is precisely the precision of the contours: the descending path lowers the resolution to understand what is there, the skips bring back the spatial detail to understand where.

What you can do now

You can intercept the feature maps of a CNN with a hook (and remove it), you know that the individual filters of a small network almost never have a sharp meaning, and you can replace flatten and the dense layer with global average pooling: on the digits the parameters go from 3,818 to 1,418 at a cost of about one point of accuracy (0.971 against 0.958). You can verify that the logit is the mean of a CAM plus the bias and read the CAM of a class without trusting a 4×4 map too much. You can build a U-Net with transposed convolutions and skip connections, and judge it with IoU: in the experiment the skips lifted the IoU from 0.880 to 0.948, while the model that always says "background" had 0.845 pixel accuracy. In the next lesson, transfer learning, you reuse an already trained CNN for a new task.

Previous lesson in the course: Dropout and Batch Normalization: regularizing and stabilizing a CNN.