52 - Residual networks, 1x1 convolution and Inception blocks

Imagine you have to proofread a hundred-page text. One method is to rewrite it from scratch, page by page: you risk making it worse, because every rewrite can introduce new mistakes. Another method is to leave the text as it is and write in the margin only the corrections: if a page is already right, the margin stays empty and the page goes through unchanged. A residual network (ResNet) builds its layers with the second method: each block does not rewrite the signal, it computes a correction and adds it to the incoming signal. This simple idea made it possible to train networks with tens, hundreds and even thousands of layers, and it won the ILSVRC image classification challenge in 2015.

Prerequisites: you know how backpropagation computes the gradient, you know stride, padding and pooling (lesson on counting parameters) and you have already trained a small convolutional network (lesson on LeNet with PyTorch). Here you find three tools that complete each other: the residual block, which lets you stack many layers; the 1×1 convolution, which changes the number of channels at low cost; the Inception block, which widens the network in parallel instead of deepening it. All the listings were run with PyTorch 2.12.0 and scikit-learn 1.9.0 on CPU, twice in a row, with identical output; the whole path takes about a minute on two threads (a single measurement, 2026-11-12).

The degradation problem

The lesson on backpropagation showed you the gradient that vanishes in a deep network with sigmoid, and the lesson on layered networks showed that good initialization and the ReLU mitigate it. One might expect that, with these techniques, adding layers would never hurt, at least on the training error: in the worst case the new layers could learn the identity function, and the deep network would do as well as the shallow one. In 2015 K. He and colleagues observed the opposite: on CIFAR-10 a «plain» network (without residual connections) with 56 layers had a higher training error than one with 20. It is not overfitting (the training error would be lower, not higher): it is an optimization problem. In theory a deep solution that is just as good exists (it would be enough for the added layers to be identities), but gradient descent does not manage to find it: learning an identity function with a stack of nonlinear layers is surprisingly hard. The phenomenon is called degradation.

The residual idea

The solution of He et al. is to change what the block has to learn. If the mapping a block should produce is H(x), instead of asking its layers to approximate H(x) directly, you ask them to approximate the residual F(x) = H(x) − x, and the output of the block is:

y = x + F(x)

The term x reaches the output through a shortcut connection (skip connection) that crosses no weight and no nonlinearity. If the optimal transformation is close to the identity (and for many layers it is), the block only has to bring F(x) close to zero, which is much easier than building the identity out of nonlinear layers: small weights are enough.

The drawing shows the block: the input x follows two paths. On top it crosses the residual branch (two 3×3 convolutions with a ReLU in between) and becomes F(x); on the bottom it passes unchanged along the shortcut. The two outputs are added element by element, and a ReLU closes the block. For the two paths to be addable they must have the same shape: we will see this shortly.

Why the gradient now gets through: the derivation

Stack L residual blocks, with x_{l+1} = x_l + F(x_l) (we simplify by removing the final ReLU, as the «pre-activation» variant of ResNet does). Unrolling the recurrence from block l to block L gives:

x_L = x_l + Σ_{i=l}^{L−1} F(x_i)

The output of a deep block is the input of a shallow block plus a sum of corrections. Now apply the chain rule to the loss E, differentiating with respect to x_l:

∂E/∂x_l = (∂E/∂x_L) · (∂x_L/∂x_l) = (∂E/∂x_L) · ( 1 + ∂/∂x_l Σ_{i=l}^{L−1} F(x_i) )

Look at the factor in parentheses: it contains a 1, which comes from the shortcut, plus a term that depends on the weights of the F branches. In the «plain» network the same derivative is a product of L − l matrices (one per layer), which tends to zero or to infinity exponentially; here the gradient ∂E/∂x_L reaches x_l directly, added (not multiplied) to the contribution of the branches. The term 1 guarantees that the gradient of the upper level does not vanish, no matter how small or large the weights of the branches are. It is a highway for the error.

The experiment: deep networks with and without shortcuts

We measure the effect with a minimal dense version that isolates the idea: a stack of dense layers of width 64, with ReLU, on Digits, in the «plain» variant (h = relu(W h)) and in the residual one (h = h + relu(W h)). A third residual variant scales the correction branch by 1/√L, for a reason we will see in the results. For each one we measure the norm of the gradient of the first layer (it tells us whether the error signal reaches it) and then we train.

import numpy as np
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X_tmp, X_te, y_tmp, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
X_tr, X_va, y_tr, y_va = train_test_split(X_tmp, y_tmp, test_size=0.25, stratify=y_tmp, random_state=0)
mean, std = X_tr.mean(), X_tr.std()
T_tr, T_va = [torch.tensor((a - mean) / std, dtype=torch.float32) for a in (X_tr, X_va)]
L_tr, L_va = torch.tensor(y_tr), torch.tensor(y_va)

class Plain(nn.Module):                                       # h <- relu(W h), L times
    def __init__(self, depth, width=64):
        super().__init__()
        self.input = nn.Linear(64, width)
        self.hidden = nn.ModuleList([nn.Linear(width, width) for _ in range(depth)])
        self.output = nn.Linear(width, 10)
    def forward(self, x):
        h = torch.relu(self.input(x))
        for layer in self.hidden:
            h = self.step(h, layer)
        return self.output(h)
    def step(self, h, layer):
        return torch.relu(layer(h))

class Residual(Plain):                                        # h <- h + relu(W h): the shortcut
    def step(self, h, layer):
        return h + torch.relu(layer(h))

class ScaledResidual(Plain):                                  # h <- h + (1/sqrt(L)) relu(W h)
    def step(self, h, layer):
        return h + torch.relu(layer(h)) / len(self.hidden) ** 0.5

def first_layer_gradient_norm(model):
    model.zero_grad()
    nn.CrossEntropyLoss()(model(T_tr), L_tr).backward()
    return model.input.weight.grad.norm().item()

print("gradient norm of the first layer (mean over the whole training set)")
print("depth        plain             residual          scaled residual")
for d in (5, 20, 50):
    values = []
    for cls in (Plain, Residual, ScaledResidual):
        torch.manual_seed(0)
        values.append(first_layer_gradient_norm(cls(d)))
    print("%5d        %-16.5f   %-16.5f   %.5f" % ((d,) + tuple(values)))

gradient norm of the first layer (mean over the whole training set)
depth        plain             residual          scaled residual
    5        0.00601            1.43556            0.69622
   20        0.00000            54.24111           0.65498
   50        0.00000            45191.06641        0.75554

What the code does. The three classes differ by a single line, the step method: in the plain network it is relu(layer(h)); in the residual one h + relu(layer(h)); in the third the branch is divided by √L. first_layer_gradient_norm computes the loss on the whole training set and reads the gradient norm of the first layer (weight.grad.norm()). Read the results. In the plain network the gradient of the first layer is 0.006 with 5 layers and practically zero (below 10⁻⁵) with 20 and 50: the error signal does not arrive. In the residual network the gradient arrives, and even too much: 1.4 with 5 layers, 54 with 20, over 45,000 with 50.

This is the reason for the third variant: without normalization or scaling, adding many corrections makes the activation and the gradient grow explosively. Real ResNets avoid the problem with batch normalization (next lesson) and careful initialization. With the branch scaled by 1/√L the gradient stabilizes around 0.7 for all depths: the signal arrives and does not explode.

def train(model, epochs=40, lr=1e-3):
    opt = torch.optim.Adam(model.parameters(), lr=lr)
    loss_fn = nn.CrossEntropyLoss()
    loader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(T_tr, L_tr), batch_size=32, shuffle=True)
    for _ in range(epochs):
        for xb, yb in loader:
            opt.zero_grad(); loss_fn(model(xb), yb).backward(); opt.step()
    model.eval()
    with torch.no_grad():
        return ((model(T_tr).argmax(1) == L_tr).float().mean().item(),
                (model(T_va).argmax(1) == L_va).float().mean().item(),
                loss_fn(model(T_tr), L_tr).item())

print("depth  model              train acc.  validation acc.  training loss")
for d in (4, 20, 50):
    for cls in (Plain, Residual, ScaledResidual):
        torch.manual_seed(0)
        a_tr, a_va, loss = train(cls(d))
        print("%5d  %-18s %10.3f  %15.3f  %13.4f" % (d, cls.__name__, a_tr, a_va, loss))

depth  model              train acc.  validation acc.  training loss
    4  Plain                   1.000            0.961         0.0008
    4  Residual                1.000            0.981         0.0005
    4  ScaledResidual          1.000            0.969         0.0012
   20  Plain                   0.101            0.100         2.3025
   20  Residual                1.000            0.975         0.0001
   20  ScaledResidual          1.000            0.967         0.0003
   50  Plain                   0.101            0.100         2.3025
   50  Residual                0.987            0.922         0.0573
   50  ScaledResidual          1.000            0.969         0.0002

What the code does. It trains each network with Adam for 40 epochs and reports training and validation accuracy. The plain network with 4 layers does fine (validation about 0.96). With 20 and 50 layers it does not learn: accuracy 0.10, that is chance, and loss 2.302 = ln 10, the value of someone who gives probability 1/10 to every class. This is degradation taken to the extreme: the deeper network is not worse because of overfitting, it does not learn at all. The residual versions learn at every depth: the simple residual reaches 0.975 with 20 layers and 0.922 with 50 (more unstable, because of the explosion we saw), the scaled one 0.967 with 20 layers and 0.969 with 50. The message, far more important than the decimals, is qualitative: the shortcut turns an impossible task into a solvable one.

An honest warning: this is a dense stack on an easy problem and it does not prove that more layers improve the results (at 20 and 50 layers the validation does not exceed that of the 4-layer network); it only shows that optimizing a deep network becomes possible.

A surprising property: removing a block

The formula x_L = x_l + Σ F(x_i) has a consequence that can be checked. If we write out the product of L blocks, each of which is «shortcut plus branch», and expand it, every block contributes either the identity term or the F term: in total you get 2^L different paths from input to output, some very short (all shortcuts) and others long (many branches). A ResNet with 20 blocks behaves like an ensemble of over a million networks of different depth that share their weights: this is the reading proposed by A. Veit, M. Wilber and S. Belongie (2016). If so, removing a single block should eliminate only a fraction of the paths and degrade the network little, whereas in a «plain» network removing a layer breaks the only chain. Let us try:

def accuracy_without_block(model, k, T, L):
    model.eval()
    with torch.no_grad():
        h = torch.relu(model.input(T))
        for i, layer in enumerate(model.hidden):
            if i == k:
                continue                                           # block k is skipped
            h = model.step(h, layer)
        return (model.output(h).argmax(1) == L).float().mean().item()

torch.manual_seed(0)
residual20 = ScaledResidual(20); train(residual20)
torch.manual_seed(0)
plain4 = Plain(4); train(plain4)

print("residual, 20 blocks:  full %.3f   without block 0, 5, 10, 15, 19: %s" % (
    accuracy_without_block(residual20, -1, T_va, L_va), [round(accuracy_without_block(residual20, k, T_va, L_va), 3) for k in (0, 5, 10, 15, 19)]))
print("plain, 4 layers:      full %.3f   without layer 0, 1, 2, 3:     %s" % (
    accuracy_without_block(plain4, -1, T_va, L_va), [round(accuracy_without_block(plain4, k, T_va, L_va), 3) for k in range(4)]))
print("distinct paths in a stack of 20 residual blocks: 2^20 = %d" % 2 ** 20)

residual, 20 blocks:  full 0.967   without block 0, 5, 10, 15, 19: [0.964, 0.969, 0.967, 0.967, 0.967]
plain, 4 layers:      full 0.961   without layer 0, 1, 2, 3:     [0.117, 0.025, 0.203, 0.225]
distinct paths in a stack of 20 residual blocks: 2^20 = 1048576

What the code does. It trains a residual network with 20 blocks and a plain one with 4 layers, then evaluates the validation set while skipping one block at a time (continue in the loop). The result is clear-cut: the residual network goes from 0.967 to values between 0.964 and 0.969 whichever block is removed, that is no appreciable difference, while the plain network collapses from 0.961 to values between 0.03 and 0.23 when any one of its layers is removed. It is not another accuracy measurement: it says that in the residual network no single block is indispensable, because the signal has many alternative roads. A cautious interpretation: in this example the residual blocks learn small corrections (the branch is scaled by 1/√L), so each one counts for little. In real ResNets the effect exists but is less extreme. It remains a numerical proof that the shortcut changes the nature of the network, from a chain to an ensemble of paths.

The convolutional residual block

In ResNets for images the F branch is a small stack of 3×3 convolutions (with batch normalization, which we will see in the next lesson). The sum requires x and F(x) to have the same shape (N, C, H, W). If the block changes the number of channels or halves the resolution (stride 2), the shortcut can no longer be the identity: you use a projection, a 1×1 convolution with the same stride that brings x to the shape of F(x).

class ResidualBlock(nn.Module):
    def __init__(self, c_in, c_out, stride=1):
        super().__init__()
        self.branch = nn.Sequential(nn.Conv2d(c_in, c_out, 3, stride, 1), nn.ReLU(), nn.Conv2d(c_out, c_out, 3, 1, 1))
        # shortcut: identity if the shapes match, otherwise a 1x1 projection
        self.shortcut = None if (c_in == c_out and stride == 1) else nn.Conv2d(c_in, c_out, 1, stride)

    def forward(self, x):
        s = x if self.shortcut is None else self.shortcut(x)
        return torch.relu(s + self.branch(x))

# key property: with the branch "switched off" (last layer at zero) the block is the identity
block = ResidualBlock(16, 16)
for p in block.branch[2].parameters():
    nn.init.zeros_(p)
x = torch.relu(torch.randn(2, 16, 8, 8))
print("zeroed branch -> output equals input:", torch.allclose(block(x), x))

without_projection = ResidualBlock(16, 16)
with_projection = ResidualBlock(16, 32, stride=2)
print("block 16->16:  shape %s  parameters %d" % (tuple(without_projection(x).shape), sum(p.numel() for p in without_projection.parameters())))
print("block 16->32 stride 2:  shape %s  parameters %d  (with the 1x1 projection)" % (tuple(with_projection(x).shape), sum(p.numel() for p in with_projection.parameters())))

zeroed branch -> output equals input: True
block 16->16:  shape (2, 16, 8, 8)  parameters 4640
block 16->32 stride 2:  shape (2, 32, 4, 4)  parameters 14432  (with the 1x1 projection)

What the code does. ResidualBlock contains the branch (two convolutions) and a shortcut that is None (identity) when the shapes match, otherwise a 1×1 convolution with the same stride. In forward, the sum s + self.branch(x) is the line that implements y = x + F(x), followed by the ReLU. The first test demonstrates the property that makes the block easy to train: if you zero the weights of the last layer of the branch, F(x) = 0 and the block returns exactly x (for a non-negative input, such as the output of a ReLU). At initialization a block with small weights is therefore «almost identity», and the deep network starts from a reasonable function instead of from a noise that gets amplified. The second part shows the shapes: the 16→16 block keeps (2, 16, 8, 8) with 4,640 parameters; the 16→32 block with stride 2 produces (2, 32, 4, 4) and has 14,432, of which 544 belong to the 1×1 projection (32·16 + 32).

The 1×1 convolution

The projection we just used is a 1×1 convolution: a kernel of a single position, which does not look at the neighbors but combines the channels at the same pixel. For every position (i, j), it computes a linear combination of the C_in channels to produce C_out channels. It is a dense layer applied independently at every pixel, with the same weights everywhere. We verify it with a matrix multiplication (einsum is a compact notation for sums of products over indices):

conv1x1 = nn.Conv2d(8, 5, kernel_size=1)
x = torch.randn(2, 8, 4, 4)
y = conv1x1(x)
y_dense = torch.einsum("oc,nchw->nohw", conv1x1.weight[:, :, 0, 0], x) + conv1x1.bias[None, :, None, None]
print("1x1 conv = channel matrix product at every pixel:", torch.allclose(y, y_dense, atol=1e-6))
print("weight shape:", tuple(conv1x1.weight.shape), "-> one 5x8 matrix over the channels, copied at all 16 positions")

1x1 conv = channel matrix product at every pixel: True
weight shape: (5, 8, 1, 1) -> one 5x8 matrix over the channels, copied at all 16 positions

What the code does. einsum("oc,nchw->nohw") says: for each example n and position (h, w), multiply the weight matrix (o, c) by the vector of channels c and produce o. The result, plus the bias, coincides with the 1×1 convolution. What is it for? To change the number of channels cheaply: to reduce it before an expensive convolution (reduction, bottleneck) or to increase it afterwards. The saving is considerable: let us compare two 3×3 convolutions on 256 channels with the «bottleneck» structure used in deep ResNets (1×1 to go down to 64 channels, 3×3 on 64, 1×1 to return to 256), which has the same input and the same output:

two_3x3 = 2 * (256 * 256 * 3 * 3)                                 # two 3x3 convolutions 256 -> 256 (no bias)
bottleneck = 256 * 64 * 1 * 1 + 64 * 64 * 3 * 3 + 64 * 256 * 1 * 1
print("two 3x3 convolutions on 256 channels: %9d parameters" % two_3x3)
print("bottleneck block 1x1 -> 3x3 -> 1x1:   %9d parameters   (%.1f times fewer)" % (bottleneck, two_3x3 / bottleneck))

two 3x3 convolutions on 256 channels:   1179648 parameters
bottleneck block 1x1 -> 3x3 -> 1x1:       69632 parameters   (16.9 times fewer)

The calculation shows that the bottleneck has 69,632 parameters against 1,179,648: about 17 times fewer. The idea is that the 3×3 convolution, the expensive one, works on 64 channels instead of 256, and the two 1×1 convolutions take care of compressing and restoring. This is why ResNets from 50 layers up use bottleneck blocks.

The Inception block: widening instead of deepening

GoogLeNet (C. Szegedy et al., 2014, winner of ILSVRC 2014) tackles another problem: which kernel size to choose? 1×1 for pointwise features, 3×3 for medium ones, 5×5 for wide ones? Instead of choosing, the Inception block uses them all in parallel on the same input and concatenates the outputs along the channel axis; the network decides which branch to use more through the learned weights. To keep the cost sustainable, in front of the 3×3 and 5×5 convolutions it inserts 1×1 convolutions that reduce the channels.

The drawing shows the four branches: 1×1; 1×1 followed by 3×3; 1×1 followed by 5×5; 3×3 max pooling followed by 1×1. The outputs have the same spatial size (thanks to the padding) and can therefore be concatenated over the channels. We build the block with the values of GoogLeNet's «3a» module (input 192 channels; branches of 64, 96→128, 16→32 and 32 channels) and compare the version with reductions against the «naive» one that applies 3×3 and 5×5 directly to the 192 channels:

class Inception(nn.Module):
    def __init__(self, c_in, c1, c3_red, c3, c5_red, c5, c_pool):
        super().__init__()
        self.b1 = nn.Conv2d(c_in, c1, 1)
        self.b3 = nn.Sequential(nn.Conv2d(c_in, c3_red, 1), nn.ReLU(), nn.Conv2d(c3_red, c3, 3, padding=1))
        self.b5 = nn.Sequential(nn.Conv2d(c_in, c5_red, 1), nn.ReLU(), nn.Conv2d(c5_red, c5, 5, padding=2))
        self.bp = nn.Sequential(nn.MaxPool2d(3, stride=1, padding=1), nn.Conv2d(c_in, c_pool, 1))

    def forward(self, x):
        return torch.relu(torch.cat([self.b1(x), self.b3(x), self.b5(x), self.bp(x)], dim=1))     # concatenation over channels

module = Inception(192, 64, 96, 128, 16, 32, 32)
x = torch.randn(1, 192, 28, 28)
print("output shape:", tuple(module(x).shape), " (64 + 128 + 32 + 32 = 256 channels)")

def conv_params(c_in, c_out, k):
    return c_in * c_out * k * k + c_out

naive = conv_params(192, 64, 1) + conv_params(192, 128, 3) + conv_params(192, 32, 5) + conv_params(192, 32, 1)
with_reductions = sum(p.numel() for p in module.parameters())
print("parameters without 1x1 reductions: %d   with reductions: %d   (%.1f times fewer)" % (naive, with_reductions, naive / with_reductions))
print("multiplications per image on the 28x28 map: %.0f M versus %.0f M" % (28 * 28 * naive / 1e6, 28 * 28 * with_reductions / 1e6))

output shape: (1, 256, 28, 28)  (64 + 128 + 32 + 32 = 256 channels)
parameters without 1x1 reductions: 393472   with reductions: 163696   (2.4 times fewer)
multiplications per image on the 28x28 map: 308 M versus 128 M

What the code does. Inception builds the four branches, and in forward applies them to the same x and concatenates them with torch.cat(..., dim=1) (dimension 1 is the channel one): the result has 64 + 128 + 32 + 32 = 256 channels with the same 28×28 resolution. The parameter comparison: the naive version, with 3×3 and 5×5 directly on 192 channels, has 393,472; the one with 1×1 reductions has 163,696: about 2.4 times fewer. In terms of multiplications per image, 308 million against 128 million. The reductions give you breadth (several kernels of different sizes) at a contained cost.

A check against the source: table 1 of Szegedy et al. reports for module 3a about 159 thousand parameters and 128 million operations. The operations match our count; for the parameters we get 163,328 without biases and 163,696 with the biases, about 3% above the rounded figure of the table, and I cannot explain the difference. For the comparison that matters here, with and without reductions, what counts is the ratio between the two counts made with the same method, not the absolute figure.

Comparing the two ideas

• ResNet: goes in depth; the shortcut adds the input to the output; the shape must match; the blocks are homogeneous and easy to replicate.
• Inception: goes in width; the parallel branches are concatenated; the network chooses the scale of each feature; the blocks are more elaborate and rich in hyperparameters (the channels of each branch).
• In common: the 1×1 convolution as a tool to manage the number of channels and the cost, and the idea of giving the network more roads between input and output.

The criterion that holds the items together is the same question: how to give the signal more roads between input and output without letting the cost explode. ResNet answers in depth and with a sum, and for this reason imposes identical shapes; Inception answers in width and with a concatenation, and for this reason makes the channels grow and forces you to choose the width of each branch. The count of hyperparameters weighs on the choice: the residual block adds only one (the projection, when the shapes change), while the Inception module in the listing has six, one for each group of channels.

Trade-offs and real traps

• Shapes that do not match in the sum: if the branch changes the channels or the resolution and the shortcut does not, the sum gives a shape error. You need the 1×1 projection with the same stride.
• Adding or concatenating: ResNet adds (the shapes must be identical); Inception and DenseNet concatenate (the number of channels grows). Confusing them produces errors or models different from the intended ones.
• Residual without normalization: as seen, the signal can explode in deep networks; ResNets use batch normalization and careful initialization.
• Thinking that more layers are always better: the shortcut makes deep networks trainable, not necessarily better. On small data a deep network overfits like the others.
• In-place operations: x += F(x) instead of x = x + F(x) can break backpropagation, because it overwrites a value that the backward pass must read again.
• Forgetting that the shortcut has parameters when it is a projection: they must be included in the counts.
• Excessive reduction in the bottleneck: reducing the channels too much removes capacity; the 4:1 ratio (256 → 64) is a reference value, not a law.
• Concatenating branches with different resolutions: the Inception branches must keep the same spatial resolution («same» padding) before the concatenation.

Three traps (the shape in the sum, the different resolutions in the concatenation, the in-place operations) stop the program with an error message and you discover them on the first run. The other five can go through without any error and just give a model worse than expected. The most insidious is the exploding gradient: in this lesson's experiment the norm exceeded 45,000 with 50 layers without any warning, so measure the gradient norm of the first layer before launching a long training.

Frequently asked questions

Doesn't the shortcut make the block useless, if it is the identity? No: the F branch can learn any correction. The shortcut does not remove capacity; it only guarantees that, if the best correction is small, the network does not have to struggle to obtain it, and that the gradient has a direct path.

Why is a 1×1 convolution useful if it does not look at the neighbors? Because it mixes the channels, that is the features detected at each point, and it is the only low-cost tool to change the number of channels. It also adds a nonlinearity if it is followed by a ReLU.

ResNet or Inception: which one to use? For a new problem you almost always start from a pretrained ResNet (lesson on transfer learning): it is simple, robust and available in all libraries. Inception is historically important and useful to know for its ideas, but recent variants are mostly based on residual blocks.

How to read a comparison between architectures

Comparisons between architectures are fertile ground for wrong conclusions, because the models almost always differ in more than one thing at a time. Before saying «the residual network is better», ask yourself: do they have the same number of parameters? If one has twice the weights, the difference may depend on capacity and not on structure. Do they have the same training budget (epochs, optimizer, learning rate, regularization)? A hyperparameter chosen well for one and badly for the other is an unintentionally rigged comparison. Were they repeated with several seeds, reporting the variability? Differences smaller than the deviation between seeds must not be interpreted.

Was the right thing measured? In this lesson the question was «does the shortcut make a deep network easier to train?», and the right measure is the training loss and the behavior of the gradients, not only the test accuracy; a result in which the shortcut does not help, or helps little, must be reported as it is.

Finally, remember the scale of the problem: the advantages declared in the literature for networks with tens or hundreds of layers and millions of images do not transfer automatically to small problems like the 8×8 digits, where a shallow network may be enough. The goal of these experiments is not to confirm a thesis, but to show the mechanism (the direct path of the gradients, the reduction of channels with the 1×1) and to give you the method to check it on your own data.

What you can do now

You can explain why a plain network with 20 layers does not learn (accuracy 0.10, loss 2.3025 = ln 10) while the residual one does, and you can write the derivation that motivates it: the factor 1 in ∂E/∂x_l carries the gradient from the output to the first layers added and not multiplied. You also know the limit of the experiment: without normalization the gradient norm of the first layer reached over 45,000 with 50 layers, and scaling the branch by 1/√L brought it back to about 0.7. You can build a block with a 1×1 projection, count the parameters of a bottleneck (69,632 against 1,179,648, 16.9 times fewer) and of an Inception module with reductions (163,696 against 393,472). In the next lesson, dropout and batch normalization, you see the two techniques that stabilize the training of these networks; to reuse an already trained residual network, go to transfer learning.

Other lessons in the series that link to this one: Reading a CNN: feature maps, global pooling and U-Net.