55 - Transfer learning: reusing a pretrained CNN

You have an image classification problem and, instead of millions of labelled examples, you have a few hundred, perhaps a few dozen. Training a convolutional network from scratch on so little data gives unstable results: the network memorises the few examples and gets new ones wrong. Transfer learning is the standard answer: you start from a network already trained on a large task, keep the part that extracts features, and adapt it to your task. The official PyTorch tutorial describes it in just this way: a network trained on a very large dataset (the example is ImageNet, 1.2 million images in 1000 categories) used either as an initialisation or as a fixed feature extractor.

Before you trust the technique, though, it is worth seeing when it works and when it does not, because transfer is not guaranteed. This lesson closes the module with three measured experiments, in which you will see transfer help a lot, help a little, and even make the result worse.

Prerequisites: you know how to train a small CNN with PyTorch (the LeNet lesson), batch normalization and its behaviour in train() and eval() (the lesson on dropout and batch normalization), and the idea that the intermediate layers of a CNN produce feature maps you can start from (the lesson on feature maps; for the residual blocks of modern backbones, the lesson on residual networks).

One choice has to be declared right away. The experiments in this lesson use neither ImageNet nor a ResNet: they use the 8×8 images of the scikit-learn Digits dataset, which are already on your machine, and a small backbone (13,152 parameters) that we pretrain ourselves. The advantage is that you can rerun everything in a few minutes on a CPU, without downloading data or weights. The price is that the numbers cannot be carried over as they are to your real problem: what carries over is the method, which is to always compare against a from-scratch baseline and over several seeds. Further on there is a sketch, not executed here, for doing the same thing with a torchvision ResNet18. All the listings were executed with Python 3.14, PyTorch 2.12.0 (CPU, 2 threads), scikit-learn 1.9.0 and NumPy 2.4.6, twice in a row with identical output.

The vocabulary

• Pretrained model: a network already trained on a starting task (the source task), with its weights available for reuse.
• Backbone: the part of the network that extracts features, everything that comes before the final classification layer.
• Head: the final layer that turns the features into the predictions of your task. You replace it with a new one, because the classes of your problem are different.
• Freezing a layer: preventing its weights from changing during training, with requires_grad = False.
• Feature extraction: using the frozen backbone as a fixed calculator and training only the new head.
• Fine-tuning: continuing to train the backbone weights as well, with a small learning rate, starting from the pretrained ones.
• Domain: the kind of data (natural photographs, X-rays, handwritten digits). The domain gap between source and target is what decides whether transfer helps.

The last three entries describe a choice, not an object: extraction and fine-tuning are the two strategies we will compare, and the domain is the variable that, as you will see in experiment 2, decides which of the two pays off. The first four entries are instead parts of the machine, and their order does not matter: you can read them in any sequence.

Why it works: features are (partly) general

The early layers of a CNN learn simple filters (oriented edges, colour blobs) and the later ones combine them into ever more specific structures. The first are general: they are useful for almost any image. The last are specific to the task the network was trained on. J. Yosinski and colleagues measured exactly this, layer by layer: their abstract reports three results. Transferability is hindered by two distinct factors: the specialisation of the higher-layer neurons to the original task, and the optimisation difficulties that arise when a network is split between neurons that were co-adapted. It also decreases as the distance between the source task and the target task grows. Finally, even from a distant task you get better features than random ones. Hence the two classic strategies, summarised in the drawing.

In feature extraction the backbone stays frozen (in grey): its outputs are fixed features and only the new head is trained, with very few parameters and therefore little risk of overfitting. In fine-tuning the whole network is updated, but starting from weights that are already good and with a low learning rate (typically 10 or 100 times smaller than usual, a rule of thumb and not a theorem), so as not to destroy what the network already knows. In between sits partial fine-tuning: you freeze the early layers and train only the last blocks and the head.

The test bench: 8×8 digits in three parts

The scikit-learn Digits dataset has 1,797 8×8 images of handwritten digits, 64 integer values from 0 to 16 per image (the docstring of load_digits says so). To do transfer learning you need a source task, a target task and a test set that touches neither. So we split the images into three disjoint parts: 900 to train the backbone (source), 400 from which to draw the small training subsets of the target (the pool), 497 for the test. The normalisation uses the mean and standard deviation of the source alone, as you would do with a pretrained model that carries its own input statistics with it.

The network is a small CNN with two convolutional layers, each followed by BatchNorm2d: the batch normalization is there on purpose, because it is what makes freezing subtle (the trap is in the next section). The Rete class explicitly separates caratteristiche (the backbone) from testa (the head), so replacing the head or freezing the backbone are operations on an attribute. The addestra function has one parameter that matters: backbone_eval=True puts the backbone back in eval() mode after every train(); we will see why shortly. The specchia parameter of a_tensore is only needed by experiment 2.

import copy
import numpy as np
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

torch.set_num_threads(2)

X, y = load_digits(return_X_y=True)                                              # 1797 8x8 images, digits 0-9
X_sorg, X_resto, y_sorg, y_resto = train_test_split(X, y, train_size=900, stratify=y, random_state=0)
X_pool, X_te, y_pool, y_te = train_test_split(X_resto, y_resto, train_size=400, stratify=y_resto, random_state=0)
media, dev = X_sorg.mean(), X_sorg.std()                                         # statistics of the SOURCE only
print("source: %s   target pool: %s   test: %s" % (X_sorg.shape, X_pool.shape, X_te.shape))

def a_tensore(A, specchia=False):
    img = A.reshape(-1, 8, 8)
    if specchia:
        img = img[:, :, ::-1]                                                    # left-right reflection
    return torch.tensor(((img - media) / dev).reshape(-1, 1, 8, 8), dtype=torch.float32)

class Rete(nn.Module):
    def __init__(self, n_classi):
        super().__init__()
        self.caratteristiche = nn.Sequential(                                    # the "backbone"
            nn.Conv2d(1, 16, 3, padding=1), nn.BatchNorm2d(16), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(), nn.MaxPool2d(2),
            nn.Flatten(), nn.Linear(128, 64), nn.ReLU())
        self.testa = nn.Linear(64, n_classi)                                     # the "head"
    def forward(self, x):
        return self.testa(self.caratteristiche(x))

def addestra(rete, X, Y, parametri, epoche=60, lr=3e-3, seme=0, backbone_eval=False):
    torch.manual_seed(seme)
    ott = torch.optim.Adam(parametri, lr=lr)
    loader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(X, Y), batch_size=16, shuffle=True)
    for _ in range(epoche):
        rete.train()
        if backbone_eval:
            rete.caratteristiche.eval()                                          # batch norm statistics frozen
        for xb, yb in loader:
            ott.zero_grad(); nn.CrossEntropyLoss()(rete(xb), yb).backward(); ott.step()

def accuratezza(rete, X, Y):
    rete.eval()
    with torch.no_grad():
        return (rete(X).argmax(1) == Y).float().mean().item()

source: (900, 64)   target pool: (400, 64)   test: (497, 64)

The three parts have the declared sizes: 900, 400 and 497 images. Now we train the backbone on the source task, which is the richest this dataset allows: recognising all ten digits. Forty epochs are enough, because the images are tiny; the seed is fixed, so the source network is the same on every run and the comparisons in the next experiments always start from the same backbone.

torch.manual_seed(0)
base = Rete(10)                                                                  # source task: all ten digits
addestra(base, a_tensore(X_sorg), torch.tensor(y_sorg), base.parameters(), epoche=40)
print("accuracy of the source network on the test set: %.3f" % accuratezza(base, a_tensore(X_te), torch.tensor(y_te)))
print("backbone parameters: %d   10-class head: %d   3-class head: %d" % (
    sum(p.numel() for p in base.caratteristiche.parameters()),
    sum(p.numel() for p in base.testa.parameters()), 64 * 3 + 3))

accuracy of the source network on the test set: 0.986
backbone parameters: 13152   10-class head: 650   3-class head: 195

The source network scores 0.986 on the test set. The backbone has 13,152 parameters and a 10-class head adds 650; the three-class head we will use in the target task has 195. What matters for what follows is the ratio: with feature extraction you train 195 parameters against the backbone's 13,152.

Experiment 1: pretrained features against training from scratch

The target task has to be different from the source task, otherwise it is not transfer. We use an arbitrary grouping of the same images: the label is the remainder of the digit divided by 3 (0, 3, 6, 9 form group 0; 1, 4, 7 group 1; 2, 5, 8 group 2). It is an artificial choice, I admit, but it has a useful property: the three groups do not correspond to a simple visual shape, so a network starting from scratch with few examples struggles, while a backbone that has already learned to tell the ten digits apart has in effect the right information, recombined in another way. The comparison is between two alternatives on the same examples: a network trained from scratch, and a logistic regression on the 64 features produced by the frozen backbone (eval() mode, no gradient). The number of examples per group ranges from 2 to 50, with five seeds.

def caratteristiche(rete, A):
    rete.eval()
    with torch.no_grad():
        return rete.caratteristiche(a_tensore(A)).numpy()                        # 64 numbers per image

def campione(etichette, n, seme):                                                # n examples per class, chosen at random
    r = np.random.default_rng(seme)
    return np.concatenate([r.choice(np.where(etichette == k)[0], n, replace=False) for k in np.unique(etichette)])

g_pool, g_te = y_pool % 3, y_te % 3                                              # target task: the digit modulo 3
F_pool, F_te = caratteristiche(base, X_pool), caratteristiche(base, X_te)
print("images per group in the pool:", np.bincount(g_pool))

print("examples per group   from scratch (CNN)        pretrained features + logistic regression")
for n in (2, 5, 10, 25, 50):
    da_zero, preaddestrato = [], []
    for seme in range(5):
        idx = campione(g_pool, n, seme)
        torch.manual_seed(seme)
        z = Rete(3)
        addestra(z, a_tensore(X_pool[idx]), torch.tensor(g_pool[idx]), z.parameters(), seme=seme)
        da_zero.append(accuratezza(z, a_tensore(X_te), torch.tensor(g_te)))
        preaddestrato.append(LogisticRegression(max_iter=3000).fit(F_pool[idx], g_pool[idx]).score(F_te, g_te))
    print("%20d   %.3f +- %.3f        %.3f +- %.3f" % (n, np.mean(da_zero), np.std(da_zero), np.mean(preaddestrato), np.std(preaddestrato)))

images per group in the pool: [160 120 120]
examples per group   from scratch (CNN)        pretrained features + logistic regression
                   2   0.619 +- 0.079        0.659 +- 0.080
                   5   0.694 +- 0.079        0.850 +- 0.076
                  10   0.780 +- 0.057        0.933 +- 0.027
                  25   0.885 +- 0.017        0.963 +- 0.009
                  50   0.936 +- 0.015        0.971 +- 0.008

Read the table by columns first and then by differences. With 2 examples per group (six images in all) the two strategies are indistinguishable: 0.619 against 0.659, with a spread between seeds of 0.08. The advantage appears with 5 examples per group (0.694 against 0.850) and stays large with 10 (0.780 against 0.933), then shrinks to eight points with 25 and to three and a half points with 50 (0.936 against 0.971). Two lessons. The first: the gain thins out as the data grows, because with enough examples a network from scratch learns suitable features on its own. The second, which is more than you would expect: the gain is not largest with the minimum number of examples. With six images in all, not even a linear classifier on excellent features has enough information, and the noise of the sample dominates. Transfer does not replace data, it makes it more effective.

The mechanics of freezing, and a subtle trap

For the next experiments we need a complete network with the new head. Freezing means setting requires_grad = False on the parameters: PyTorch no longer computes their gradients and the optimiser no longer updates them. There is, however, a trap specific to backbones with batch normalization: freezing the parameters does not freeze the running statistics (running_mean and running_var), which are buffers and not parameters. If the network is in train() mode, the module updates them at every batch with a momentum of 0.1 (the default value in the source of torch/nn/modules/batchnorm.py), and after a few iterations they reflect your data and no longer the starting data. Let us check.

rete = copy.deepcopy(base)
rete.testa = nn.Linear(64, 3)                                                    # NEW head, trained from scratch
for p in rete.parameters():
    p.requires_grad = False                                                      # freeze everything...
for p in rete.testa.parameters():
    p.requires_grad = True                                                       # ...except the new head

addestrabili = sum(p.numel() for p in rete.parameters() if p.requires_grad)
totali = sum(p.numel() for p in rete.parameters())
print("trainable parameters: %d of %d  (%.2f%%)" % (addestrabili, totali, 100 * addestrabili / totali))

batch = a_tensore(X_pool[:64])
bn = rete.caratteristiche[1]                                                     # the first BatchNorm2d layer
prima = bn.running_mean.clone()
rete.train(); rete(batch)                                                        # frozen weights, but train mode
dopo_train = bn.running_mean.clone()
rete.eval(); rete(batch)
dopo_eval = bn.running_mean.clone()
print("batch norm statistics change in train mode with frozen weights:", not torch.equal(prima, dopo_train))
print("statistics change in eval mode:", not torch.equal(dopo_train, dopo_eval))

trainable parameters: 195 of 13347  (1.46%)
batch norm statistics change in train mode with frozen weights: True
statistics change in eval mode: False

The first value, 195 trainable parameters out of 13,347, is 1.46%: the part of the network that really learns. Then the trap shows up: after a single pass in train() mode the running mean of the first batch normalization layer has changed despite the frozen weights, while in eval() it stays still. The cure, when you do feature extraction, is to keep the backbone in eval() for the whole training of the head. That is what the backbone_eval=True parameter of addestra does: remember that every call to rete.train() puts the submodules back in train() too, so the backbone has to be put back in eval() afterwards, at every epoch.

Experiment 2: head only or the whole network? It depends on the domain

The same backbone, two strategies: head only (backbone frozen and in eval(), learning rate 3·10⁻³ as for all the from-scratch trainings) or the whole network with learning rate 10⁻³, starting from the pretrained weights. We add a variable: how close the target is to the source. In the near case the images are the ones from before, the same digits the same way round. In the far case the target images (training and test) are mirrored left-right, and the backbone has never seen a mirrored digit. It is an artificial but controlled distance: the same kind of strokes, a different orientation, as happens on a small scale when you move from photographs to X-rays. The task is always the remainder modulo 3, and there is always the «from scratch» column as a term of comparison.

print("domain   examples per group   from scratch     head only        whole network    (5 seeds)")
for nome, specchia in (("near", False), ("far", True)):
    X_te_d = a_tensore(X_te, specchia)
    for n in (5, 25):
        righe = []
        for seme in range(5):
            idx = campione(g_pool, n, seme)
            X_n, g_n = a_tensore(X_pool[idx], specchia), torch.tensor(g_pool[idx])
            torch.manual_seed(seme)
            z = Rete(3)
            addestra(z, X_n, g_n, z.parameters(), seme=seme)
            torch.manual_seed(seme)
            solo_testa = copy.deepcopy(base); solo_testa.testa = nn.Linear(64, 3)
            for p in solo_testa.caratteristiche.parameters():
                p.requires_grad = False
            addestra(solo_testa, X_n, g_n, solo_testa.testa.parameters(), seme=seme, backbone_eval=True)
            torch.manual_seed(seme)
            tutta = copy.deepcopy(base); tutta.testa = nn.Linear(64, 3)
            addestra(tutta, X_n, g_n, tutta.parameters(), lr=1e-3, seme=seme)
            righe.append([accuratezza(m, X_te_d, torch.tensor(g_te)) for m in (z, solo_testa, tutta)])
        r = np.array(righe)
        print("%-8s %17d   %.3f +- %.3f   %.3f +- %.3f   %.3f +- %.3f" % ((nome, n) + tuple(v for k in range(3) for v in (r[:, k].mean(), r[:, k].std()))))

domain   examples per group   from scratch     head only        whole network    (5 seeds)
near                     5   0.694 +- 0.079   0.823 +- 0.063   0.762 +- 0.080
near                    25   0.885 +- 0.017   0.957 +- 0.014   0.948 +- 0.020
far                      5   0.699 +- 0.086   0.662 +- 0.071   0.697 +- 0.062
far                     25   0.889 +- 0.024   0.788 +- 0.015   0.884 +- 0.027

In the near domain the head alone is the best strategy: 0.823 against 0.762 for the whole network with 5 examples per group (with spreads between seeds of 0.06-0.08, the margin is modest), 0.957 against 0.948 with 25, where the difference is within the noise. In the far domain the picture flips: the head alone falls below the from-scratch network (0.788 against 0.889 with 25 examples: ten points, with spreads of 0.015-0.024, so reliable) and fine-tuning, which updates the filters, recovers the level of the from-scratch network (0.884 against 0.889) but does not beat it. The frozen filters look at the digits «the right way round», and no linear head can correct a backbone that extracts the wrong features.

This result does not confirm the rule that is often repeated («little data, freeze; more data, unfreeze»): here neither strategy wins on the amount of data alone. What decides is the domain distance. In practice: try feature extraction first (it costs a few seconds), and if it stays below a from-scratch baseline the domain is too far to freeze. Fine-tuning, for its part, beats the from-scratch baseline in the near domain (0.762 against 0.694 and 0.948 against 0.885) but does not beat the head alone, and equals the baseline in the far domain. Why updating the filters adds nothing to the head alone in the near case is a question that remains open here: one hypothesis, not verified, is that a backbone already good for the ten digits has not much more to learn, and that the 13 thousand parameters updated on a few dozen images mostly add noise.

Experiment 3: when transfer does not help

In experiment 1 the backbone had been trained on a task richer than the target (ten digits against three groups). Here we reverse the situation: the source task is poorer and the target is different. The backbone is trained on the digits 0-4 alone (five classes) and then reused to classify the digits 5-9, with very few examples. The two tasks are close (both are handwritten digits), and we would expect transfer to help.

sorgente = y_sorg < 5                                                            # source task: digits 0-4
torch.manual_seed(0)
base5 = Rete(5)
addestra(base5, a_tensore(X_sorg[sorgente]), torch.tensor(y_sorg[sorgente]), base5.parameters(), epoche=40)
bers_pool, bers_te = y_pool >= 5, y_te >= 5                                      # target task: digits 5-9
X_b, y_b = X_pool[bers_pool], y_pool[bers_pool] - 5
X_b_te, y_b_te = a_tensore(X_te[bers_te]), torch.tensor(y_te[bers_te] - 5)
print("source network (digits 0-4) on the test set of digits 0-4: %.3f" % accuratezza(base5, a_tensore(X_te[y_te < 5]), torch.tensor(y_te[y_te < 5])))
print("images per digit in the target pool:", np.bincount(y_b))

print("examples per class   from scratch     head only        whole network    (5 seeds)")
for n in (5, 25):
    righe = []
    for seme in range(5):
        idx = campione(y_b, n, seme)
        X_n, y_n = a_tensore(X_b[idx]), torch.tensor(y_b[idx])
        torch.manual_seed(seme)
        z = Rete(5)
        addestra(z, X_n, y_n, z.parameters(), seme=seme)
        torch.manual_seed(seme)
        solo_testa = copy.deepcopy(base5); solo_testa.testa = nn.Linear(64, 5)
        for p in solo_testa.caratteristiche.parameters():
            p.requires_grad = False
        addestra(solo_testa, X_n, y_n, solo_testa.testa.parameters(), seme=seme, backbone_eval=True)
        torch.manual_seed(seme)
        tutta = copy.deepcopy(base5); tutta.testa = nn.Linear(64, 5)
        addestra(tutta, X_n, y_n, tutta.parameters(), lr=1e-3, seme=seme)
        righe.append([accuratezza(m, X_b_te, y_b_te) for m in (z, solo_testa, tutta)])
    r = np.array(righe)
    print("%20d   %.3f +- %.3f   %.3f +- %.3f   %.3f +- %.3f" % ((n,) + tuple(v for k in range(3) for v in (r[:, k].mean(), r[:, k].std()))))

source network (digits 0-4) on the test set of digits 0-4: 1.000
images per digit in the target pool: [41 40 40 39 40]
examples per class   from scratch     head only        whole network    (5 seeds)
                   5   0.924 +- 0.024   0.606 +- 0.032   0.906 +- 0.027
                  25   0.988 +- 0.005   0.833 +- 0.009   0.976 +- 0.004

The expectation is refuted. The source network is perfect on its own task (1.000 on the test set of the digits 0-4), but on the target the from-scratch network scores 0.924 with 5 examples per class and 0.988 with 25; fine-tuning scores 0.906 and 0.976, a little worse; the frozen features 0.606 and 0.833, clearly worse. The pretraining did not help and, when frozen, did harm. There are two reasons. The features learned to separate five digits are very specific to those five: the network has learned to notice the strokes that tell them apart and to ignore the others, which are precisely the ones useful for the other five. And the target task is already easy: with 5 examples per class a network from scratch reaches 92%, so there is little room.

Reading the three experiments together

The three results complete one another. In experiment 1 the backbone had been trained on a task richer than the target, and its features beat training from scratch: by fifteen points with 5 and 10 examples per group, by three and a half with 50. In experiment 2, with the same backbone, the choice between freezing and unfreezing followed the domain distance: near, the head alone is better; far, the head alone does harm and fine-tuning barely reaches the baseline. In experiment 3 the pretraining on a poor task (five classes) produced features that were too specific, harmful when frozen. If I had to reduce it all to one sentence: transfer brings knowledge along with it, and knowledge is useful only if it is general enough and close to your problem. One last precaution on the numbers: with five seeds and a test set of 497 images, differences of less than a couple of points should not be told as victories. The correct way to report them is «mean ± deviation over N seeds, against a from-scratch baseline».

From 8×8 digits to a ResNet18: the sketch, not executed

On real images you do not build the pretrained backbone: you download it. The following sketch follows the official PyTorch tutorial on transfer learning and was not executed here, because torchvision is not installed in this environment and downloading the weights was not allowed: it is a sketch to adapt, not a result. It shows the three operations you have already done by hand in the experiments: replacing the head, freezing the backbone, moving on to fine-tuning.

# SKETCH TO ADAPT, NOT EXECUTED in this lesson (it needs torchvision and the download of the weights)
import torch.nn as nn
from torchvision import models

n_classi = 10                                                       # the classes of YOUR problem
rete = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)     # weights pretrained on ImageNet
rete.fc = nn.Linear(rete.fc.in_features, n_classi)                  # new head, trained from scratch

for p in rete.parameters():
    p.requires_grad = False                                         # freeze everything...
for p in rete.fc.parameters():
    p.requires_grad = True                                          # ...except the new head

# while training the head: rete.eval() for the batch norm statistics,
# then rete.train() and requires_grad = True everywhere, with lr ~10 times lower, for fine-tuning.
# Preprocessing: the same resizing and normalisation as the weights (see ResNet18_Weights.DEFAULT.transforms()).

The two lines not to forget are the ones about preprocessing. Pretrained weights expect images prepared as they were in the original dataset (size, channel order, normalisation mean and deviation): using another scale degrades the features without any error message. In the experiments of this lesson we respected the same principle by using the source normalisation on the target as well.

A practical recipe

1. Start from a backbone pretrained on a task richer and more varied than yours, and check the licence of the weights and of the original dataset.
2. Respect the original preprocessing: sizes, channel order, mean and standard deviation.
3. Replace the head with one suited to your classes.
4. Begin with feature extraction (backbone frozen, in eval()): it is cheap and serves as a baseline.
5. Move on to fine-tuning if the head alone stays below a from-scratch baseline (a sign of a far domain), with a learning rate lower than usual, possibly unfreezing the layers from the top down.
6. Always compare against a from-scratch baseline and over several seeds: transfer is not guaranteed.
7. Use augmentation and early stopping on the validation set, and report the gain with the variability across seeds.

The order matters here: step 4 comes before step 5 because feature extraction is cheap and is the baseline that tells you whether anything more is needed, and step 6 closes the circle by reporting what you found. Before release to production remember too that the model inherits the limits and the biases of the original data: if the destination domain is very different, evaluate the difficult subgroups separately instead of trusting a single aggregate metric, and note in the documentation which checkpoint you used.

The table summarises what experiments 1 and 2 measured and should be read as a rule of thumb: they are starting points to validate on your problem, not guaranteed results.

Real traps

• Wrong normalisation: using the statistics of your dataset in place of those of the pretrained model, or forgetting to scale the pixels.
• Batch normalization and freezing: frozen weights but the backbone in train(), with the statistics changing (shown above).
• Learning rate too high in fine-tuning: it can destroy the pretrained features in the first iterations (so-called catastrophic forgetting).
• Random head and gradients: at the start an untrained head produces noisy gradients that propagate into the unfrozen backbone; that is why you train the head alone first.
• Forgetting to replace the head: the model keeps producing the classes of the source task.
• Data leakage between source and target: if the test images are present in the pretraining dataset, the performance is inflated. In the experiments here the three sets are disjoint by construction.
• Treating transfer as guaranteed: in experiments 2 (far domain) and 3 it did not beat the baseline.
• Weights with a restrictive licence and inherited biases: the pretrained model carries the limits of its original data with it.

The first two traps are silent: neither produces an error, they produce slightly worse numbers and leave you convinced the model is just like that. The others are caught with the model's documentation and with the comparison against a baseline. The order does not change the meaning of the list: it is a reminder, not a sequence.

Frequently asked questions

Can you do transfer learning with non-image data? Yes: in natural language it is the standard (large models are pretrained on generic text and adapted to specific tasks); for audio and tabular data it is less common. The principle is identical.

How many layers should you unfreeze? You start from the head alone, then progressively unfreeze blocks from the top down until the validation improves. It is rarely worth modifying the early layers.

Why use a linear classifier on the features? It is fast, has few parameters and makes an honest baseline. If it does not work, the problem lies in the features (a domain too different), not in the head: in experiment 2, in the far domain, not even a well-trained head recovers.

Three exercises

Exercise 1. In experiment 1 reduce the source task to 200 images (train_size=200 in the first split, adapting the others) and see how much of the advantage survives: it is the amount of pretraining that counts, besides its quality.

Exercise 2. Unfreeze only the second convolutional layer and the head (rete.caratteristiche[4] and the following) and compare it with «head only» and «whole network» in the far domain at 25 examples per group.

Exercise 3. In experiment 3 swap the roles (source: digits 5-9, target: 0-4) and check whether the qualitative result repeats.

What you can do now

You can build a backbone with a replaceable head, freeze it without falling into the batch normalization trap (195 trainable parameters out of 13,347, and the backbone in eval()), and compare feature extraction and fine-tuning against a from-scratch baseline, over several seeds. You can also recognise the cases in which transfer does not help: a source poorer than the target, a far domain. With this lesson the module closes: from linear regression to neurons, to backpropagation, to convolutional networks and to the reuse of already trained networks, you have built the pieces by hand and measured them. The next step of the course starts from here.

Next lesson: Recurrent Neural Networks (RNN): Giving the Network a Memory.