69 - FractalNet: Deep Networks Without Explicit Shortcuts, With Drop-Path

In the previous lesson we saw a first solution to the problem of very deep networks: placing next to every layer a gate and a direct path. Fractal networks (FractalNet), proposed by Larsson, Maire and Shakhnarovich in 2016, take another road: no identity shortcut and no gate, but a recursive construction rule that produces, inside a single block, many parallel paths of different depth, from the shortest (a single transformation) to the longest (dozens). The signal always has at least one short path to get where it is needed, and a regularisation method called drop-path forces every path to work on its own as well. In this lesson you build the recursive block in NumPy, with the backpropagation written by hand, you count the paths and you measure something surprising: the individual "columns" of a fractal network, extracted after training, can be complete and accurate networks, but only if the network was trained with drop-path.

The words of this lesson

A column is a path from the input to the output of the fractal block: in a block with C columns, column j contains 2^(j−1) units (1, 2, 4, 8…). The join is the operation that combines the paths that rejoin, and here it is the element-wise mean of the incoming tensors. Drop-path randomly switches off whole paths during training, as dropout does with single neurons: in the local version each join randomly discards some of its inputs while leaving at least one, in the global one a single column is chosen for the whole batch. Batch Normalization normalises the activations with the mean and standard deviation of the current batch during training, and in evaluation uses a moving average (running_mean, running_var) accumulated during training. An "anytime" prediction is an approximate answer right away, with a short path, and a better one later, with a long one. In the code the basic unit is "linear without bias, BatchNorm, ReLU"; in the original paper it is a convolution in place of the linear layer, and the recursive structure is identical.

The analogy: a road network with many routes

Imagine a city where to go from a point A to a point B there are many routes: a direct street of a single block, a route crossing two neighbourhoods, one crossing four, and so on, built with a rule that repeats at every scale. The traffic (the signal) splits among the routes and recombines on arrival. To keep the traffic from always depending on the same ring road, every now and then a road is closed at random: drivers must learn to work with any subset. In the end every route, even alone, gets you to the destination.

The expansion rule

The fractal block f_C is defined by recursion. The base case is a single unit. The recursive step takes two copies of the previous block in series and places them next to a single unit, rejoining the two paths with a mean:

f_1(x) = unit(x)
f_{C}(x) = join( f_{C−1}( f_{C−1}(x) ),  unit(x) )

For C = 2 there are two paths: one of a single unit and one of two in series, rejoined by the mean. For C = 3 we start over: a path with a single unit and a path made of two copies in series of the block f(2), each with its own two paths and its own mean. In the drawing the units are labelled "conv", as in the original paper. The number of units and the length of the longest path follow two formulas that we derive by hand and then check with the code: the units N(C) = 2·N(C−1) + 1 (two copies plus one), hence N(C) = 2^C − 1; the longest path L(C) = 2·L(C−1), hence L(C) = 2^(C−1).

def conv_count(C):
    """Units in f_C:  f_1 = 1 unit;  f_{C+1} = f_C(f_C(x)) joined with 1 unit."""
    return 1 if C == 1 else 2 * conv_count(C - 1) + 1

def longest_path(C):
    return 1 if C == 1 else 2 * longest_path(C - 1)

for C in range(1, 6):
    print(f"C={C}: units {conv_count(C):3d} (= 2^C - 1 = {2**C - 1:3d}),  longest path {longest_path(C):2d} (= 2^(C-1) = {2**(C-1):2d}),  columns {C}")

C=1: units   1 (= 2^C - 1 =   1),  longest path  1 (= 2^(C-1) =  1),  columns 1
C=2: units   3 (= 2^C - 1 =   3),  longest path  2 (= 2^(C-1) =  2),  columns 2
C=3: units   7 (= 2^C - 1 =   7),  longest path  4 (= 2^(C-1) =  4),  columns 3
C=4: units  15 (= 2^C - 1 =  15),  longest path  8 (= 2^(C-1) =  8),  columns 4
C=5: units  31 (= 2^C - 1 =  31),  longest path 16 (= 2^(C-1) = 16),  columns 5

The code implements the two recurrences with two recursive functions and compares them with the closed formulas. With C = 5 the block has 31 units, but the longest path crosses only 16 of them and the shortest 1. The parameter that matters is not "how many layers do I have", it is "how long is the longest path".

The fractal block in NumPy, with backpropagation

PyTorch is not available on the machine on which I ran the listings, and downloading MNIST requires the network: the network below is therefore written in NumPy, with dense units in place of convolutions, and trained on the 1797 8×8 digits of scikit-learn's load_digits. It is a smaller and easier test bench than that of the previous version of this lesson, and the absolute numbers must not be compared with those of MNIST. The structure that matters, recursion, mean join and drop-path, is the same. The unit contains the most delicate piece, backpropagation through the BatchNorm: the formula for dz is the standard one, in which the gradient of each sample also depends on the statistics of the whole batch.

import numpy as np

class Unit:
    """Linear (no bias) + BatchNorm + ReLU, with its own backward pass."""
    def __init__(self, rng, n_in, n_out, eps=1e-5, momentum=0.1):
        lim = 1.0 / np.sqrt(n_in)
        self.W = rng.uniform(-lim, lim, (n_in, n_out))
        self.g, self.b = np.ones(n_out), np.zeros(n_out)
        self.rm, self.rv = np.zeros(n_out), np.ones(n_out)       # running mean / variance, used in eval
        self.eps, self.momentum = eps, momentum
        self.n_seen = 0

    def forward(self, x, train):
        z = x @ self.W
        if train:
            mu, var = z.mean(0), z.var(0)
            if self.momentum is None:                            # plain average of all the batches seen
                self.n_seen += 1; k = 1.0 / self.n_seen
            else:
                k = self.momentum
            self.rm = (1 - k) * self.rm + k * mu
            self.rv = (1 - k) * self.rv + k * var
        else:
            mu, var = self.rm, self.rv
        self.inv = 1.0 / np.sqrt(var + self.eps)
        self.xhat = (z - mu) * self.inv
        y = self.g * self.xhat + self.b
        self.x, self.mask = x, y > 0
        return np.maximum(y, 0)

    def backward(self, dout):
        dy = dout * self.mask
        self.dg, self.db = (dy * self.xhat).sum(0), dy.sum(0)
        dxh = dy * self.g
        n = len(dy)
        dz = self.inv / n * (n * dxh - dxh.sum(0) - self.xhat * (dxh * self.xhat).sum(0))
        self.dW = self.x.T @ dz
        return dz @ self.W.T

    def params(self):
        return [(self, "W", "dW"), (self, "g", "dg"), (self, "b", "db")]

    def reset_stats(self):
        self.rm, self.rv, self.n_seen, self.momentum = np.zeros_like(self.rm), np.ones_like(self.rv), 0, None

forward computes z = x·W, then in training the batch mean and variance, which it also uses to update the moving averages with weight momentum (0.1); in evaluation it uses the moving averages. reset_stats is needed further on: it zeroes the moving averages and switches to momentum = None, that is, to the plain arithmetic mean of all the batches seen, the convention that PyTorch calls by the same name. I checked backward with finite differences on a small network (3 columns, width 5, all parameters): the maximum difference between analytic and numerical gradient is 1.3·10⁻⁷, and the check is not in the listing.

The Fractal class translates the recursion almost word for word. It always contains the "short" unit and, if C > 1, two sub-blocks of level C − 1. The forward has three modes: with a numeric col it runs a single column (column 1 is the short unit, column j > 1 is column j − 1 of two copies in series, and this is what gives 2^(j−1) units); with col=None it computes the deep path and the short one and rejoins them with the mean; in training, with local_drop > 0, each path is discarded with that probability, with the rule that at least one survives. The discarded paths are computed too, as in the original code, so they update the BatchNorm statistics even though they do not contribute to the output. The backward divides the gradient by the number of paths kept and sends it only to those.

class Fractal:
    """f_1 = unit ;  f_C = join( f_{C-1}( f_{C-1}(x) ),  unit(x) ).  Column j has 2^(j-1) units."""
    def __init__(self, rng, C, n_in, n_out):
        self.C = C
        self.shallow = Unit(rng, n_in, n_out)                    # the 1-unit path (column 1)
        if C > 1:
            self.a = Fractal(rng, C - 1, n_in, n_out)
            self.b = Fractal(rng, C - 1, n_out, n_out)

    def forward(self, x, train, rng, col=None, local_drop=0.0):
        if col is not None:                                      # run ONE column only
            if col == 1:
                self.mode = "s"; return self.shallow.forward(x, train)
            self.mode = "c"
            return self.b.forward(self.a.forward(x, train, rng, col - 1), train, rng, col - 1)
        if self.C == 1:
            self.mode = "s"; return self.shallow.forward(x, train)
        self.mode = "full"
        deep = self.b.forward(self.a.forward(x, train, rng, None, local_drop), train, rng, None, local_drop)
        paths = [deep, self.shallow.forward(x, train)]
        self.keep = [True, True]
        if train and local_drop > 0:                             # LOCAL drop-path: drop each input of the join
            self.keep = [bool(rng.random() > local_drop) for _ in paths]
            if not any(self.keep):                               # never drop everything
                self.keep[int(rng.integers(len(paths)))] = True
        return np.mean([p for p, k in zip(paths, self.keep) if k], axis=0)   # the join: mean of the survivors

    def backward(self, dout):
        if self.mode == "s":
            return self.shallow.backward(dout)
        if self.mode == "c":
            return self.a.backward(self.b.backward(dout))
        d = dout / sum(self.keep)
        dx = 0
        if self.keep[0]:
            dx = dx + self.a.backward(self.b.backward(d))
        if self.keep[1]:
            dx = dx + self.shallow.backward(d)
        return dx

    def units(self):
        return [self.shallow] + (self.a.units() + self.b.units() if self.C > 1 else [])

FractalNet stacks two blocks with a linear classifier on top. The global drop-path chooses, with probability global_drop, a single column for the whole batch, and the network works as if it were that column. A network like this has 18,634 parameters (counted by the program).

class FractalNet:
    def __init__(self, rng, C=3, width=32, in_dim=64, classes=10):
        self.C = C
        self.blocks = [Fractal(rng, C, in_dim, width), Fractal(rng, C, width, width)]
        lim = 1.0 / np.sqrt(width)
        self.Wo, self.bo = rng.uniform(-lim, lim, (width, classes)), rng.uniform(-lim, lim, classes)

    def forward(self, x, train=False, rng=None, col=None, local_drop=0.0, global_drop=0.0):
        if train and col is None and rng.random() < global_drop:
            col = int(rng.integers(1, self.C + 1))               # GLOBAL drop-path: one single column for this batch
        for blk in self.blocks:
            x = blk.forward(x, train, rng, col, local_drop)
        self.h = x
        return x @ self.Wo + self.bo

    def loss_and_backward(self, logits, y):
        z = logits - logits.max(1, keepdims=True)
        P = np.exp(z); P /= P.sum(1, keepdims=True)
        n = len(y); loss = -np.log(P[np.arange(n), y]).mean()
        dl = P; dl[np.arange(n), y] -= 1; dl /= n
        self.dWo, self.dbo = self.h.T @ dl, dl.sum(0)
        d = dl @ self.Wo.T
        for blk in reversed(self.blocks):
            d = blk.backward(d)
        return loss

    def params(self):
        ps = [(self, "Wo", "dWo"), (self, "bo", "dbo")]
        for blk in self.blocks:
            for u in blk.units():
                ps += u.params()
        return ps

    def units(self):
        return [u for blk in self.blocks for u in blk.units()]

    def n_params(self):
        return sum(getattr(o, n).size for o, n, _ in self.params())

print(FractalNet(np.random.default_rng(0)).n_params(), "parameters")

18634 parameters

The experiment: what is a column worth on its own?

We train the same network in two ways: without drop-path and with drop-path (local 0.15 and global 0.5, that is, half of the batches on a single column, like the 50% and 50% mixture of the original paper). For 40 epochs, Adam with learning rate 0.002 and mini-batches of 64, written by hand. Then we measure the accuracy of the complete network and of each column taken alone: 1 = one unit per block, 3 = four units per block. A parameter that is not on the active path receives no gradient, and as in PyTorch it is not updated.

import copy
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X / 16.0, y, test_size=0.3, random_state=0, stratify=y)

def accuracy(net, col=None):
    return (net.forward(Xte, train=False, col=col).argmax(1) == yte).mean()

def train(local_drop, global_drop, epochs=40, seed=0, only_col=None, lr=2e-3, batch=64):
    rng = np.random.default_rng(seed)
    net = FractalNet(rng)
    ps = net.params()
    m = [np.zeros_like(getattr(o, n)) for o, n, _ in ps]
    v = [np.zeros_like(getattr(o, n)) for o, n, _ in ps]
    t = [0] * len(ps)
    for _ in range(epochs):
        perm = rng.permutation(len(Xtr))
        for i in range(0, len(Xtr), batch):
            idx = perm[i:i + batch]
            for o, _, gn in ps:
                setattr(o, gn, None)                             # a parameter off the active path gets no gradient
            logits = net.forward(Xtr[idx], train=True, rng=rng, col=only_col, local_drop=local_drop, global_drop=global_drop)
            net.loss_and_backward(logits, ytr[idx])
            for k, (o, n, gn) in enumerate(ps):                  # Adam, one parameter array at a time
                g = getattr(o, gn)
                if g is None:
                    continue                                     # same as PyTorch: no gradient, no update
                t[k] += 1
                m[k] = 0.9 * m[k] + 0.1 * g; v[k] = 0.999 * v[k] + 0.001 * g * g
                getattr(o, n)[...] -= lr * (m[k] / (1 - 0.9 ** t[k])) / (np.sqrt(v[k] / (1 - 0.999 ** t[k])) + 1e-8)
    return net

The first block loads the data, splits it 70/30 with stratification, and defines the accuracy measure and the training loop with Adam. The loop clears the gradients at every step, because a parameter outside the active path must stay without a gradient, and it skips those parameters in the update. The next block uses it on two configurations, without and with drop-path, and prints the complete network and the single columns.

for name, ld, gd in [("no drop-path", 0.0, 0.0), ("drop-path (local 0.15 + global 0.5)", 0.15, 0.5)]:
    net = train(ld, gd)
    cols = {j: round(float(accuracy(net, col=j)), 4) for j in (1, 2, 3)}
    print(f"{name:38s} full net {accuracy(net):.4f}   single columns {cols}")

no drop-path                           full net 0.9889   single columns {1: 0.8611, 2: 0.4722, 3: 0.4741}
drop-path (local 0.15 + global 0.5)    full net 0.9833   single columns {1: 0.9722, 2: 0.9704, 3: 0.9481}

accuracy(net, col) evaluates in evaluation mode on the test set (540 images), choosing a column with col=j or the whole network with col=None. Without drop-path the complete network gets 98.9%, but columns 2 and 3 taken alone are worth 47.2% and 47.4%, while column 1 reaches 86.1%: the deep columns are "co-adapted" and do not work without the others. With drop-path the complete network gets 98.3%, half a point less, and the columns become usable: 97.2%, 97.0% and 94.8%. Drop-path does not make the complete network better on this problem; it makes each path autonomous, which is what it was meant to achieve. One seed and 540 test examples: one example is worth 0.19 points, so differences of a point or less between the complete network with and without drop-path are not a result.

BatchNorm statistics: how much they really weigh

In evaluation the BatchNorm uses the moving average of the statistics computed in training. With drop-path the flow crosses a variable mix of paths, and the accumulated statistics are those of that mix; evaluating a column alone, the data follow a different path, with activations of a different scale. The proposed cure is to recalibrate the statistics for that column: zero them and have them recomputed as an average over a few training batches passed only through the chosen column, without updating the weights. Recalibration is a widespread technique, but it is not in the original fractal-network paper (searched in the PDF text: no occurrence of "recalib" or similar): I treat it here as a check of mine, not as a result of the authors.

import copy
import numpy as np

def accuracy_after_bn_recalibration(net, col, n_batches=15, batch=64):
    """Evaluate ONE column after re-estimating the BatchNorm statistics for that column alone."""
    net = copy.deepcopy(net)
    for u in net.units():
        u.reset_stats()                                          # momentum None: plain average of the batches seen
    rng = np.random.default_rng(0)
    perm = rng.permutation(len(Xtr))[:n_batches * batch]
    for i in range(0, len(perm), batch):
        net.forward(Xtr[perm[i:i + batch]], train=True, rng=rng, col=col)    # forward only: no weight changes
    return accuracy(net, col=col)

The function just defined is applied to both networks, the one without and the one with drop-path, repeating the training with the same seed. For each it prints three lines: the columns with the original statistics, the columns recalibrated one by one, and the complete network before and after. The last line trains each column alone, for the comparison in the next section.

for name, ld, gd in [("no drop-path", 0.0, 0.0), ("drop-path", 0.15, 0.5)]:
    net = train(ld, gd)
    print(f"{name:13s} columns as is        :", {j: round(float(accuracy(net, col=j)), 4) for j in (1, 2, 3)})
    print(f"{name:13s} BN recalibrated/col. :", {j: round(float(accuracy_after_bn_recalibration(net, j)), 4) for j in (1, 2, 3)})
    print(f"{name:13s} full net as is / recalibrated:", round(float(accuracy(net)), 4), round(float(accuracy_after_bn_recalibration(net, None)), 4))
print("trained ALONE:", {j: round(float(accuracy(train(0.0, 0.0, only_col=j), col=j)), 4) for j in (1, 2, 3)})

no drop-path  columns as is        : {1: 0.8611, 2: 0.4722, 3: 0.4741}
no drop-path  BN recalibrated/col. : {1: 0.8852, 2: 0.4907, 3: 0.5407}
no drop-path  full net as is / recalibrated: 0.9889 0.9926
drop-path     columns as is        : {1: 0.9722, 2: 0.9704, 3: 0.9481}
drop-path     BN recalibrated/col. : {1: 0.9778, 2: 0.9722, 3: 0.9519}
drop-path     full net as is / recalibrated: 0.9833 0.9815

The function copies the network, zeroes the statistics, runs 15 batches of 64 forward through the chosen column (almost the whole training set of 1257 images) and measures the accuracy. No weight is touched. The result is less spectacular than I expected: with drop-path the recalibration takes the columns from 97.2/97.0/94.8% to 97.8/97.2/95.2%, a gain of 0.2 to 0.6 points, that is, from one to three test examples out of 540. Without drop-path columns 2 and 3 rise by 2 and 7 points but stay below 55%: the flaw was not in the statistics, it was in the weights. The main effect of this lesson is therefore due to drop-path, not to recalibration. Recalibration weighs more when the network is trained little, as the last block of this article shows.

Comparison with columns trained alone

To know whether the gain is real we compare with each column trained alone, as an independent network with the same recipe (40 epochs, same seed). The last line of the previous block already reports the result, obtained with train(0.0, 0.0, only_col=j): at every batch the network runs only column j. In summary:

                                       col 1    col 2    col 3    full net
no drop-path                           0.861    0.472    0.474    0.989
drop-path, statistics as is            0.972    0.970    0.948    0.983
drop-path, BN recalibrated             0.978    0.972    0.952    0.982
trained alone (reference)              0.982    0.980    0.980     --

What the table says, bearing in mind that a single seed and 540 examples cannot distinguish differences of half a point:

• The columns extracted from a network with drop-path are 0.4-3 points from the same columns trained alone (97.2 against 98.2 for column 1, 97.0 against 98.0 for column 2, 94.8 against 98.0 for column 3). A single training produced three usable networks, at a cost of a few points on the deepest one.
• Without drop-path column 1 works because it is the unit that receives direct signal, the other two do not: extraction without drop-path is not practicable.
• The complete network does not significantly beat column 3 alone (98.2 against 98.0): on this small problem the mean of the paths adds no accuracy.

This scales down the story of the previous version, in which a recalibrated column matched the one trained alone after an enormous recovery, from 29% to 88%. On these figures and with this network the recovery thanks to recalibration is small and the extracted deep columns remain a little below those trained alone. I cannot say whether the difference depends on the convolutions, on the dataset or on the length of training, because the PyTorch version was not rerun.

When recalibration weighs: short training

I repeat the experiment with drop-path, but with 3 and 10 epochs instead of 40, to see whether recalibration matters more when the network is still far from convergence.

for epochs in (3, 10):
    net = train(0.15, 0.5, epochs=epochs)
    print(epochs, "epochs, drop-path: as is   :", {j: round(float(accuracy(net, col=j)), 4) for j in (1, 2, 3)})
    print(epochs, "epochs, drop-path: recalib.:", {j: round(float(accuracy_after_bn_recalibration(net, j)), 4) for j in (1, 2, 3)})

3 epochs, drop-path: as is   : {1: 0.8481, 2: 0.65, 3: 0.5796}
3 epochs, drop-path: recalib.: {1: 0.85, 2: 0.6944, 3: 0.6167}
10 epochs, drop-path: as is   : {1: 0.95, 2: 0.9333, 3: 0.8944}
10 epochs, drop-path: recalib.: {1: 0.9556, 2: 0.9426, 3: 0.9111}

With 3 epochs columns 2 and 3 go from 65.0% and 58.0% to 69.4% and 61.7% (plus 4.4 and plus 3.7 points), with 10 epochs the gains are 0.6, 0.9 and 1.7 points, with 40 epochs 0.6, 0.2 and 0.4. The trend is consistent with intuition, namely the shorter the training the further the stored statistics are from the right ones, but it remains a trend observed on one seed. In all cases recalibration costs 15 forward passes per column: I would always apply it, even if here it does not change the conclusion.

What the "anytime" prediction gains

With 2 blocks, column 1 has 2 units in all (one per block), column 2 has 4, column 3 has 8. With the network trained for 40 epochs and drop-path, the accuracy of the recalibrated column is 97.8% with 2 units, 97.2% with 4 and 95.2% with 8: in our test bench the deepest column is not the best, because the problem is simple and the short path is enough. The "anytime" compromise remains valid as a mechanism (one model, three different cost points), but here there is no accuracy gain to justify it; I saw it work only as a mechanism. The authors apply drop-path to much deeper networks on CIFAR and ImageNet, which I did not reproduce.

Fractal versus highway versus residual

The highway network mixes transformation and identity with a learned gate; the gradient has a direct path with factor 1 − T. The residual uses a fixed identity, y = x + F(x), and is the simplest. The fractal uses no identity: the short path is a real transformation and drop-path imposes that every path works on its own. The authors' idea is that the network does not need residuals to be deep, it is enough that short paths exist to carry the signal and the gradient; the abstract declares results on a par with residual networks on CIFAR and ImageNet, benchmarks that I do not reproduce and that are not comparable with our mini-experiment. One observation that the experiment supports: if an independent short path exists, the network can become deep without the gradient having to cross all the units.

Trade-offs

The cost grows quickly: a block with C columns has 2^C − 1 units, 31 with C = 5. Drop-path generally requires more epochs to converge (in our tests the columns at 3 epochs are between 58% and 85%, at 40 between 95% and 97%; here one duration at a time). Extracting subnetworks is free in theory, but it requires recalibrating the BatchNorm or a normalisation that does not depend on the batch. And the short path is not an identity: it does not preserve the signal, it transforms it, and its usefulness lies in being short.

Real pitfalls

1. Evaluating the columns of a network trained without drop-path. Here columns 2 and 3 are worth 47% against the 98.9% of the complete network. One concludes, rightly, that the long path is not autonomous, but it is not an evaluation flaw.
2. Not recalibrating the BatchNorm after short training. At 3 epochs the difference reaches 4 points on the deep columns.
3. Using drop-path in evaluation. It is to be used only in training; in evaluation you use the whole network or a chosen column.
4. Discarding all the paths in a join. The code imposes that at least one survives: without this rule the mean would be over zero elements.
5. Fixing a single generator. Drop-path uses a random generator in addition to the initialisation one; to reproduce a result both must be controlled (here a single default_rng with a seed).
6. Comparing networks with different budgets. A fractal network has more parameters than a column alone; the fair comparison depends on what you want to compare, inference cost or training cost.

The six pitfalls share the same root: an evaluation number depends on how you trained and how you evaluated, not only on the model. Before saying that drop-path "does not work" or "works", check the training duration, the evaluation mode and the normalisation statistics.

A worked example by hand: the probability of local drop-path

In a join with two inputs and discard probability p = 0.15 for each, we compute the probabilities of the four outcomes (independent discard):

both kept:        0.85 × 0.85 = 0.7225
only first kept:  0.85 × 0.15 = 0.1275
only second kept: 0.15 × 0.85 = 0.1275
none kept:        0.15 × 0.15 = 0.0225   → not allowed

The last case is corrected by randomly choosing one of the two inputs to keep (as in the code, with rng.integers): each receives half of 0.0225, that is 0.01125. The final probabilities are therefore 72.25% for two paths, 13.875% for only the first and 13.875% for only the second. In about one join in four (27.75%) the network works with a single path, and that is where each path is forced to work on its own. With more joins in series and nested, the probability that the whole network works on a specific path multiplies: hence the effect of "training many sub-networks together". Global drop-path is more brutal: with probability 0.5 a single column is chosen for the whole batch, so one batch in two is trained on one column, about 17% each of the three. With p = 0.3 the same formulas give 0.49 for both kept and 0.21 + 0.045 = 0.255 for only the first.

Frequently asked questions

Why the mean as the join and not the sum? The mean keeps the scale of the activations constant whatever the number of surviving paths: with the sum, switching off a path would change the order of magnitude. The original paper uses the mean for the join.

Is drop-path like dropout? Same idea, different scale: dropout switches off single neurons, drop-path whole paths. Dropout reduces co-adaptation between neurons, drop-path between paths, and that is what is needed to extract the columns.

Is it useful on an easy problem like these digits? As a test bench yes, because the columns can be seen clearly; but the complete network is no better than the deep column, and the gain of FractalNets shows, according to the authors, on much bigger datasets and networks.

Can I use LayerNorm or GroupNorm instead of BatchNorm? A normalisation that does not depend on the batch statistics has no moving averages to recalibrate. I did not try it: it is a consequence of how it works, not a measurement.

What you can do now

You can build a fractal network with the rule f_C = join(f_{C−1} ∘ f_{C−1}, unit), you know that it has 2^C − 1 units distributed over columns of depth 1, 2, 4…, and you can write by hand the mean join, the local and global drop-path and the backpropagation through the BatchNorm. You measured that without drop-path the extracted deep columns are worth 47%, with drop-path 95-97%, a few points from those trained alone (98%), and that BatchNorm recalibration adds from 0.2 to 4 points depending on how short the training is. In the next lesson we change register: from networks that classify to networks that compress and reconstruct, with autoencoders.

In the series, the previous lesson is Highway Networks: A Gate to Decide How Much to Transform and How Much to Pass Through.