68 - Highway Networks: A Gate to Decide How Much to Transform and How Much to Pass Through

A deeper network can represent more complex functions, but it is harder to train: the signal and the gradient must cross dozens of layers, and in the lessons on recurrent networks we saw what happens to a signal multiplied many times by matrices and nonlinearities. Highway Networks, proposed by Srivastava, Greff and Schmidhuber in 2015 (the LSTM group), bring into depth the same idea the LSTM applies over time: next to the transformation they put a direct path and a gate that decides, for each input, how much transformation and how much direct pass-through to mix. In this lesson you write the layer by hand in NumPy, you write its backpropagation too, you measure what happens to the gradients in networks of 10 and 50 layers with and without the gate, you train networks from 5 to 50 layers, and you discover that the decisive number in this architecture is the initial bias of the gate.

The words of this lesson

Depth is the number of stacked layers: there is no fixed threshold for calling a network "deep", and here we use the word for networks of about twenty layers and more. A plain layer computes y = ReLU(W x + b), where the ReLU zeroes out negative values. A shortcut connection (skip connection) carries the input of a layer further ahead, skipping the transformation; the identity y = x is the simplest one. The transform gate T is a vector with values in (0, 1), computed from the input: like the gates of the LSTM, it decides "how much" to let through. The per-layer gradient is the norm of the gradient of the loss with respect to the weights of that layer, and it measures how much learning signal reaches it. Degradation, finally, is the phenomenon in which a deeper network does worse than a shallower one even on the training set, because optimisation fails and not because capacity is lacking.

The analogy: a motorway with variable-toll booths

A journey of 50 stops. In a plain network every stop is a town where you are obliged to stop and be reshaped: after fifty towns you are no longer recognisable, and if you set off wrongly there is no way back. In a highway every stop has a booth with an overtaking lane: the booth T decides, based on who you are, how much of you enters the town to be transformed and how much carries on undisturbed in the through lane. At the start the toll for entering the town is high (the gate is almost closed): nearly everyone stays on the direct lane, the journey is quick and lossless, and the network learns gradually in which towns it is worth stopping.

The mathematics

A plain layer computes y = H(x), with H(x) = ReLU(W_H x + b_H). The highway layer adds a second computation, the gate T(x) = σ(W_T x + b_T) (a sigmoid, hence one value in (0, 1) per unit), and mixes:

y = T(x) ⊙ H(x) + (1 − T(x)) ⊙ x

In the original paper there are two gates, a transform gate T and a carry gate C; for simplicity the authors themselves set C = 1 − T, and that is the form used here. If T = 1 the layer is plain (y = H(x)); if T = 0 it is the identity (y = x). To mix input and output the dimensions must be equal: x, H(x) and T(x) all have the same width, which therefore stays constant along the stack (if needed, a linear layer is inserted to change it).

The gradient. Differentiating y with respect to x and, for clarity, neglecting the dependence of T on x:

∂y/∂x = diag(T) · ∂H/∂x + diag(1 − T)

The second term is the difference from the plain network. There the Jacobian is just ∂H/∂x: the product of fifty Jacobians, each typically with norm below one, vanishes. In the highway network diag(1 − T) appears, a direct path with factor 1 − T per unit, close to 1 when T is small. With 50 layers and T = 0.1 in every layer the product along the shortcut is 0.9⁵⁰ ≈ 0.005; with T = 0.02 it is 0.98⁵⁰ ≈ 0.36. For comparison, in a plain network where every Jacobian has norm 0.8, 0.8⁵⁰ ≈ 1.4·10⁻⁵. The highway network does not guarantee a constant gradient (the factor is 1 − T, not 1), but it shifts the bet: the signal has a direct channel whose width is controlled by a learned parameter, instead of depending on every layer preserving it by chance.

The drawing shows the three paths of the input x. At the bottom the gate T(x) is computed from the input (a small linear transformation followed by the sigmoid). In the middle the transformation H(x) is multiplied by T. At the top the direct path carries x, multiplied by 1 − T, up to the final sum. The two lines under the drawing give the intuitive reading: gate near 0, layer almost transparent; near 1, ordinary layer.

Code by hand: the behaviour of the gate

With NumPy and four units we see the effect of the gate bias. The bias is the number added to the gate computation before the sigmoid: if it is very negative the sigmoid gives values near 0, and the gate starts closed. Here H uses the hyperbolic tangent to keep the numbers small; in the experiments below the ReLU comes back.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

rng = np.random.default_rng(0)
d = 4
W_H, b_H = rng.normal(0, 0.5, (d, d)), np.zeros(d)         # the "transform" path H(x)
W_T = rng.normal(0, 0.5, (d, d))                            # the gate T(x)

def highway(x, b_T):
    H = np.tanh(W_H @ x + b_H)                              # candidate new representation
    T = sigmoid(W_T @ x + b_T)                              # transform gate, one value in (0,1) per unit
    y = T * H + (1 - T) * x                                 # blend: transform a bit, carry the rest
    return y, T

x = rng.normal(size=d)
for bias in (0.0, -2.0, -5.0):
    y, T = highway(x, np.full(d, bias))
    print(f"b_T={bias:5.1f}  mean gate T={T.mean():.3f}   |y - x| = {np.linalg.norm(y - x):.3f}")

b_T=  0.0  mean gate T=0.513   |y - x| = 0.360
b_T= -2.0  mean gate T=0.128   |y - x| = 0.083
b_T= -5.0  mean gate T=0.007   |y - x| = 0.005

highway(x, b_T) computes the transformation H, the gate T and the blend y = T·H + (1 − T)·x; the loop tries three biases and measures the mean value of the gate and the distance ‖y − x‖ between output and input. With bias 0 the gate is 0.51 on average and the layer changes the input a lot (distance 0.36). With −2 the gate is 0.13 and the distance falls to 0.083. With −5 it is 0.007 and the distance is 0.005: the layer is practically the identity. This is the idea of negative initialisation: a fifty-layer highway network with bias −2 or −5 starts as an almost transparent network, the signal reaches the end intact, and training can start opening the gates where needed. This is exactly what the authors propose: the gate bias initialised to a negative value (−1, −3 in their examples), with a declared inspiration from the practice of Gers and colleagues for the LSTM.

A deep network written in NumPy, with its backpropagation

The network is written in NumPy, backpropagation included; further on, the same network is rebuilt in PyTorch and compared. It is more work, but it has a teaching advantage: you see exactly where the highway layer adds a term to the gradient. The code builds two families of networks with the same width (64 units) and variable depth: plain layers, or highway layers with gate bias −2. The weights are drawn from a uniform distribution in ±1/√n_input, the rule PyTorch uses by default for nn.Linear (a known fact, not rechecked against the documentation for this work). The most delicate step is in the backward of the highway layer: the gradient towards the input has three contributions, the one from the direct path dh·(1 − t), the one that goes through the transformation and the one that goes through the gate.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def init_linear(rng, n_in, n_out):
    # same rule as torch.nn.Linear: uniform in (-1/sqrt(n_in), 1/sqrt(n_in))
    lim = 1.0 / np.sqrt(n_in)
    return rng.uniform(-lim, lim, (n_in, n_out)), rng.uniform(-lim, lim, n_out)

def make_net(kind, depth, rng, width=64, in_dim=64, classes=10, gate_bias=-2.0):
    p = {"kind": kind, "depth": depth}
    p["Wi"], p["bi"] = init_linear(rng, in_dim, width)
    p["Wo"], p["bo"] = init_linear(rng, width, classes)
    p["layers"] = []
    for _ in range(depth):
        L = {}
        L["Wh"], L["bh"] = init_linear(rng, width, width)
        if kind == "highway":
            L["Wt"], _ = init_linear(rng, width, width)
            L["bt"] = np.full(width, gate_bias)             # gate starts close to "carry"
        p["layers"].append(L)
    return p

def forward(p, X):
    cache = []
    h = np.maximum(0, X @ p["Wi"] + p["bi"]); h0 = h
    for L in p["layers"]:
        H = np.maximum(0, h @ L["Wh"] + L["bh"])
        if p["kind"] == "highway":
            t = sigmoid(h @ L["Wt"] + L["bt"])
            out = t * H + (1 - t) * h
        else:
            t, out = None, H
        cache.append((h, H, t)); h = out
    logits = h @ p["Wo"] + p["bo"]
    return logits, (X, h0, cache, h)

def loss_and_grads(p, X, y):
    logits, (X, h0, cache, hL) = forward(p, X)
    z = logits - logits.max(1, keepdims=True)
    P = np.exp(z); P /= P.sum(1, keepdims=True)
    n = len(y)
    loss = -np.log(P[np.arange(n), y]).mean()
    dl = P.copy(); dl[np.arange(n), y] -= 1; dl /= n
    g = {"Wo": hL.T @ dl, "bo": dl.sum(0), "layers": [None] * len(cache)}
    dh = dl @ p["Wo"].T
    for i in range(len(cache) - 1, -1, -1):
        h, H, t = cache[i]; L = p["layers"][i]
        if p["kind"] == "highway":
            dzh = dh * t * (H > 0)
            dzt = dh * (H - h) * t * (1 - t)
            g["layers"][i] = {"Wh": h.T @ dzh, "bh": dzh.sum(0), "Wt": h.T @ dzt, "bt": dzt.sum(0)}
            dh = dh * (1 - t) + dzh @ L["Wh"].T + dzt @ L["Wt"].T
        else:
            dzh = dh * (H > 0)
            g["layers"][i] = {"Wh": h.T @ dzh, "bh": dzh.sum(0)}
            dh = dzh @ L["Wh"].T
    dz0 = dh * (h0 > 0)
    g["Wi"], g["bi"] = X.T @ dz0, dz0.sum(0)
    return loss, g

Before trusting a hand-written backpropagation I compared it with finite differences (step 10⁻⁶) on a small 3-layer network of width 6: the maximum error on the weights Wh, Wt and on the gate bias of the second layer is of the order of 10⁻¹⁰, for both the plain and the highway network. This check is not part of the listing, but it is the guarantee that the norms measured below are true gradients and not artefacts of an error in the code.

Now the measurement: a single forward and backward pass on random data with random labels, and for each layer the norm of the gradient of the weights Wh. There is no training: we measure how much gradient reaches each layer at initialisation.

def grad_norms_per_layer(kind, depth, seed=0):
    rng = np.random.default_rng(seed)
    p = make_net(kind, depth, rng)
    X = rng.normal(size=(256, 64)); y = rng.integers(0, 10, 256)
    _, g = loss_and_grads(p, X, y)
    return [np.linalg.norm(L["Wh"]) for L in g["layers"]]   # one number per layer

for depth in (10, 50):
    for kind in ("plain", "highway"):
        g = grad_norms_per_layer(kind, depth)
        print(f"{kind:8s} depth {depth:3d}: grad norm first layer {g[0]:.2e}, last layer {g[-1]:.2e}, ratio {g[0] / g[-1]:.2e}")

plain    depth  10: grad norm first layer 2.36e-05, last layer 1.65e-02, ratio 1.43e-03
highway  depth  10: grad norm first layer 3.36e-03, last layer 4.28e-03, ratio 7.84e-01
plain    depth  50: grad norm first layer 4.28e-21, last layer 8.99e-03, ratio 4.76e-19
highway  depth  50: grad norm first layer 2.37e-05, last layer 7.17e-04, ratio 3.31e-02

In a plain network of 10 layers the gradient of the first layer is already about 700 times smaller than that of the last (ratio 1.4·10⁻³). At 50 layers it is 4·10⁻²¹ in the first layer, a ratio of 10⁻¹⁹ with the last: the first layer receives no useful information. In the highway network, at 10 layers the ratio is 0.78 (almost uniform) and at 50 layers it is 0.033: the gradient of the first layer is 2.4·10⁻⁵, about sixteen orders of magnitude above that of the plain network. It is not perfectly uniform (0.033, not 1), consistent with the factor 1 − T seen earlier, but it is a signal the optimiser can use. One seed, random data, untrained network: we measure the structure, not the final result.

Training on handwritten digits, with growing depth

Now the test that matters: really training. The MNIST file requires a download and here I do not use it: I take the small load_digits set from scikit-learn (1797 8×8 images of digits, values from 0 to 16, 10 classes), split 70/30 with stratification. It is an easier and smaller problem than MNIST, so the absolute figures are not comparable with those of MNIST; the comparison that matters, plain against highway, remains valid. I train each network for 30 epochs with Adam (learning rate 0.001, mini-batches of 64) written by hand and measure the accuracy on the test set of 540 images. The depth 100 of the previous version of this lesson was not rerun: the PyTorch version and its numbers were not reproduced here.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from b2 import make_net, forward, loss_and_grads

X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X / 16.0, y, test_size=0.3, random_state=0, stratify=y)

def adam_step(p, g, m, v, t, lr=1e-3, b1=0.9, b2=0.999, eps=1e-8):
    def upd(a, ga, ma, va):
        ma[:] = b1 * ma + (1 - b1) * ga; va[:] = b2 * va + (1 - b2) * ga * ga
        a -= lr * (ma / (1 - b1 ** t)) / (np.sqrt(va / (1 - b2 ** t)) + eps)
    for k in ("Wi", "bi", "Wo", "bo"):
        upd(p[k], g[k], m[k], v[k])
    for i, L in enumerate(p["layers"]):
        for k in L:
            upd(L[k], g["layers"][i][k], m["layers"][i][k], v["layers"][i][k])

def zeros_like(p):
    z = {k: np.zeros_like(p[k]) for k in ("Wi", "bi", "Wo", "bo")}
    z["layers"] = [{k: np.zeros_like(a) for k, a in L.items()} for L in p["layers"]]
    return z

The first block loads the data and defines Adam. adam_step updates every parameter with the moving averages of the gradient (m) and of its square (v), corrected for the bias of the first iterations. The second block contains the training loop: for each epoch it shuffles the indices, scans the mini-batches, computes loss and gradients with the backpropagation from before and takes a step.

def train_eval(kind, depth, gate_bias=-2.0, epochs=30, seed=0, batch=64):
    rng = np.random.default_rng(seed)
    p = make_net(kind, depth, rng, gate_bias=gate_bias)
    m, v, t = zeros_like(p), zeros_like(p), 0
    for _ in range(epochs):
        perm = rng.permutation(len(Xtr))
        for i in range(0, len(Xtr), batch):
            idx = perm[i:i + batch]
            loss, g = loss_and_grads(p, Xtr[idx], ytr[idx])
            t += 1; adam_step(p, g, m, v, t)
    acc = (forward(p, Xte)[0].argmax(1) == yte).mean()
    return p, acc, loss

The final loop tries the three depths, one plain network and one highway network each, with the same seed for the weights and for the order of the mini-batches. It prints the test accuracy and the loss of the last training mini-batch. Thirty epochs are my choice, not an optimised value: they are enough for the networks that do train to reach a low training loss, as the last column shows, but I did not try other durations.

for depth in (5, 20, 50):
    for kind in ("plain", "highway"):
        _, acc, last = train_eval(kind, depth)
        print(f"depth {depth:3d} {kind:8s} test accuracy {acc:.4f}   (last batch loss {last:.3f})")

depth   5 plain    test accuracy 0.9481   (last batch loss 0.012)
depth   5 highway  test accuracy 0.9630   (last batch loss 0.024)
depth  20 plain    test accuracy 0.3593   (last batch loss 1.555)
depth  20 highway  test accuracy 0.9500   (last batch loss 0.090)
depth  50 plain    test accuracy 0.1019   (last batch loss 2.303)
depth  50 highway  test accuracy 0.9426   (last batch loss 0.023)

The last column is the loss of the last mini-batch, so it is noisy: it serves only to recognise chance level (ln 10 ≈ 2.30 for ten classes). What you see, in four points:

1. At 5 layers both work (94.8% the plain one, 96.3% the highway one).
2. At 50 layers the plain network learns nothing: 10.2%, which coincides with the share of the most frequent class in the test set (measured: 0.1019), and loss 2.303, that is ln 10. At 20 layers it reaches only 35.9%: optimisation started badly and did not recover in 30 epochs.
3. The highway network works at 20 and at 50 layers (95.0% and 94.3%).
4. Depth does not improve accuracy: the 5-layer highway network gets 96.3%, the 20- and 50-layer ones a point or two less. The result says that the highway network allows deep networks to be trained, not that having them helps on this problem.

With 540 test examples a single example weighs 0.19 points, and a single seed does not say how much the result varies from one initialisation to another: "35.9% against 95.0%" is an enormous difference, "96.3% against 95.0%" is within what a change of seed could move. I reported the numbers as they are, without building a ranking on them.

The decisive number: the gate bias

We said that a negative initial bias makes the layer almost transparent. How much does it really matter? Try a 50-layer network and three initial biases, then look at the gates of a 20-layer network after training. mean_gates redoes the forward path by hand, on the test images, and records the mean value of the gate of each layer.

def mean_gates(p, X):
    """Average transform gate T of every layer, on real digits."""
    h = np.maximum(0, X @ p["Wi"] + p["bi"]); gates = []
    for L in p["layers"]:
        t = 1.0 / (1.0 + np.exp(-(h @ L["Wt"] + L["bt"])))
        gates.append(t.mean())
        H = np.maximum(0, h @ L["Wh"] + L["bh"])
        h = t * H + (1 - t) * h
    return gates

print("share of the most frequent class in the test set:", round(np.bincount(yte).max() / len(yte), 4))
for gate_bias in (0.0, -2.0, -5.0):
    _, acc, _ = train_eval("highway", 50, gate_bias=gate_bias)
    print(f"depth 50, gate bias {gate_bias:5.1f}: test accuracy {acc:.4f}")

p, acc, _ = train_eval("highway", 20, gate_bias=-2.0)
g = mean_gates(p, Xte)
print("mean transform gate per layer (depth 20, after training):", [round(float(v), 2) for v in g])

share of the most frequent class in the test set: 0.1019
depth 50, gate bias   0.0: test accuracy 0.1000
depth 50, gate bias  -2.0: test accuracy 0.9426
depth 50, gate bias  -5.0: test accuracy 0.9611
mean transform gate per layer (depth 20, after training): [0.19, 0.16, 0.17, 0.21, 0.11, 0.15, 0.12, 0.14, 0.12, 0.07, 0.1, 0.16, 0.11, 0.08, 0.14, 0.13, 0.11, 0.03, 0.06, 0.11]

The result is clear on the point that matters. With bias 0 (gate halfway at the start) the 50-layer network does not train: 10.0%, chance level, like the plain network. At every layer the transformation path weighs about 50% and the factor along the direct path is 0.5⁵⁰ ≈ 10⁻¹⁵. With −2 and −5 the network trains (94.3% and 96.1%); the difference between these two is ten examples out of 540 and with a single seed I do not count it as a result. After training a 20-layer highway network with bias −2, the mean gates lie between 0.03 and 0.21, against the starting 0.12 (σ(−2)): some opened a little, others closed, none flung wide open. The network uses its layers sparingly and works in effect like a much shallower network with small corrections distributed along the stack, which explains why depth does not help on a problem this simple.

The authors of the original paper, on other problems, report a similar pattern: strongly negative initial biases that for the most part keep decreasing during training.

Highway, residual network and where they are used today

A few months after Highway Networks, He and colleagues (2015) proposed residual networks (ResNet), in which the shortcut is a fixed identity without a gate: y = x + F(x). It is simpler (no extra gate parameters) and the direct term of the gradient is exactly 1: ∂y/∂x = I + ∂F/∂x. ResNets became the standard in vision. Highway networks found specific uses: Kim and colleagues (2016) put one on top of character-level convolutions in a language model. The choice between the two is practical: if you do not need the network to decide, as a function of the input, how much to transform, the residual one is simpler and usually enough.

Whoever works in PyTorch can translate the layer in a few lines. The following block was run (PyTorch 2.12, CPU, 2 threads, twice with the same output) and makes a real check: it builds with make_net the NumPy network of 5 highway layers, copies its weights into a stack of HighwayLayer and compares the logits on 8 random examples, in 64 bits because the NumPy weights are.

import numpy as np
import torch
import torch.nn as nn
from b2 import make_net, forward      # the NumPy network of this lesson

torch.set_num_threads(2)

class HighwayLayer(nn.Module):
    def __init__(self, dim, gate_bias=-2.0):
        super().__init__()
        self.H = nn.Linear(dim, dim)
        self.T = nn.Linear(dim, dim)
        nn.init.constant_(self.T.bias, gate_bias)   # the gate starts close to "carry"
    def forward(self, x):
        t = torch.sigmoid(self.T(x))
        return t * torch.relu(self.H(x)) + (1 - t) * x

# NumPy network: 5 highway layers, width 64. We copy ITS weights into PyTorch.
p = make_net("highway", 5, np.random.default_rng(0))
X = np.random.default_rng(1).normal(size=(8, 64))
logits_np, _ = forward(p, X)

def linear(W, b):
    # NumPy stores W as (n_in, n_out), nn.Linear as (n_out, n_in): transpose.
    m = nn.Linear(W.shape[0], W.shape[1]).double()
    with torch.no_grad():
        m.weight.copy_(torch.tensor(W.T)); m.bias.copy_(torch.tensor(b))
    return m

layers = []
for L in p["layers"]:
    layer = HighwayLayer(64).double()
    layer.H, layer.T = linear(L["Wh"], L["bh"]), linear(L["Wt"], L["bt"])
    layers.append(layer)
net = nn.Sequential(linear(p["Wi"], p["bi"]), nn.ReLU(), *layers, linear(p["Wo"], p["bo"]))
with torch.no_grad():
    logits_pt = net(torch.tensor(X)).numpy()

print("output shape:", logits_pt.shape)
print("max difference on the logits, PyTorch vs NumPy:", np.abs(logits_pt - logits_np).max())
print("first row, PyTorch:", logits_pt[0, :4].round(4))
print("first row, NumPy:  ", logits_np[0, :4].round(4))

fresh = HighwayLayer(64)
print("gate bias in a fresh layer:", fresh.T.bias[:3].tolist())
print("parameters of one layer:", sum(q.numel() for q in fresh.parameters()))

# Output:
# output shape: (8, 10)
# max difference on the logits, PyTorch vs NumPy: 2.220446049250313e-16
# first row, PyTorch: [-0.0732 -0.1108  0.114  -0.1583]
# first row, NumPy:   [-0.0732 -0.1108  0.114  -0.1583]
# gate bias in a fresh layer: [-2.0, -2.0, -2.0]
# parameters of one layer: 8320

What the output shows. The maximum difference on the logits is 2.22e-16, double-precision noise: the PyTorch layer and our forward compute the same function, and the first four outputs agree to the fourth digit. The useful part is the convention on the weights: NumPy stores them as (n_inputs, n_outputs) and computes h @ W, while nn.Linear stores them as (n_outputs, n_inputs) and computes x @ W.T, so copying them needs the transpose. Without .T, with 64×64 layers, the shapes match and PyTorch raises no error: the result is wrong, though (checked on a random matrix, the difference from x @ W was 2.5). A fresh HighwayLayer(64) starts with gate biases at −2.0 and has 8320 parameters, the same 2·(64·64 + 64) counted in the trade-offs.

Trade-offs

• Parameters. Each highway layer has about twice the parameters of a plain one: the gate matrix is added to the transformation matrix. With width 64 that is 2·(64·64 + 64) = 8320 against 4160.
• Constant width. Input and output must have the same dimension in order to be mixed.
• A hyperparameter that weighs a lot. The initial gate bias decides whether the network trains (measured: 10.0% with 0, above 94% with −2 and −5).
• Depth is not accuracy. Being able to train 50 layers does not make them useful.

Read together, the four costs say that the highway network is a bet paid for in parameters and in care over initialisation, to obtain networks that can be trained at high depth. If your problem is solved with five layers, the gate is a cost without return, and the measurement above (96.3% at 5 layers) confirms it.

Real pitfalls

1. Leaving the gate bias at zero. With 50 layers the network does not start (10.0% measured). The default bias of a linear layer is small and centred on zero, so the gate starts at about 0.5: set the bias explicitly (−1, −2 or lower).
2. Increasing depth for no reason. If the problem does not need 50 layers the gates stay almost closed and you pay twice the parameters for nothing: always compare with a shallow network.
3. Forgetting that the gate depends on the input. A fixed gate, equal for all inputs, is a weighted residual connection, not a highway network.
4. Measuring only the loss. With 10 classes a loss of 2.30 is chance level; if you see it, the network has learned nothing, and an accuracy close to the share of the most frequent class confirms it.
5. A single seed. Here every result comes from one seed: the difference between 10% and 94% is clear, the one between 94.3% and 96.1% is not.
6. Learning rate not adapted. With very deep networks a learning rate that is too high can still destabilise; here Adam at 0.001 without clipping worked up to 50 layers, but I did not try other values.

These are six different ways of reaching the same underlying error: mistaking a training result for a property of the architecture. The bias, the seed, the learning rate and the size of the problem decide how much of what you see is due to the gate.

A worked example by hand: a highway layer and the gradient over many layers

One forward step. Width 2, input x = (1.0, −0.5). Suppose the transformation gives H(x) = (2.0, 0.0) (after the ReLU) and the gate T(x) = (0.1, 0.8). The output is y = T·H + (1 − T)·x component by component: y₁ = 0.1·2.0 + 0.9·1.0 = 1.10 and y₂ = 0.8·0.0 + 0.2·(−0.5) = −0.10. The first component is almost the input (gate closed: 10% transformation), the second is mostly transformation (gate 80% open): for each component the gate chooses how much to transform.

The gradient over many layers. The factor along the direct path of a stack of L layers is (1 − T)^L. I recompute it with two loops for three gate values (barely open, 10% open, halfway) and three depths, so the table does not depend on a hand calculation that might contain an arithmetic error. The result depends only on T and L, not on data or seeds.

for T in (0.02, 0.10, 0.50):
    print(f"T = {T:.2f}  ", "   ".join(f"L={L}: {(1 - T) ** L:.3g}" for L in (10, 30, 50)))

T = 0.02   L=10: 0.817   L=30: 0.545   L=50: 0.364
T = 0.10   L=10: 0.349   L=30: 0.0424   L=50: 0.00515
T = 0.50   L=10: 0.000977   L=30: 9.31e-10   L=50: 8.88e-16

With T = 0.5 the direct path is unusable already at 10 layers (0.001): this is the reason why with bias 0 the 50-layer network does not train. With T = 0.1 at 10 layers 35% of the signal is preserved, but at 50 less than 1%. With T = 0.02 at 50 layers 36% of it is still preserved. After training, in the 20-layer network, we measured mean gates between 0.03 and 0.21: the shortcut remains the main channel.

Frequently asked questions

Why not always use a residual network? The residual y = x + F(x) has no extra parameters and a direct gradient term of exactly 1; often it is enough. The highway network makes sense if you want the network to decide, as a function of the input, how much to transform.

Which initial bias do I choose? A negative value, typically −1, −2 or −3; the authors cite −1 and −3 as examples. In our experiments −2 and −5 work and 0 does not. More negative means a more "transparent" network at the start.

Does the width have to be constant? Yes, in order to mix input and output. To change it, insert a linear projection layer between two highway blocks.

Are highway networks still used? In niches, for example on top of character-level convolutions in the language models of Kim and colleagues; for very deep networks on images residual connections are mostly used.

Does the optimiser matter? Probably a lot, but here I tried only Adam, so I do not measure the comparison with SGD. Normalisations and careful initialisations (He) also help plain networks and may shrink the advantage of the highway network: I did not measure it.

Can I use highway networks for sequences? Yes, and it is an idea related to the gates of the LSTM applied in depth instead of in time: the authors themselves declare the inspiration.

What you can do now

You can write a highway layer y = T·H(x) + (1 − T)·x and its backpropagation, and you know why the gradient has a direct path with factor 1 − T. You measured that at 50 layers the gradient of the first layer is of the order of 10⁻²¹ in the plain network and 10⁻⁵ in the highway one, and that on a small digits problem the 50-layer plain network stays at chance level while the highway one exceeds 94%. You know that the decisive detail is the initial gate bias: with 0 the 50-layer network does not start, with −2 or −5 it does. And after training the gates stay low, a reminder that depth and accuracy are different things. In the next lesson another way to train deep networks, without explicit shortcuts: fractal networks.

In the series, the previous lesson is ConvLSTM in Practice: Predicting the Next Frame of Moving Digits.