Skip to main content

02 - Differential Calculus for DL: Gradients and Backpropagation

How a network learns: chain rule, activation derivatives, backpropagation by hand and a finite-difference gradient check on an XOR network.

What do you think about this article?

At a Glance

How a network learns: chain rule, activation derivatives, backpropagation by hand and a finite-difference gradient check on an XOR network.

How a network learns: the derivative tells you which way to move each weight

A model learns by changing its parameters \theta so as to bring a loss L(\theta) down. To know how much to change each parameter you need the gradient \nabla_\theta L, that is, how much the loss varies as each one varies. The update rule is a single one, \theta \leftarrow \theta - \eta\,\nabla_\theta L with \eta the learning step, and the rest of the article is about computing that gradient without mistakes: the chain rule to break it down, backpropagation to compute it in a single backward pass, and the finite-difference check to verify it. If you come from the article on linear algebra you will find the same matrix products again, this time differentiated.

The derivative as slope, and why it can be computed numerically

The derivative of f at a point is the slope of the tangent: how much f changes for a small shift of x.

f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}

If it is positive the function rises, if it is negative it falls, if it is zero the point is stationary (minimum, maximum or saddle). Take f(x) = x^3 at x = 3: the power rule gives f'(3) = 3 \cdot 3^2 = 27. The definition also suggests an approximate computation, with h small but nonzero. With h = 0.1 the forward difference quotient is (3.1^3 - 27)/0.1 = 27.91, with error 0.91; the central quotient, (f(x+h) - f(x-h))/(2h), is 27.01, with error 0.01.

import numpy as np

f = lambda x: x ** 3
x0 = 3.0                                   # true derivative: 3 * 3**2 = 27
print(f"{'h':>8} {'forward error':>18} {'central error':>16}")
for h in [1e-1, 1e-3, 1e-5, 1e-8, 1e-11, 1e-14]:
    fwd = (f(x0 + h) - f(x0)) / h
    ctr = (f(x0 + h) - f(x0 - h)) / (2 * h)
    print(f"{h:8.0e} {abs(fwd - 27):18.2e} {abs(ctr - 27):16.2e}")

# Output:
#        h      forward error    central error
#    1e-01           9.10e-01         1.00e-02
#    1e-03           9.00e-03         1.00e-06
#    1e-05           9.00e-05         2.21e-10
#    1e-08           1.02e-07         7.53e-08
#    1e-11           8.66e-05         8.66e-05
#    1e-14           7.11e-01         7.11e-01

The table measures the order of the error. The forward error halves together with h (it is about 9h, that is, proportional to h); the central one falls with the square of h (h^2: from 10^{-2} to 10^{-6} when h goes from 10^{-1} to 10^{-3}). But at some point it stops improving and gets worse: below h \approx 10^{-8} two nearly equal numbers are subtracted, and the significant digits lost in floating-point arithmetic (about 16 for a float64) dominate the error. This is why gradient checking uses a step of the order of 10^{-5} or 10^{-6} and not the smallest possible one.

Activation derivatives and the vanishing gradient

Activation functions appear in every layer, so their derivatives appear in every backward step. For the ReLU f(x) = \max(0, x) the derivative is 1 for x > 0 and 0 for x < 0 (at x = 0 it does not exist, and by convention 0 is taken). For the sigmoid \sigma(x) = 1/(1+e^{-x}) it is \sigma'(x) = \sigma(x)(1 - \sigma(x)), for the hyperbolic tangent \tanh'(x) = 1 - \tanh^2(x). The listing verifies them against central finite differences: the maximum deviation is of the order of 10^{-11}, so the formulas are right.

import numpy as np

sigmoid = lambda x: 1 / (1 + np.exp(-x))
d_sigmoid = lambda x: sigmoid(x) * (1 - sigmoid(x))
d_tanh = lambda x: 1 - np.tanh(x) ** 2
d_relu = lambda x: (x > 0).astype(float)        # for x = 0 we choose 0

x = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])
h = 1e-6
for name, f, df in [("sigmoid", sigmoid, d_sigmoid), ("tanh", np.tanh, d_tanh),
                    ("relu", lambda t: np.maximum(0, t), d_relu)]:
    num = (f(x + h) - f(x - h)) / (2 * h)
    print(f"{name:8} formula = {np.round(df(x), 4)}  max deviation from numerical = {np.abs(df(x) - num)[x != 0].max():.1e}")

print("maximum of sigmoid'(x):", d_sigmoid(0.0))
for depth in (1, 5, 10, 20):
    print(f"{depth:2d} sigmoid layers, best case: factor {0.25 ** depth:.2e}")

# Output:
# sigmoid  formula = [0.105 0.235 0.25  0.235 0.105]  max deviation from numerical = 2.2e-11
# tanh     formula = [0.0707 0.7864 1.     0.7864 0.0707]  max deviation from numerical = 4.0e-12
# relu     formula = [0. 0. 0. 1. 1.]  max deviation from numerical = 2.9e-11
# maximum of sigmoid'(x): 0.25
#  1 sigmoid layers, best case: factor 2.50e-01
#  5 sigmoid layers, best case: factor 9.77e-04
# 10 sigmoid layers, best case: factor 9.54e-07
# 20 sigmoid layers, best case: factor 9.09e-13

The sigmoid has maximum derivative 0.25, at x = 0 (by hand: \sigma(0) = 0.5, hence 0.5 \cdot 0.5). By the chain rule, the gradient crossing n sigmoid layers is multiplied by a factor of at most 0.25^n: 0.25^5 \approx 10^{-3}, 0.25^{10} \approx 10^{-6}, and the last lines of the listing confirm it. In the best case, because with pre-activations far from zero the derivative is much smaller. This is the vanishing gradient: the layers close to the input receive an almost null signal and learn slowly. The ReLU attenuates it because for x > 0 it lets the gradient through unchanged, but it introduces another problem: a neuron whose input is always negative has zero derivative and no longer learns («dead» neuron). No activation comes without costs.

Partial derivatives, gradient and gradient descent

When the function depends on several variables, like a loss that depends on all the weights, one partial derivative per variable is computed, holding the others fixed. The vector of all the partial derivatives is the gradient:

\nabla f = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right)^T

It points in the direction of steepest increase of f, so to bring the loss down you move in the opposite direction. Take f_1(x, y) = x^2 + xy + y^2, with gradient (2x + y,\ x + 2y). At (3, 2) it equals f_1 = 9 + 6 + 4 = 19 and the gradient is (8, 7). With \eta = 0.1 one step leads to (3 - 0.8,\ 2 - 0.7) = (2.2;\ 1.3), where f_1 = 4.84 + 2.86 + 1.69 = 9.39: the function is more than halved.

Now the same thing with an apparently similar function, f_2(x, y) = x^2 + 3xy + y^2. The listing compares the two. The difference lies in the matrix of second derivatives, the Hessian, which for f_1 is \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix} and for f_2 is \begin{pmatrix} 2 & 3 \\ 3 & 2 \end{pmatrix}.

import numpy as np

def descend(grad, start, lr, steps):
    p = np.array(start, dtype=float)
    for _ in range(steps):
        p = p - lr * grad(p)
    return p

# f1 = x^2 + x*y + y^2   (Hessian [[2,1],[1,2]])      f2 = x^2 + 3*x*y + y^2   (Hessian [[2,3],[3,2]])
f1 = lambda p: p[0] ** 2 + p[0] * p[1] + p[1] ** 2
g1 = lambda p: np.array([2 * p[0] + p[1], p[0] + 2 * p[1]])
f2 = lambda p: p[0] ** 2 + 3 * p[0] * p[1] + p[1] ** 2
g2 = lambda p: np.array([2 * p[0] + 3 * p[1], 3 * p[0] + 2 * p[1]])

p0 = [3.0, 2.0]
print("f1(3,2) =", f1(p0), " gradient =", g1(np.array(p0)))
p1 = descend(g1, p0, 0.1, 1)
print("after 1 step:", np.round(p1, 4), " f1 =", round(f1(p1), 4))
p = descend(g1, p0, 0.1, 100)
print("after 100 steps:", np.round(p, 6), " f1 =", f"{f1(p):.2e}")

print("Hessian eigenvalues f1:", np.linalg.eigvalsh([[2, 1], [1, 2]]))
print("Hessian eigenvalues f2:", np.linalg.eigvalsh([[2, 3], [3, 2]]))
p = descend(g2, p0, 0.1, 30)
print("f2 after 30 steps:", np.round(p, 2), " f2 =", round(f2(p), 1))
print("f2 along (1, -1) at distance t = 5:", f2(np.array([5.0, -5.0])))

# Output:
# f1(3,2) = 19.0  gradient = [8. 7.]
# after 1 step: [2.2 1.3]  f1 = 9.39
# after 100 steps: [ 1.3e-05 -1.3e-05]  f1 = 1.76e-10
# Hessian eigenvalues f1: [1. 3.]
# Hessian eigenvalues f2: [-1.  5.]
# f2 after 30 steps: [ 8.72 -8.72]  f2 = -76.1
# f2 along (1, -1) at distance t = 5: -25.0

The first step reproduces the hand calculation, and in a hundred steps f_1 reaches 10^{-10} near the minimum at (0, 0). The second function has no minimum. The eigenvalues of its Hessian are 2 \pm 3 = 5 and -1: positive in one direction, negative in the other, the sign of a saddle point. Along (1, -1) it is f_2(t, -t) = t^2 - 3t^2 + t^2 = -t^2, which falls without limit, and gradient descent follows it: after 30 steps f_2 = -76.1 and it keeps going. The practical rule is that the sign of the Hessian eigenvalues classifies a critical point: all positive, minimum; all negative, maximum; mixed, saddle. In deep networks saddle points are very frequent: Dauphin and colleagues (2014) argue that, in high-dimensional spaces, the main difficulty is the proliferation of saddle points and not of local minima, and that around them there are plateaus that slow learning down.

The chain rule on one neuron, with numbers

If y = f(g(x)) then \dfrac{dy}{dx} = f'(g(x))\,g'(x): the sensitivity of a composition is the product of the sensitivities of its parts. A network is a composition of many parts, and the loss depends on each weight through a chain. For a neuron with z = wx + b, \hat{y} = \sigma(z) and L = (y - \hat{y})^2 the chain has three links:

\frac{\partial L}{\partial w} = \underbrace{2(\hat{y} - y)}_{\text{error}} \cdot \underbrace{\sigma'(z)}_{\text{activation sensitivity}} \cdot \underbrace{x}_{\text{input}}

with \partial L/\partial b the same but with 1 in place of x. Numbers: x = 2, y = 1, w = 0.5, b = 0.1. Then z = 1.1, \hat{y} = \sigma(1.1) \approx 0.7503 and L \approx 0.0624. The three links are 2(0.7503 - 1) \approx -0.4995, \sigma'(z) = 0.7503 \cdot 0.2497 \approx 0.1874 and x = 2, so \partial L/\partial w \approx -0.1872 and \partial L/\partial b \approx -0.0936. The negative sign says that increasing w and b lowers the loss.

import numpy as np

sigmoid = lambda t: 1 / (1 + np.exp(-t))
x, y, w, b = 2.0, 1.0, 0.5, 0.1

def loss(w, b):
    return (y - sigmoid(w * x + b)) ** 2

z = w * x + b
y_hat = sigmoid(z)
dL_dyhat = 2 * (y_hat - y)               # derivative of the squared error
dyhat_dz = y_hat * (1 - y_hat)           # derivative of the sigmoid
dz_dw, dz_db = x, 1.0
dL_dw = dL_dyhat * dyhat_dz * dz_dw
dL_db = dL_dyhat * dyhat_dz * dz_db
print(f"z = {z:.4f}  y_hat = {y_hat:.4f}  loss = {loss(w, b):.5f}")
print(f"dL/dy_hat = {dL_dyhat:.4f}  dy_hat/dz = {dyhat_dz:.5f}  dL/dw = {dL_dw:.5f}  dL/db = {dL_db:.5f}")

e = 1e-6
print(f"finite differences: dL/dw = {(loss(w + e, b) - loss(w - e, b)) / (2 * e):.5f}  dL/db = {(loss(w, b + e) - loss(w, b - e)) / (2 * e):.5f}")

lr = 0.5
for epoch in range(2001):
    y_hat = sigmoid(w * x + b)
    g = 2 * (y_hat - y) * y_hat * (1 - y_hat)
    w, b = w - lr * g * x, b - lr * g
    if epoch in (0, 10, 100, 500, 2000):
        print(f"epoch {epoch:4d}: loss = {loss(w, b):.5f}  w = {w:.3f}  b = {b:.3f}")

# Output:
# z = 1.1000  y_hat = 0.7503  loss = 0.06237
# dL/dy_hat = -0.4995  dy_hat/dz = 0.18737  dL/dw = -0.18717  dL/db = -0.09359
# finite differences: dL/dw = -0.18717  dL/db = -0.09359
# epoch    0: loss = 0.04347  w = 0.594  b = 0.147
# epoch   10: loss = 0.00969  w = 0.946  b = 0.323
# epoch  100: loss = 0.00108  w = 1.413  b = 0.556
# epoch  500: loss = 0.00021  w = 1.748  b = 0.724
# epoch 2000: loss = 0.00005  w = 2.033  b = 0.866

The block computes the three links separately (dL_dyhat, dyhat_dz, dz_dw), multiplies them and compares the result with the finite differences: the two lines match the hand calculation. The loop that follows applies gradient descent, and the epoch lines print the loss after the update. Note that the loss falls fast at first and then ever more slowly: as \hat{y} approaches 1 the sigmoid saturates, \sigma'(z) tends to zero and the gradient of this loss fades out just when it would still be needed. Combining sigmoid and squared error is a choice that slows learning down; the article on loss functions shows how cross-entropy removes the factor \sigma'.

The computational graph: why the backward pass is cheap

Computing the gradient by writing out the complete formula of a network by hand is impractical: the text by Deisenroth and colleagues notes that the explicit expression can become very long and more expensive to evaluate than the function itself. A computational graph breaks the function down into elementary operations; in the forward pass each node computes its own value, and in the backward pass each node receives the gradient of the loss with respect to its own output, multiplies it by its own local derivative and passes it on to the nodes it depends on. Each node is visited once per direction, so the cost of the gradient is of the same order as that of computing the loss, whatever the number of parameters. This is backpropagation (Rumelhart, Hinton and Williams, 1986) and it is what PyTorch and TensorFlow automate with automatic differentiation.

For a vector layer two new matrices appear. The Jacobian of \mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m is the m \times n matrix with element J_{ij} = \partial f_i / \partial x_j: it generalizes the gradient to functions with several outputs, and it is the «local coefficient» of the backward step. The Hessian, already met, collects the second derivatives. In the code that follows the Jacobian is never built: we just multiply the downstream gradient by its transpose, and this is why the formulas contain transposes of weight matrices.

Backpropagation on a two-layer network, with gradient checking

Let us put everything together on XOR, the classic problem that no linear model solves: two inputs, output 1 only if they differ. The network has 2 inputs, 4 sigmoid hidden neurons and a sigmoid output, that is, 17 parameters. With the loss L = \frac{1}{N}\sum (a_2 - y)^2 the backward pass, in matrix form, follows four steps, in the order in which the gradient flows:

  1. \delta_2 = \dfrac{2}{N}(a_2 - y) \odot a_2(1 - a_2): the output error multiplied by the derivative of the sigmoid, element by element
  2. \nabla_{\mathbf{W}_2} = a_1^T \delta_2 and \nabla_{\mathbf{b}_2} = \sum \delta_2: the gradients of the output layer
  3. \delta_1 = (\delta_2 \mathbf{W}_2^T) \odot a_1(1 - a_1): the error carried back through the weights, then multiplied by the derivative of the first layer
  4. \nabla_{\mathbf{W}_1} = \mathbf{X}^T \delta_1 and \nabla_{\mathbf{b}_1} = \sum \delta_1: the gradients of the hidden layer

The order cannot be changed, because \delta_1 is computed from \delta_2 through \mathbf{W}_2^T: it is the chain rule in matrix form, and the transpose appears because the error travels from the outputs toward the inputs. The product \odot is element by element, like NumPy's *, while \delta_2 \mathbf{W}_2^T is a true matrix product.

import numpy as np

sigmoid = lambda t: 1 / (1 + np.exp(-t))
X = np.array([[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]])
y = np.array([[0.0], [1.0], [1.0], [0.0]])             # XOR

rng = np.random.default_rng(1)
params = {"W1": rng.normal(0, 1, (2, 4)), "b1": np.zeros((1, 4)),
          "W2": rng.normal(0, 1, (4, 1)), "b2": np.zeros((1, 1))}

def forward(p):
    a1 = sigmoid(X @ p["W1"] + p["b1"])
    a2 = sigmoid(a1 @ p["W2"] + p["b2"])
    return a1, a2

def loss(p):
    return np.mean((forward(p)[1] - y) ** 2)

def gradients(p):
    a1, a2 = forward(p)
    dz2 = 2 * (a2 - y) / len(X) * a2 * (1 - a2)         # (4,1)
    dz1 = (dz2 @ p["W2"].T) * a1 * (1 - a1)             # (4,4)
    return {"W2": a1.T @ dz2, "b2": dz2.sum(0, keepdims=True),
            "W1": X.T @ dz1, "b1": dz1.sum(0, keepdims=True)}

# 1) check: backpropagation against central finite differences, on every parameter
g = gradients(params)
eps, worst = 1e-6, 0.0
for name, arr in params.items():
    for idx in np.ndindex(arr.shape):
        old = arr[idx]
        arr[idx] = old + eps; lp = loss(params)
        arr[idx] = old - eps; lm = loss(params)
        arr[idx] = old
        num = (lp - lm) / (2 * eps)
        rel = abs(g[name][idx] - num) / max(abs(g[name][idx]) + abs(num), 1e-12)
        worst = max(worst, rel)
print(f"parameters checked: {sum(a.size for a in params.values())}, maximum relative error: {worst:.1e}")

# 2) training
lr = 2.0
for epoch in range(20001):
    g = gradients(params)
    for k in params:
        params[k] = params[k] - lr * g[k]
    if epoch in (0, 1000, 5000, 20000):
        print(f"epoch {epoch:5d}: loss = {loss(params):.5f}")
print("predictions:", np.round(forward(params)[1].ravel(), 3), " target:", y.ravel())

# Output:
# parameters checked: 17, maximum relative error: 1.8e-08
# epoch     0: loss = 0.25626
# epoch  1000: loss = 0.00231
# epoch  5000: loss = 0.00025
# epoch 20000: loss = 0.00005
# predictions: [0.005 0.992 0.994 0.009]  target: [0. 1. 1. 0.]

The block does two things, in this order. First it verifies the backpropagation: for each of the 17 parameters it computes the derivative with central finite differences (\varepsilon = 10^{-6}) and compares it with the analytic one, with relative error |g_a - g_n| / (|g_a| + |g_n|). The maximum found is 1.8 \cdot 10^{-8}. There is no universal threshold: what matters is the order of magnitude, and 10^{-8} is very far from the values a wrong gradient produces, so the gradients are right. Trying two wrong implementations on the same network, the forgotten factor 2 in \delta_2 gives a maximum relative error of 0.33, and the weights \mathbf{W}_2 ignored in the computation of \delta_1 give 0.94. Then it trains: the loss falls from 0.256 to 5 \cdot 10^{-5} and the predictions are almost (0, 1, 1, 0).

Three warnings. The finite-difference check costs two evaluations of the loss per parameter: it should be done once, on a small network, not during training. The seed fixes the initial weights: trying seeds from 0 to 7 with the same code, the loss after 20000 epochs lies between 4 \cdot 10^{-5} and 7 \cdot 10^{-5}, so here training is not fragile, but the values at a given epoch change from one seed to another. Finally, the squared loss plus the step \eta = 2 are choices that work on this toy problem, not a recipe: the article on optimization explains how the step is chosen.

What you can do now

You can compute by hand the derivative of a loss with respect to the weights of a neuron by multiplying three factors, write in matrix form the backward pass of a two-layer network and verify it with finite differences before trusting the training. You can also read the symptoms: a gradient that vanishes through a stack of sigmoids, a descent that does not converge because the point is a saddle and not a minimum, a gradient check that is off by orders of magnitude. To understand how the direction of the gradient is then turned into a step, the next stop is the article on probability, which gives the loss its statistical meaning.

How I checked the listings: I ran each one on its own on 2026-11-21 with Python 3.14.4, NumPy 2.3.5, and the decimal figures quoted in the text match the ones produced. With other library versions the last digits may change.

Sources

Try These Tools

Free developer tools related to this topic.

Commenti

Caricamento commenti...

Sign in to comment

New articles and tools on Telegram too

The Telegram channel flags the site's new articles and tools.

Did you like this article?

Explore other content on the blog or discover my projects