21 - Calculus 1: Limits and Derivatives

Machine Learning Engineer course · Module 02 · Mathematics and statistics · Lesson 6

Goal. Understand what a derivative is (the slope of a curve at a point, obtained as a limit), know how to compute it from the definition and with the rules (which we will prove), apply it to the functions you meet in neural networks (sigmoid, hyperbolic tangent, ReLU), and know the limits of the numerical computation of a derivative, which are the limits of the floating-point numbers seen in module 01. At the end you will know why "where the slope is zero" is the starting point of all optimisation.

Myth to dispel. "Calculus is a list of formulas to memorise." The formulas are few and all descend from a single idea: approaching a value indefinitely. If you understand the idea, you rebuild the formulas when needed.

Introduction

Machine learning learns by adjusting numbers (the parameters) until an error becomes small. To know in which direction to move a parameter you must know how the error changes when the parameter changes a little: this is exactly what the derivative measures. Without derivatives the training of neural networks would not exist. In this lesson we build the tool for functions of one variable; in the next we extend it to many variables (the gradient).

The path goes from average to instantaneous speed, introduces the limit with the ε-δ definition told as a game, defines the derivative and proves the differentiation rules. It then applies everything to the functions of neural networks (sigmoid, tanh, ReLU), explains the two errors of the numerical derivative and the Taylor expansion, and closes with maxima and minima. As prerequisites, the floating-point numbers and the binomial theorem from the combinatorics lesson are enough.

From average to instantaneous speed

The analogy: the speedometer

A car covers 120 km in 2 hours: the average speed is 60 km/h. But the speedometer shows, at every instant, a different number: the instantaneous speed. How is it defined, if in an instant "no time passes" and therefore the distance covered is zero? One looks at an ever smaller interval h around the instant and observes what number the ratio tends to

(distance covered in h) / h.

The number it tends to, when h approaches zero without ever being it, is the instantaneous speed. That "tends to" is a limit. Translated into mathematics: if f(x) is the position at time x, the instantaneous speed is the derivative f'(x).

The limit

We write lim_{x→a} f(x) = L and read "when x approaches a, f(x) approaches L". Two observations that surprise at the beginning:

• It does not matter what f is worth at a, nor even whether it is defined at a. What counts is what f does near a.
• "Approaching" must be understood from both sides.

The classic example: f(x) = (x² − 1)/(x − 1) is not defined at x = 1 (zero divided by zero). But for x ≠ 1 one can simplify: x² − 1 = (x − 1)(x + 1), so f(x) = x + 1, and f(x) approaches 2.

import math

f = lambda x: (x * x - 1) / (x - 1)
print("   x      f(x)")
for d in (0.5, 0.1, 0.01, 0.001, 1e-6):
    print(f"{1 - d:9.6f}  {f(1 - d):.6f}     {1 + d:9.6f}  {f(1 + d):.6f}")

g = lambda x: math.sin(x) / x
print("sin(x)/x for x → 0:", [round(g(x), 8) for x in (0.5, 0.1, 0.01, 0.001)])

Output:

   x      f(x)
 0.500000  1.500000      1.500000  2.500000
 0.900000  1.900000      1.100000  2.100000
 0.990000  1.990000      1.010000  2.010000
 0.999000  1.999000      1.001000  2.001000
 0.999999  1.999999      1.000001  2.000001
sin(x)/x for x → 0: [0.95885108, 0.99833417, 0.99998333, 0.99999983]

What the code does. It computes f for values of x approaching 1 from the left (1 − d) and from the right (1 + d): all the values approach 2 from both sides, even though f(1) does not exist. The second line shows a less obvious limit: sin(x)/x tends to 1 for x → 0 (with x in radians), a limit that is not obtained by algebraic simplification but from a geometric argument. Computing nearby values suggests the limit; it does not prove it: for that the definition is needed.

The precise definition: a two-player game

How do we say with rigour "f(x) approaches L as closely as we like"? As a game. You (the challenger) choose a tolerance ε > 0 (how close you want f(x) to be to L). I (who asserts the limit) must answer with a distance δ > 0 such that all the x within δ of a (excluding a itself) produce a value of f within ε of L. If, whatever ε you choose, I can always find a δ, the limit is L. It is the logic of the quantifiers from lesson 1:

lim_{x→a} f(x) = L ⇔ ∀ε > 0 ∃δ > 0 : 0 < |x − a| < δ ⇒ |f(x) − L| < ε.

Worked example: lim_{x→1} 3x = 3. Given ε, I must find δ. It holds that |3x − 3| = 3|x − 1|. I want it to be less than ε: |x − 1| < ε/3 is enough. So I choose δ = ε/3 and the answer holds for any ε. ∎

import numpy as np

def gioco(eps, delta, a=1.0, L=3.0, funzione=lambda x: 3 * x, punti=100_001):
    """Numerically check my answer δ to the challenge ε: test many x within δ of a."""
    xs = np.linspace(a - delta, a + delta, punti)
    xs = xs[xs != a]
    return bool(np.all(np.abs(funzione(xs) - L) < eps))

for eps in (1.0, 0.1, 1e-3, 1e-6):
    print(f"challenge ε = {eps:<7}: answer δ = ε/3 = {eps / 3:.2e} -> holds: {gioco(eps, eps / 3 * 0.999)}")
print("wrong answer δ = ε:", gioco(0.1, 0.1))

Output:

challenge ε = 1.0    : answer δ = ε/3 = 3.33e-01 -> holds: True
challenge ε = 0.1    : answer δ = ε/3 = 3.33e-02 -> holds: True
challenge ε = 0.001  : answer δ = ε/3 = 3.33e-04 -> holds: True
challenge ε = 1e-06  : answer δ = ε/3 = 3.33e-07 -> holds: True
wrong answer δ = ε: False

What the code does. gioco checks that, for many x within δ of a, the distance of f(x) from L is less than ε. The correct answer δ = ε/3 (reduced by a thousandth to avoid the exact boundary) holds for every challenge; the wrong answer δ = ε fails (here |3x − 3| can reach 3ε). It is a check, not a proof: the proof is the algebraic argument above.

Continuity. A function is continuous at a if lim_{x→a} f(x) = f(a): the limit exists, f(a) exists, and they coincide. Intuitively, the graph is drawn without lifting the pen.

The derivative

Definition

The derivative of f at the point x is

f'(x) = lim_{h→0} (f(x + h) − f(x)) / h

if the limit exists. The ratio (f(x+h) − f(x))/h is the difference quotient: the slope of the secant line joining two points of the graph at distance h. When h approaches zero the secant rotates and settles on the tangent: the derivative is the slope of the tangent.

Other equivalent names and notations: f'(x), df/dx, "rate of change", "sensitivity": how much the output changes if you move the input a little. A positive derivative means the function is increasing at that point, negative decreasing, zero "flat".

Computation from the definition

The parabola. For f(x) = x²: (f(x+h) − f(x))/h = ((x+h)² − x²)/h = (2xh + h²)/h = 2x + h. For h → 0: f'(x) = 2x.

The power xⁿ. By the binomial theorem (lesson 2): (x + h)ⁿ = xⁿ + n xⁿ⁻¹ h + C(n,2) xⁿ⁻² h² + …. Subtracting xⁿ and dividing by h: n xⁿ⁻¹ + C(n,2) xⁿ⁻² h + …. All terms beyond the first contain h and vanish for h → 0: (xⁿ)' = n xⁿ⁻¹.

def derivata_incr(f, x, h):
    return (f(x + h) - f(x)) / h

x = 3.0
print("(x^2)' at x=3, exact 6:  ", [round(derivata_incr(lambda t: t ** 2, x, h), 6) for h in (1, 0.1, 0.01, 0.001)])
print("(x^5)' at x=3, exact 405:", [round(derivata_incr(lambda t: t ** 5, x, h), 3) for h in (1, 0.1, 0.01, 0.001)])

Output:

(x^2)' at x=3, exact 6:   [7.0, 6.1, 6.01, 6.001]
(x^5)' at x=3, exact 405: [781.0, 432.915, 407.709, 405.27]

What the code does. It computes the difference quotient for ever smaller h and shows that it approaches the exact values 2·3 = 6 and 5·3⁴ = 405. For the parabola, the ratio is exactly 2x + h, so the convergence is visible: 7, 6.1, 6.01, 6.001.

The differentiation rules

Computing a limit every time would be tiring. A few rules are enough, each proved with the definition.

Linearity: (a·f + b·g)' = a·f' + b·g'. The difference quotient of a sum is the sum of the quotients, and constants come out of the limit.

Product rule: (f·g)' = f'·g + f·g'. Proof. f(x+h)g(x+h) − f(x)g(x) is rewritten by adding and subtracting f(x+h)g(x):

= f(x+h)[g(x+h) − g(x)] + g(x)[f(x+h) − f(x)].

Dividing by h and letting h → 0: the first piece becomes f(x)·g'(x) (because f(x+h) → f(x) by the continuity of f, a consequence of differentiability), the second g(x)·f'(x). ∎

Chain rule: (f(g(x)))' = f'(g(x)) · g'(x). It is the most important rule for machine learning: a neural network is a composition of functions, and its derivative is a product of derivatives. Proof (for g non-constant in the neighbourhood): with u = g(x) and k = g(x+h) − g(x),

(f(g(x+h)) − f(g(x)))/h = [(f(u + k) − f(u))/k] · [(g(x+h) − g(x))/h].

For h → 0 also k → 0 (g continuous), the first factor tends to f'(u) = f'(g(x)) and the second to g'(x). ∎ (The case in which k can vanish is treated with a somewhat finer argument, but the result does not change.)

The chain analogy: the gears. If one wheel turns 3 times faster than another, and this one 2 times faster than a third, the first turns 3 · 2 = 6 times faster than the third. The rates of change multiply.

Quotient. (f/g)' = (f'g − fg')/g² is obtained from the product rule applied to f · (1/g).

The "basic" derivatives

• Constant: (c)' = 0. Line: (ax + b)' = a.
• Exponential eˣ: the number e ≈ 2.71828 is defined so that (eˣ)' = eˣ: the function equals its slope. Numerically, (e^h − 1)/h → 1 for h → 0, which is the derivative at zero.
• Natural logarithm: from e^{ln x} = x, differentiating both sides with the chain rule: e^{ln x} · (ln x)' = 1, that is x · (ln x)' = 1, hence (ln x)' = 1/x.
• Sine and cosine: (sin x)' = cos x, (cos x)' = −sin x.

These four lines, combined with linearity, product, quotient and chain rule, are enough to differentiate almost any function you will meet. The logarithm is the only case that is not learned by heart: it is derived from the chain rule, and it is a good model of the reasoning to repeat when a new derivative is needed.

h = 1e-6
for nome, f, df in [("exp", math.exp, math.exp), ("ln", math.log, lambda x: 1 / x), ("sin", math.sin, math.cos)]:
    x = 1.7
    num = (f(x + h) - f(x - h)) / (2 * h)
    print(f"{nome:>3}: numerical derivative {num:.8f} | formula {df(x):.8f}")

print("(e^h - 1)/h for h = 1e-3, 1e-6 :", [round((math.exp(hh) - 1) / hh, 7) for hh in (1e-3, 1e-6)])

Output:

exp: numerical derivative 5.47394739 | formula 5.47394739
 ln: numerical derivative 0.58823529 | formula 0.58823529
sin: numerical derivative -0.12884449 | formula -0.12884449
(e^h - 1)/h for h = 1e-3, 1e-6 : [1.0005002, 1.0000005]

What the code does. It compares, for three functions, the numerical derivative (with the central difference (f(x+h) − f(x−h))/(2h), which we explain better later) with the formula: they coincide up to the eighth digit. The last line verifies that the ratio (e^h − 1)/h tends to 1.

The machine learning functions

An activation function is the non-linear function applied to the output of each neuron; its derivative must exist (almost everywhere) for training to work.

Sigmoid: σ(x) = 1/(1 + e⁻ˣ). It squashes the real numbers into (0, 1) and is read as a probability. Derivative, with the chain rule: σ(x) = (1 + e⁻ˣ)⁻¹, so

σ'(x) = −(1 + e⁻ˣ)⁻² · (−e⁻ˣ) = e⁻ˣ/(1 + e⁻ˣ)² = σ(x) · (1 − σ(x)),

because 1 − σ = e⁻ˣ/(1 + e⁻ˣ). It is a derivative expressed with the function itself: very convenient, because the value σ(x) has already been computed in the forward pass. Its shape has a known defect: the maximum derivative is 1/4 (at x = 0) and for large |x| it approaches zero, which slows learning ("vanishing gradient").

Hyperbolic tangent: tanh(x) = (eˣ − e⁻ˣ)/(eˣ + e⁻ˣ), with values in (−1, 1). Derivative 1 − tanh²(x).

ReLU: max(0, x). Derivative 0 for x < 0, 1 for x > 0, and it does not exist at x = 0: the two slopes (0 on the left, 1 on the right) do not coincide, the graph has a corner. In practice one chooses a convention (usually 0) for that isolated point, which is almost never hit.

sigma = lambda x: 1 / (1 + np.exp(-x))
x = np.linspace(-6, 6, 7)

dsig_formula = sigma(x) * (1 - sigma(x))
h = 1e-5
dsig_numerica = (sigma(x + h) - sigma(x - h)) / (2 * h)
print("x             :", x)
print("σ'(x) formula :", np.round(dsig_formula, 5))
print("σ'(x) numerical:", np.round(dsig_numerica, 5))
print("maximum of σ' :", dsig_formula.max(), "(= 1/4 at x = 0)")

tanh_deriv = 1 - np.tanh(x) ** 2
print("tanh'(x) formula vs numerical agree:", np.allclose(tanh_deriv, (np.tanh(x + h) - np.tanh(x - h)) / (2 * h)))

relu = lambda x: np.maximum(0.0, x)
print("ReLU slope to the left and right of 0:", (relu(0) - relu(-1e-8)) / 1e-8, (relu(1e-8) - relu(0)) / 1e-8)

Output:

x             : [-6. -4. -2.  0.  2.  4.  6.]
σ'(x) formula : [0.00247 0.01766 0.10499 0.25    0.10499 0.01766 0.00247]
σ'(x) numerical: [0.00247 0.01766 0.10499 0.25    0.10499 0.01766 0.00247]
maximum of σ' : 0.25 (= 1/4 at x = 0)
tanh'(x) formula vs numerical agree: True
ReLU slope to the left and right of 0: 0.0 1.0

What the code does. It computes the derivative of the sigmoid at seven points in two ways, with the formula σ(1−σ) and by finite differences, and the two rows coincide; it shows that the maximum is 0.25 and that at the edges (x = ±6) the derivative is ~0.002, almost zero. It then verifies the derivative of tanh. The last line measures the two slopes of ReLU around zero: 0 and 1, different, so the derivative at 0 does not exist.

The numerical derivative and its two errors

The computer does not know limits: it computes the difference quotient with a small but finite h. This introduces two competing errors.

1. Truncation error. The ratio is not the derivative, it is an approximation of it. To understand its size the Taylor expansion is needed (next section): for the forward difference the error is proportional to h, for the central one to h². The smaller h is, the better.
2. Rounding error. Subtracting two almost equal numbers (f(x+h) − f(x)) is the catastrophic cancellation of the floats lesson: the result loses significant digits; then one divides by a tiny h which amplifies the error. The error grows as ε_machine / h. The smaller h is, the worse.

The total is a V-shaped curve: an intermediate optimal h exists. The two errors go in opposite directions, so neither extreme choice is correct: with a large h truncation wins, with a tiny h rounding wins. The curve produced by the listing below shows where they meet for the function eˣ.

x0 = 1.0
esatto = math.exp(x0)                       # (e^x)' = e^x
print("   h       forward error       central error")
for k in range(1, 17):
    h = 10.0 ** (-k)
    avanti = (math.exp(x0 + h) - math.exp(x0)) / h
    centr = (math.exp(x0 + h) - math.exp(x0 - h)) / (2 * h)
    print(f"1e-{k:02d}   {abs(avanti - esatto):15.3e}   {abs(centr - esatto):15.3e}")

eps = np.finfo(float).eps
print("machine ε:", eps, "| optimal h forward ~ √ε =", f"{eps ** 0.5:.1e}", "| central ~ ε^(1/3) =", f"{eps ** (1 / 3):.1e}")

Output:

   h       forward error       central error
1e-01         1.406e-01         4.533e-03
1e-02         1.364e-02         4.530e-05
1e-03         1.360e-03         4.530e-07
1e-04         1.359e-04         4.531e-09
1e-05         1.359e-05         5.859e-11
1e-06         1.359e-06         1.635e-10
1e-07         1.399e-07         5.859e-11
1e-08         6.603e-09         6.603e-09
1e-09         2.154e-07         6.603e-09
1e-10         1.548e-06         6.727e-07
1e-11         3.263e-05         1.043e-05
1e-12         4.323e-04         2.103e-04
1e-13         4.559e-04         4.559e-04
1e-14         9.338e-03         9.338e-03
1e-15         3.903e-01         1.683e-01
1e-16         2.718e+00         2.718e+00
machine ε: 2.220446049250313e-16 | optimal h forward ~ √ε = 1.5e-08 | central ~ ε^(1/3) = 6.1e-06

What the code does. It computes the error of the two formulas for h from 10⁻¹ to 10⁻¹⁶ on the function eˣ at x = 1, where the exact answer is known. The forward difference reaches its minimum around h ≈ 10⁻⁸ (about √ε, with ε ≈ 2.2·10⁻¹⁶ the machine epsilon) and then worsens; the central one has its minimum lower (about 6·10⁻¹¹) and at a larger h, around 10⁻⁵, even though near the minimum the values do not fall smoothly (5.9·10⁻¹¹ at 10⁻⁵ and 10⁻⁷, 1.6·10⁻¹⁰ at 10⁻⁶) because rounding is random (theory gives ε^{1/3} ≈ 6·10⁻⁶). At h = 10⁻¹⁶ the ratio is 0 or garbage: x + h coincides with x because of rounding. Practical lesson: smaller is not better.

Taylor expansion

The analogy: the magnification

Zoom in on a smooth curve at a point: the more you magnify, the more the curve resembles its tangent (a line). The line is the best approximation of order 1. If you want something better you add the curvature: a parabola. And so on. Taylor's expansion (1715) does this systematically:

f(x + h) ≈ f(x) + f'(x)·h + f''(x)·h²/2 + f'''(x)·h³/6 + …

with f'' the derivative of the derivative (the second derivative, the curvature). The term of order n is f⁽ⁿ⁾(x) hⁿ/n!: the factorial from the combinatorics lesson appears.

From here the error of the forward difference can be read immediately: (f(x+h) − f(x))/h = f'(x) + f''(x)·h/2 + …, so the gap from the derivative is about f''·h/2: proportional to h. With the central one the even terms cancel and f'''·h²/6 remains: proportional to h².

def taylor_exp(h, ordine):
    """Approximation of e^h around 0 with the first (ordine+1) terms."""
    return sum(h ** k / math.factorial(k) for k in range(ordine + 1))

h = 1.0
print("e^1 =", round(math.exp(h), 8))
for ordine in (1, 2, 3, 5, 8):
    ap = taylor_exp(h, ordine)
    print(f"order {ordine}: {ap:.8f}  error {abs(ap - math.exp(h)):.2e}")

print("sigmoid near 0: σ(x) ≈ 1/2 + x/4  ->", [(x, round(float(sigma(x)), 5), round(0.5 + x / 4, 5)) for x in (0.1, 0.5, 1.0)])

Output:

e^1 = 2.71828183
order 1: 2.00000000  error 7.18e-01
order 2: 2.50000000  error 2.18e-01
order 3: 2.66666667  error 5.16e-02
order 5: 2.71666667  error 1.62e-03
order 8: 2.71827877  error 3.06e-06
sigmoid near 0: σ(x) ≈ 1/2 + x/4  -> [(0.1, 0.52498, 0.525), (0.5, 0.62246, 0.625), (1.0, 0.73106, 0.75)]

What the code does. taylor_exp sums the first terms of the expansion of eˣ (all the derivatives of eˣ are eˣ, which at 0 is 1, so the coefficients are 1/k!). The error falls rapidly with the order: with 9 terms (order 8) the error is already below 10⁻⁵. The last line linearises the sigmoid at zero, using σ(0) = 1/2 and σ'(0) = 1/4: the line 1/2 + x/4 is a good approximation near 0 and worsens moving away. This linearisation (approximating locally with a line) is at the basis of every optimisation method we discuss in the next lesson.

Maxima and minima

If a smooth function has a local maximum or minimum at x*, there the tangent is horizontal: f'(x*) = 0 (Fermat's theorem). Why: if f'(x*) > 0, moving a little to the right the function increases, so x* is not a maximum; if f'(x*) < 0 moving to the left it increases; only f'(x*) = 0 remains.

The condition is necessary, not sufficient: f(x) = x³ has derivative zero at 0 but has neither a maximum nor a minimum. To distinguish, the second derivative is used: f'' > 0 (cup-shaped curve opening upwards) indicates a minimum, f'' < 0 a maximum, f'' = 0 requires more care.

Important example: the mean minimises the squared error. Given y₁, …, yₙ, which number c summarises them while minimising S(c) = Σ (yᵢ − c)²? Differentiating with respect to c: S'(c) = −2 Σ (yᵢ − c) = 0 ⇒ Σ yᵢ = n·c ⇒ c = ȳ. And S''(c) = 2n > 0: it is a minimum. The mean is the summary that minimises the squared error. (With the absolute error Σ|yᵢ − c| the minimum is the median: hence mean and median behave differently with outliers, statistics lesson.)

y = np.array([2.0, 3.0, 4.0, 9.0, 12.0])

S = lambda c: ((y - c) ** 2).sum()
A = lambda c: np.abs(y - c).sum()
cs = np.linspace(0, 15, 15001)
print("minimum of Σ(y-c)²  at c =", round(float(cs[np.argmin([S(c) for c in cs])]), 3), "| mean =", y.mean())
print("minimum of Σ|y-c|   at c =", round(float(cs[np.argmin([A(c) for c in cs])]), 3), "| median =", np.median(y))

# find the zero-derivative point by bisection on the derivative of S
dS = lambda c: -2 * (y - c).sum()
lo, hi = 0.0, 15.0
for _ in range(60):
    medio = (lo + hi) / 2
    lo, hi = (medio, hi) if dS(medio) < 0 else (lo, medio)
print("zero of the derivative found by bisection:", round((lo + hi) / 2, 6))

Output:

minimum of Σ(y-c)²  at c = 6.0 | mean = 6.0
minimum of Σ|y-c|   at c = 4.0 | median = 4.0
zero of the derivative found by bisection: 6.0

What the code does. It looks for the minimum of two error functions on a grid of values of c: for the squared error it falls on the mean (6), for the absolute one on the median (4). Then it finds the minimum of the first with a different method: bisection on the derivative S'(c), which is negative to the left of the minimum and positive to the right; by repeatedly halving the interval in which it changes sign one converges to the zero of S'. It is a taste of what an optimiser does: looking for where the slope vanishes.

Typical mistakes (symptom, cause, remedy)

• Derivative of the product as product of the derivatives. (fg)' ≠ f'g'. Remedy: the product rule, with the two terms.
• Forgetting the inner derivative in the chain rule. Remedy: write f'(g(x)) · g'(x) in full.
• h too small in the numerical derivative. Remedy: h ≈ √ε (forward) or ε^{1/3} (central), and central difference.
• Derivative zero = minimum. Remedy: check the second derivative or compare the values.
• Differentiating where you cannot. The absolute value, ReLU, a step: corners. Remedy: a declared convention or sub-gradient.
• Confusing f'(x) with f(x): the function and its slope are different objects.
• Neglecting units. The derivative has units units of f / units of x (km/h, euros per customer).

The first two mistakes are algebraic and are exposed by the numerical check of the mini-project; the third and the fourth are errors of judgement about the computation; the last three concern the meaning of what is being computed. A good defence against all of them is to write down the unit of the derivative before using it: if you cannot state it, you are probably differentiating the wrong thing.

How to work in practice

Verify every analytic derivative with the numerical derivative (central difference, h ≈ 10⁻⁵): it is the gradient check, a test you will use in neural networks. Express the derivative with values already computed (σ' = σ(1−σ)), plot the function and its derivative to see where the almost-zero slopes and the corners are, and remember that for the computer limits do not exist: only a finite h does.

The essential vocabulary. The limit is the value f(x) approaches when x approaches a point; a function is continuous if the limit equals its value. The derivative is the slope of the tangent, that is the limit of the difference quotient (the slope of the secant). The chain rule says that the derivative of a composition is the product of the derivatives, the second derivative measures the curvature and to linearise means to replace the function locally with its tangent. The machine epsilon is the smallest relative spacing between two floats near 1.

Mini-project

1. Implement derivata_centrale(f, x, h=1e-5) and use it to verify the derivatives of x³, ln(1+x²) and sin(x²) at three points each, comparing with the formulas you obtain with the chain rule.
2. Plot (print a table of) the error of the central derivative of sin at x = 1 for h from 10⁻¹ to 10⁻¹⁵ and find the minimum.
3. Compute the Taylor expansion of sin x up to the fifth order and show for which x the error stays below 10⁻⁴.

The first point is developed in the worked challenge below. The second makes you reproduce by hand the V curve with the sine function, and the third applies Taylor to a function you have already differentiated: you expect the error to grow as you move away from zero, and the question is how fast.

A worked challenge: the derivative checker

The mini-project proposed building a small tool that checks hand-written derivatives. It is the gradient check in its one-variable version, and it is a habit that will save you many times. The three test functions require the chain rule: x³ (the derivative is 3x²), ln(1 + x²) (derivative 2x/(1 + x²): the derivative of the logarithm 1/u times the inner derivative 2x) and sin(x²) (derivative cos(x²)·2x).

import math

def derivata_centrale(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

casi = [
    ("x^3",         lambda x: x ** 3,               lambda x: 3 * x ** 2),
    ("ln(1 + x^2)", lambda x: math.log(1 + x * x),  lambda x: 2 * x / (1 + x * x)),
    ("sin(x^2)",    lambda x: math.sin(x * x),      lambda x: 2 * x * math.cos(x * x)),
]
print(f"{'function':<12} {'x':>5} {'numerical':>14} {'analytic':>14} {'relative error':>16}")
for nome, f, df in casi:
    for x in (0.5, 1.0, 2.0):
        num, ana = derivata_centrale(f, x), df(x)
        print(f"{nome:<12} {x:>5} {num:>14.8f} {ana:>14.8f} {abs(num - ana) / max(abs(ana), 1e-12):>16.2e}")

# error as the step varies, for sin(x) at x = 1: the V of the lesson
print()
x0, esatto = 1.0, math.cos(1.0)
for k in (2, 4, 5, 6, 8, 10, 12, 14):
    h = 10.0 ** (-k)
    print(f"h = 1e-{k:02d}: error = {abs(derivata_centrale(math.sin, x0, h) - esatto):.2e}")

Output:

function         x      numerical       analytic   relative error
x^3            0.5     0.75000000     0.75000000         1.32e-10
x^3            1.0     3.00000000     3.00000000         3.25e-11
x^3            2.0    12.00000000    12.00000000         1.77e-11
ln(1 + x^2)    0.5     0.80000000     0.80000000         5.10e-11
ln(1 + x^2)    1.0     1.00000000     1.00000000         1.57e-11
ln(1 + x^2)    2.0     0.80000000     0.80000000         1.00e-12
sin(x^2)       0.5     0.96891242     0.96891242         4.34e-11
sin(x^2)       1.0     1.08060461     1.08060461         2.21e-10
sin(x^2)       2.0    -2.61457448    -2.61457448         3.77e-10

h = 1e-02: error = 9.00e-06
h = 1e-04: error = 9.00e-10
h = 1e-05: error = 1.11e-11
h = 1e-06: error = 2.77e-11
h = 1e-08: error = 2.58e-09
h = 1e-10: error = 5.85e-08
h = 1e-12: error = 1.23e-05
h = 1e-14: error = 3.71e-03

How to read the code. derivata_centrale implements the central difference (f(x+h) − f(x−h))/(2h) with h = 10⁻⁵, the value that the lesson indicates as a good compromise. The list casi associates with each function its derivative computed by hand; the loop compares the two at three points and prints the relative error (the difference divided by the value, with a minimum in the denominator to avoid dividing by zero). A relative error of the order of 10⁻¹⁰ or 10⁻¹¹ says that the hand formula is correct; an error of the order of 10⁻¹ or more would have signalled a calculation mistake (a forgotten factor, a wrong sign): this is how, for example, the forgotten inner derivative in the chain rule is unmasked. The second part repeats the measurement of the V curve on the sine function: the error falls down to h ≈ 10⁻⁵-10⁻⁶ and then rises because of rounding. Keep it in mind when you choose h: smaller is not better.

Challenge

• Easy. Compute from the definition the derivative of f(x) = 3x² + 2x.
• Medium. Prove (1/g)' = −g'/g² with the definition of limit and derive the quotient rule.
• Hard. Derive tanh'(x) = 1 − tanh²(x) starting from tanh = 2σ(2x) − 1, using σ' = σ(1 − σ) and the chain rule.
• Expert. Prove with the Taylor expansion that the truncation error of the central difference is f'''(x)·h²/6 + …, and derive the h that minimises the sum of truncation and rounding error (hint: the rounding error is about ε·|f|/h).
• Enterprise. Take a cost function from your domain (total cost as a function of a threshold, a price, a capacity) and find numerically where the derivative vanishes; compare it with the optimum you derive by hand and discuss whether it is a minimum.

The first two challenges consolidate the definition and the quotient, the third links sigmoid and hyperbolic tangent, the fourth joins Taylor and rounding in a single argument. The last is the closest to real work: looking for the zero of the derivative of a cost function is exactly what an optimiser does.

What you can do now

You can compute a derivative from the definition and with the rules, differentiate sigmoid, tanh and ReLU and say where ReLU is not differentiable, choose a reasonable h for the numerical derivative knowing why the curve is V-shaped, and recognise in a maximum or a minimum the point where the slope vanishes. In the next lesson, gradient, integrals and gradient descent, the same idea moves from one variable to many; the computation that gives the derivative of a whole network is developed in Backpropagation, and gradient descent in Gradient descent.

The same series also includes: 20 - Statistics: Estimating, Comparing, Deciding (the previous lesson), 16 - Sets and Logic: the Language of Mathematics, 23 - Linear Algebra 1: Vectors, Matrices, Systems (which refer back to this lesson).