27 - Entropy, Cross-Entropy and KL Divergence: Why Loss Functions Are All the Same Thing

Machine Learning Engineer Course · Module 02 · Mathematics and statistics · Lesson 12

Goal. To derive entropy from the notion of surprise and to recognise the same formula under three different names. By the end you can compute an entropy in bits and in nats, you can prove with Jensen that KL(p‖q) ≥ 0, you can show in two lines that a classifier's negative log-likelihood is its cross-entropy, and you can recognise a tree's information gain as a mutual information.

Myth to dispel. "The KL divergence measures the distance between two distributions." It is not a distance: it is not symmetric and it does not respect the triangle inequality. With three distributions on a sheet of paper you can build the counterexample to both properties in a minute, and this is not pedantry: it is the difference that decides which of the two directions you are minimising, and therefore what shape the model you get will have.

Introduction

Three lessons you have already read use the same formula without ever naming it. In the lesson on decision trees you pick the question to put in a node by comparing entropies. In the lesson on logistic regression you train by minimising the negative log-likelihood. In the lesson on the Variational AutoEncoder a KL term appears inside the ELBO, taken to be non-negative without proof. Three names for one single quantity.

Here we close the circle, building the theory from the bottom up: the surprise of an event, its average, the cost of getting the distribution wrong. This is not elegance for its own sake: once you know that NLL = H(p, q) = H(p) + KL(p‖q), you also know which piece of the loss can go down and which cannot, and you stop expecting zero when the data do not allow it.

A note on scope. Shannon's information theory strictly speaking — source coding, channel capacity, error-correcting codes — has a dedicated series on this site. Here we look only at the loss-function side.

Final result of this lesson. You will be able to move from memory between "negative log-likelihood", "cross-entropy" and "entropy plus KL divergence", and to say, faced with a loss that has stopped going down, whether the culprit is the model or the data.

Lesson roadmap.

1. The surprise of an event, and why it has to be a logarithm.
2. Entropy as average surprise; bits versus nats.
3. The maximum of entropy, proved with Lagrange multipliers.
4. Cross-entropy, and the algebra that identifies it with the negative log-likelihood.
5. The KL divergence as waste, and Jensen's inequality proved in full.
6. Why KL is not a distance, and what changes between its two directions.
7. Mutual information, and the discovery that it is a tree's information gain.
8. How to implement all of this without producing nan.

The eight steps have a single thread: every quantity is the average or the gap of the previous one. Entropy is the average surprise, cross-entropy is the average surprise with the wrong belief, the KL divergence is the difference between the two, mutual information is a particular KL. Keeping this chain in mind makes memorising the formulas unnecessary.

Surprise and its average

The analogy: the telegram

Imagine sending a telegram, where you pay by the character, to report tomorrow's weather in Lecce. "Tomorrow the sun rises" is not worth one character: the reader already knew. "Tomorrow it snows" is worth many: it is rare, it was not predictable, and it genuinely changes the recipient's plans.

In engineering terms: the information in a message does not depend on the message, but on how improbable it was. We call this quantity surprise (surprisal, or self-information), and we look for the function S(p) that measures it from the probability p.

Why the logarithm, exactly

The logarithm is not chosen by taste: it follows from two requirements anyone would accept before seeing the formula.

• Requirement 1 — a certain event is not surprising. S(1) = 0, and S is decreasing in p.
• Requirement 2 — additivity over independent events. If two independent events have probabilities p and q, observing both must surprise as much as the sum: S(p·q) = S(p) + S(q).

The two requirements determine the function up to a constant. Set u = −ln p (hence p = e^{−u}, u ≥ 0) and define g(u) = S(e^{−u}). Requirement 2 becomes g(u + v) = g(u) + g(v), Cauchy's functional equation; requirement 1 makes g monotone with g(0) = 0. A monotone solution of Cauchy's equation is necessarily linear, g(u) = c·u with c > 0. Going back:

S(p) = −c · ln p.

The rule to take away. The constant c is not a degree of freedom: it is the choice of unit, that is, of the base. With c = 1/ln 2 you get S(p) = −log₂ p in bits, with c = 1 you get S(p) = −ln p in nats. Surprise is −log p not by convention, but because it is the only function that zeroes out the certain and adds up the independent.

Entropy: average surprise

Definition. The entropy of a discrete distribution p = (p₁, …, pₙ) is the expected surprise, that is, the average of the surprises weighted by the probabilities of the events themselves:

H(p) = E_p[ −log p ] = −Σᵢ pᵢ log pᵢ.

Read it as a sentence: how surprised am I on average, if the world really behaves as p says? It is the only average in which the weights and the quantity being weighted come from the same distribution, and that is why entropy is a property of the distribution, not of an outcome. If pᵢ = 0 the term is set to zero: the limit of p log p as p → 0 is 0, so this is not a shortcut but the correct value.

Bits or nats: the same quantity in two units

Changing the base multiplies the entropy by a constant, and nothing more. Since log₂ x = ln x / ln 2, we have H_bit = H_nat / ln 2, with ln 2 ≈ 0.693147. It is not a conceptual difference: it is a difference of unit, like between metres and feet.

The library you use picks nats for a practical reason: the derivative of ln x is 1/x, without the 1/ln 2 factor that would appear in every gradient. Scikit-learn works in bits because in trees the quantity is read rather than differentiated, and "0.456 bits of gain" is more interpretable than "0.316 nats".

import math

def sorpresa_bit(p):
    # surprise of a single outcome of probability p
    return -math.log2(p)

def entropia_bit(distribuzione):
    # average surprise: the weights are the probabilities themselves
    return -sum(p * math.log2(p) for p in distribuzione if p > 0)

moneta_truccata = [0.9, 0.1]
dado_equo = [1 / 6] * 6

print(f"surprise of 'heads' (p = 0.9): {sorpresa_bit(0.9):.6f} bit")
print(f"surprise of 'tails' (p = 0.1): {sorpresa_bit(0.1):.6f} bit")
print(f"entropy of the biased coin       : {entropia_bit(moneta_truccata):.6f} bit")
print(f"entropy of the fair die          : {entropia_bit(dado_equo):.6f} bit")
print(f"the same coin, in nats           : {entropia_bit(moneta_truccata) * math.log(2):.6f} nat")

Output obtained (Python 3.14.4):

surprise of 'heads' (p = 0.9): 0.152003 bit
surprise of 'tails' (p = 0.1): 3.321928 bit
entropy of the biased coin       : 0.468996 bit
entropy of the fair die          : 2.584963 bit
the same coin, in nats           : 0.325083 nat

What the code does. The if p > 0 filter implements the convention 0 · log 0 = 0 and avoids the exception from math.log2(0); the last line converts to nats by multiplying by ln 2. The number to look at is the third: a rare outcome is very surprising (3.32 bits for p = 0.1), but precisely because it is rare it carries little weight in the average, and the biased coin stops at 0.47 bits. The fair die, with six equally likely outcomes, reaches log₂ 6 ≈ 2.58 bits: for a given number of outcomes, uncertainty is maximal when none is privileged.

The maximum of entropy, proved with Lagrange

Theorem. For a distribution over n outcomes we have 0 ≤ H(p) ≤ log n. The minimum is attained if and only if the distribution is deterministic, the maximum if and only if it is uniform. The minimum is immediate: each term −pᵢ log pᵢ is non-negative and the sum vanishes only if every pᵢ is 0 or 1. The maximum is a constrained optimisation problem, that is, the use case for the Lagrange multipliers of the previous lesson in this module. Let us write the Lagrangian, with λ attached to the normalisation constraint (we work in nats; the base changes nothing):

L(p, λ) = −Σᵢ pᵢ ln pᵢ − λ · (Σᵢ pᵢ − 1)

The derivative of −p ln p is −ln p − 1, so stationarity reads ∂L/∂pᵢ = −ln pᵢ − 1 − λ = 0, that is pᵢ = e^{−1−λ}. The crucial point: the right-hand side does not depend on i. All the pᵢ are equal; the constraint Σᵢ pᵢ = 1 gives pᵢ = 1/n and hence H = ln n. That this is a maximum follows from strict concavity: −p ln p has second derivative −1/p < 0, and a strictly concave function on the simplex has a single stationary point, which is the global maximum. ∎

Later we will obtain the same theorem in one line and without derivatives: KL(p‖u) = ln n − H(p) ≥ 0 with u uniform. It is worth proving it twice: the first proof explains why the maximum is the uniform distribution, the second deduces it from a more general inequality.

import math

def entropia_bit(distribuzione):
    return -sum(p * math.log2(p) for p in distribuzione if p > 0)

uniforme = [0.25, 0.25, 0.25, 0.25]
sbilanciata = [0.7, 0.1, 0.1, 0.1]
deterministica = [1.0, 0.0, 0.0, 0.0]

print(f"H(uniform)        : {entropia_bit(uniforme):.6f} bit")
print(f"H(unbalanced)     : {entropia_bit(sbilanciata):.6f} bit")
print(f"H(deterministic)  : {entropia_bit(deterministica):.6f} bit")
print(f"upper limit       : {math.log2(4):.6f} bit  (= log2 n, with n = 4)")

Output obtained:

H(uniform)        : 2.000000 bit
H(unbalanced)     : 1.356780 bit
H(deterministic)  : -0.000000 bit
upper limit       : 2.000000 bit  (= log2 n, with n = 4)

What the code does. Three distributions over the same four outcomes, written by hand and not sampled, so that the numbers are exact: the uniform one touches the bound log₂ 4 = 2, the skewed one loses about 0.64 bits, the deterministic one never informs. Note the -0.000000 on the third line: that is negative zero, the IEEE 754 value produced by -(1.0 * 0.0), not a rounding error (the same phenomenon as in the lesson on floating-point numbers). Since 0.0 == -0.0, a comparison never notices, but a print does. Entropy measures how hard it is to guess, not how important the outcome is: two sections from now this number becomes the unbreachable floor of any loss function.

Cross-entropy: the cost of the wrong code

Definition. Given two distributions p and q over the same set of outcomes, the cross-entropy of q relative to p is

H(p, q) = −Σᵢ pᵢ log qᵢ.

Read it as a sentence: the average surprise I experience if the world behaves like p but I was expecting q. The weights come from reality, the surprises from my belief; in the telegram picture, it is the average length when the code is tuned on q and the messages arrive with frequencies p. If q = p we fall back on entropy. Two properties to fix straight away: it is not symmetric (H(p, q) ≠ H(q, p)) and it is never smaller than the entropy (H(p, q) ≥ H(p), with equality only if q = p) — we will prove the latter shortly, it is Gibbs' inequality.

Negative log-likelihood is cross-entropy

Here comes the identification that gives the lesson its title, in four lines of algebra. You have n examples (xᵢ, yᵢ) with yᵢ ∈ {1, …, K} and a classifier producing a distribution q_θ(· | xᵢ) over the K classes. The average negative log-likelihood — the loss you minimise when training a logistic regression or a neural network — is

NLL(θ) = −(1/n) · Σᵢ ln q_θ(yᵢ | xᵢ)

Let us introduce the one-hot distribution of the true label, p⁽ⁱ⁾_k = 1 if k = yᵢ and 0 otherwise: it sums to 1, so it is a legitimate distribution. Then ln q_θ(yᵢ | xᵢ) = Σ_k p⁽ⁱ⁾_k · ln q_θ(k | xᵢ), because only one summand survives. Substituting:

NLL(θ) = −(1/n) · Σᵢ Σ_k p⁽ⁱ⁾_k ln q_θ(k | xᵢ) = (1/n) · Σᵢ H(p⁽ⁱ⁾, q_θ(· | xᵢ))

The average negative log-likelihood is the average of the per-example cross-entropies. Not "is related to": it is the same sum rewritten. The loss in the lesson on logistic regression and the loss in the lesson on the network from scratch are the same line of code, and that is why PyTorch calls CrossEntropyLoss what a statistician would call a negative log-likelihood.

import math

# four examples, three classes; predictions written by hand to get exact numbers
etichette = [0, 1, 2, 0]
previsioni = [
    [1 / 2, 1 / 4, 1 / 4],      # almost right
    [1 / 8, 3 / 4, 1 / 8],      # right and confident
    [1 / 4, 1 / 4, 1 / 2],      # almost right
    [1 / 8, 1 / 8, 3 / 4],      # wrong AND confident: this is the one that costs
]

def una_calda(classe, k):
    # the one-hot distribution of the true label
    return [1.0 if i == classe else 0.0 for i in range(k)]

nll, entropia_incrociata = 0.0, 0.0
for y, q in zip(etichette, previsioni):
    p = una_calda(y, 3)
    termine_nll = -math.log2(q[y])                                   # looks only at the true class
    termine_ce = -sum(p_k * math.log2(q_k) for p_k, q_k in zip(p, q) if p_k > 0)
    print(f"true class {y}: -log2 q[y] = {termine_nll:.6f} bit")
    nll += termine_nll / 4
    entropia_incrociata += termine_ce / 4

print(f"average negative log-likelihood: {nll:.6f} bit = {nll * math.log(2):.6f} nat")
print(f"average cross-entropy          : {entropia_incrociata:.6f} bit = {entropia_incrociata * math.log(2):.6f} nat")
print(f"they are the same number: {abs(nll - entropia_incrociata) < 1e-12}")

Output obtained:

true class 0: -log2 q[y] = 1.000000 bit
true class 1: -log2 q[y] = 0.415037 bit
true class 2: -log2 q[y] = 1.000000 bit
true class 0: -log2 q[y] = 3.000000 bit
average negative log-likelihood: 1.353759 bit = 0.938354 nat
average cross-entropy          : 1.353759 bit = 0.938354 nat
they are the same number: True

What the code does. The loop computes the same quantity by two routes: termine_nll looks only at the probability of the true class, termine_ce sums over all three weighting by the one-hot vector, which zeroes out every summand but one. Look at the fourth example: the model puts 3/4 on the wrong class and 1/8 on the right one, and pays 3 bits, more than twice the average. Cross-entropy punishes wrong confidence far more than it rewards right confidence, because −log q grows without bound as q → 0: a single near-zero prediction on the correct class can dominate the loss of an entire minibatch.

The KL divergence: the waste, not the total cost

Definition. The Kullback-Leibler divergence from p to q is the difference between cross-entropy and entropy:

KL(p‖q) = H(p, q) − H(p) = Σᵢ pᵢ log (pᵢ / qᵢ)

Read it as a sentence: how much extra I pay for having used q instead of p. It is not the total cost, it is the waste. The cost H(p, q) contains a piece that does not depend on me (H(p), the intrinsic uncertainty of the data) and one that depends only on the model. In the exercise a few lines ago: 2 bits were the cross-entropy, 1.75 the entropy, 0.25 the KL divergence.

When you train, H(p) does not move. Only KL(p‖q) goes down.

The entropy of the data is a property of the dataset, not of the parameters: its derivative with respect to θ is identically zero. Minimising cross-entropy and minimising KL are therefore the same problem, with objectives differing by an additive constant. And if the data have noisy labels — the same input with different labels — then H(p) > 0 and the loss will never reach zero, however large the network. It is not a bug: it is the floor.

Jensen's inequality, proved

We have claimed twice that KL ≥ 0. The tool to prove it is Jensen's inequality, the same instrument the lesson on the Variational AutoEncoder uses to derive the ELBO: there it is used, here it is proved. Statement: let φ be convex and X a discrete random variable with values x₁, …, xₙ and probabilities w₁, …, wₙ; then φ(Σᵢ wᵢxᵢ) ≤ Σᵢ wᵢ φ(xᵢ), and if φ is strictly convex equality holds only for constant X.

Base case (n = 2). This is the definition of convexity: φ(t·x₁ + (1−t)·x₂) ≤ t·φ(x₁) + (1−t)·φ(x₂) for every t ∈ [0, 1]. Geometrically, the chord between two points of the graph lies above the graph.

Inductive step. True for n−1 points, take n of them with weights summing to 1. If wₙ = 1 there is nothing to prove; otherwise set s = 1 − wₙ > 0 and y = Σ_{i<n} (wᵢ/s)·xᵢ, a convex combination of n−1 points because the weights wᵢ/s sum to 1. Then Σᵢ wᵢxᵢ = s·y + wₙ·xₙ, and applying first the two-point case and then the inductive hypothesis:

φ(Σᵢ wᵢxᵢ) = φ(s·y + wₙ·xₙ) ≤ s·φ(y) + wₙ·φ(xₙ) ≤ s·Σ_{i<n}(wᵢ/s)·φ(xᵢ) + wₙ·φ(xₙ) = Σᵢ wᵢ·φ(xᵢ)

where the s cancels in the last step. ∎

Application: Gibbs' inequality. The function φ(x) = −ln x is strictly convex on x > 0, because φ″(x) = 1/x² > 0. Take as random variable the ratio X = qᵢ/pᵢ, with i drawn according to p:

−KL(p‖q) = Σᵢ pᵢ ln(qᵢ/pᵢ) = E_p[ln X] ≤ ln E_p[X] = ln ( Σᵢ pᵢ · qᵢ/pᵢ ) = ln ( Σ_{i: pᵢ > 0} qᵢ ) ≤ ln 1 = 0

The first inequality is Jensen applied to −ln, the second holds because a partial sum of the qᵢ cannot exceed 1. Changing sign: KL(p‖q) ≥ 0. Equality requires both relations with an equals sign: strict convexity forces qᵢ/pᵢ to be constant wherever pᵢ > 0, and the fact that both sum to 1 forces that constant to be 1, that is q = p. ∎

The rule to take away. KL ≥ 0 is not an axiom: it is Jensen applied to the logarithm. And since KL(p‖q) = H(p, q) − H(p), the same argument proves that no model can cost less than the entropy of the data. It is the most important result of this lesson.

KL is not a distance: asymmetry, triangle inequality and the two directions

The name "divergence" instead of "distance" is not a nuance. A distance must be symmetric and respect the triangle inequality; KL fails both, and you can see it with three distributions over three outcomes chosen with convenient numbers: p = (3/4, 1/8, 1/8), q = (1/2, 1/4, 1/4), r = (1/8, 3/4, 1/8).

import math

def divergenza_kl_bit(p, q):
    totale = 0.0
    for p_i, q_i in zip(p, q):
        if p_i == 0.0:
            continue                 # 0 · log(0/q) = 0, by continuity
        if q_i == 0.0:
            return math.inf          # the model rules out what the data observe
        totale += p_i * math.log2(p_i / q_i)
    return totale

p = [0.75, 0.125, 0.125]
q = [0.5, 0.25, 0.25]
r = [0.125, 0.75, 0.125]

print(f"KL(p || q) = {divergenza_kl_bit(p, q):.6f} bit")
print(f"KL(q || p) = {divergenza_kl_bit(q, p):.6f} bit   <- different: it is not symmetric")
print(f"KL(q || r) = {divergenza_kl_bit(q, r):.6f} bit")
print(f"KL(p || q) + KL(q || r) = {divergenza_kl_bit(p, q) + divergenza_kl_bit(q, r):.6f} bit")
print(f"KL(p || r)              = {divergenza_kl_bit(p, r):.6f} bit   <- more than the sum")
print(f"KL(p || p) = {divergenza_kl_bit(p, p):.6f} bit")

Output obtained:

KL(p || q) = 0.188722 bit
KL(q || p) = 0.207519 bit   <- different: it is not symmetric
KL(q || r) = 0.853759 bit
KL(p || q) + KL(q || r) = 1.042481 bit
KL(p || r)              = 1.615602 bit   <- more than the sum
KL(p || p) = 0.000000 bit

What the code does. The two limiting cases are handled explicitly: where p is zero the term is skipped, where q is zero but p is not the divergence is infinite and the function says so instead of producing a silent nan. The numbers say two things. KL(p‖q) = 0.188722 against KL(q‖p) = 0.207519: no symmetry. And the direct route from p to r costs 1.615602 bits while going through q costs 1.042481: the triangle inequality is violated by more than half a bit. Only what we proved survives, KL ≥ 0 and KL(p‖p) = 0: enough to use it as an objective function, not enough to call it a distance.

Forward KL and reverse KL: covering or seeking a mode

Since the two directions do not coincide, choosing which to minimise is a design decision. The sharpest way to see it is to fit a single Gaussian to a bimodal distribution, in both senses. Target: the mixture p(x) = ½·N(x; −2, 1) + ½·N(x; +2, 1), two identical hills separated by a valley.

Forward, min_q KL(p‖q). The average is taken according to p: wherever p puts mass the model pays ln(p/q), and if q is nearly zero where p is not, the cost explodes. It is zero-avoiding, and the solution "covers" the whole support. For an exponential family such as the Gaussians the solution is in closed form: minimising KL(p‖q) is equivalent to maximising E_p[ln q], whose stationarity conditions require matching the moments. Hence μ = 0 and σ² = ½(1 + 4) + ½(1 + 4) − 0² = 5, that is σ = √5 ≈ 2.236068: a wide bell, centred right in the valley.

Reverse, min_q KL(q‖p). Here the average is taken according to q: the model pays only where it puts mass, and is not punished for ignoring a region it never visits. It is zero-forcing, and the solution glues itself to one mode. In the limit of well-separated modes the calculation can be done by hand: near the right-hand hill p(x) ≈ ½·N(x; 2, 1), hence KL(q‖p) ≈ KL(q‖N(2,1)) + ln 2, minimised by q = N(2, 1) with residual ln 2 ≈ 0.693147 nats. The symmetric solution at −2 is equally optimal: the problem has two minima, and which one you find depends on where the optimiser starts.

import math

# bimodal mixture, parameters written by hand: two Gaussians of weight 1/2
pesi = [0.5, 0.5]
medie = [-2.0, 2.0]
varianze = [1.0, 1.0]

# forward KL: the solution is in closed form, it matches mean and variance
media_mistura = sum(w * m for w, m in zip(pesi, medie))
secondo_momento = sum(w * (v + m * m) for w, m, v in zip(pesi, medie, varianze))
varianza_mistura = secondo_momento - media_mistura ** 2

print(f"forward  -> N(mu = {media_mistura:.6f}, sigma = {math.sqrt(varianza_mistura):.6f})")
print(f"reverse  -> N(mu = {medie[1]:.6f}, sigma = {math.sqrt(varianze[1]):.6f})")
print(f"residual KL(q || p) in the limit of separated modes: {math.log(2):.6f} nat")
print(f"ratio of the two widths: {math.sqrt(varianza_mistura) / math.sqrt(varianze[1]):.6f}")

Output obtained:

forward  -> N(mu = 0.000000, sigma = 2.236068)
reverse  -> N(mu = 2.000000, sigma = 1.000000)
residual KL(q || p) in the limit of separated modes: 0.693147 nat
ratio of the two widths: 2.236068

What the code does. The forward direction needs no numerical optimisation: the mean and the second moment of the mixture have exact formulae, and the variance is E[x²] − E[x]². The reverse direction is reported at its analytic limiting value, valid when the modes are separated enough to neglect the other's tail: with four standard deviations of separation, as here, numerical optimisation would move the parameters only in the third decimal place. The ratio between the widths is 2.236068: the forward solution is more than twice as wide as the reverse one.

Trade-off. Neither direction is "the right one". If you are estimating a density and want the model never to declare impossible anything that happens, you need the forward one: it pays dearly for wrong zeros. If you are building an approximation you will have to sample from, the reverse one is often preferable: a wide Gaussian centred in the valley would generate samples in a region the target never visits. The question is always the same: do I need coverage or precision?

Mutual information

Definition (two ways). The mutual information between X and Y is the reduction in uncertainty about one when the other is observed, I(X; Y) = H(Y) − H(Y | X), where H(Y | X) = Σ_x p(x) · H(Y | X = x) is the average of the entropies of the "branches". But it is also the KL divergence between the joint and the product of the marginals, I(X; Y) = KL( p(x, y) ‖ p(x)·p(y) ), that is, how far reality departs from independence. The two definitions coincide; let us start from the second and use p(x, y) = p(x)·p(y|x):

KL( p(x,y) ‖ p(x)p(y) ) = Σ_{x,y} p(x,y) log ( p(x,y) / (p(x)p(y)) ) = Σ_{x,y} p(x,y) log ( p(y|x) / p(y) )

= Σ_{x,y} p(x,y) log p(y|x) − Σ_{x,y} p(x,y) log p(y) = −H(Y|X) + H(Y)

In the last step the first summand is by definition −H(Y|X), and in the second the sum over x marginalises p(x,y) into p(y). ∎ Two properties follow for free: mutual information is symmetric, because the second definition obviously is, and it is zero if and only if the variables are independent, because it is a KL and a KL vanishes only when the two distributions coincide. Two corollaries of KL ≥ 0 that would have been laborious to get from the first definition.

import math

# sixteen synthetic examples, written by hand. "canale" (channel) is informative, "rumore" (noise) is not.
canale    = [0] * 8 + [1] * 8
etichetta = [0, 0, 0, 0, 0, 0, 0, 1] + [1, 1, 1, 1, 1, 1, 1, 0]
rumore    = [0, 0, 0, 1, 1, 1, 1, 0] * 2

def entropia_bit(frequenze):
    totale = sum(frequenze)
    return -sum((c / totale) * math.log2(c / totale) for c in frequenze if c > 0)

def conta(valori, maschera=None):
    coppie = valori if maschera is None else [v for v, m in zip(valori, maschera) if m]
    return [coppie.count(0), coppie.count(1)]

def mutua_per_riduzione(x, y):
    # I(X; Y) = H(Y) - H(Y | X)
    h_y = entropia_bit(conta(y))
    h_condizionata = 0.0
    for valore in (0, 1):
        maschera = [v == valore for v in x]
        peso = sum(maschera) / len(x)
        h_condizionata += peso * entropia_bit(conta(y, maschera))
    return h_y, h_condizionata, h_y - h_condizionata

def mutua_per_kl(x, y):
    # I(X; Y) = KL( joint || product of the marginals )
    n = len(x)
    totale = 0.0
    for a in (0, 1):
        for b in (0, 1):
            congiunta = sum(1 for u, v in zip(x, y) if u == a and v == b) / n
            if congiunta == 0.0:
                continue
            marginale_x = sum(1 for u in x if u == a) / n
            marginale_y = sum(1 for v in y if v == b) / n
            totale += congiunta * math.log2(congiunta / (marginale_x * marginale_y))
    return totale

h_y, h_cond, mutua = mutua_per_riduzione(canale, etichetta)
print(f"H(label)                  : {h_y:.6f} bit")
print(f"H(label | channel)        : {h_cond:.6f} bit")
print(f"I(label; channel)         : {mutua:.6f} bit")
print(f"I via KL(joint||prod.)    : {mutua_per_kl(canale, etichetta):.6f} bit")
print(f"the two routes agree within 1e-12: {abs(mutua - mutua_per_kl(canale, etichetta)) < 1e-12}")
print(f"I(channel; label)         : {mutua_per_kl(etichetta, canale):.6f} bit   <- symmetric")
print(f"I(label; noise)           : {mutua_per_kl(rumore, etichetta):.6f} bit   <- independent")

Output obtained:

H(label)                  : 1.000000 bit
H(label | channel)        : 0.543564 bit
I(label; channel)         : 0.456436 bit
I via KL(joint||prod.)    : 0.456436 bit
the two routes agree within 1e-12: True
I(channel; label)         : 0.456436 bit   <- symmetric
I(label; noise)           : 0.000000 bit   <- independent

What the code does. The data are sixteen made-up examples written out explicitly: where canale is 0 the label is 0 seven times out of eight and vice versa, while rumore splits the dataset into two halves with exactly the same label composition. mutua_per_riduzione implements H(Y) − H(Y|X), weighting the entropies of the branches by their frequency; mutua_per_kl reconstructs the 2×2 table of the joint. The two routes give the same number down to the last digit, and rumore gives exactly zero — not "nearly zero by chance", but zero because the joint is the product of the marginals. Observing canale takes the uncertainty from 1 bit to 0.543564: the gain is 0.456436 bits. Keep it in mind for thirty seconds.

A tree's information gain is mutual information

A decision tree with the entropy criterion picks the variable to split on by maximising the information gain, that is, the parent's entropy minus the weighted average of the children's entropies:

IG(Y, X) = H(Y) − Σ_v (n_v / n) · H(Y | X = v)

Compare it with the conditional entropy H(Y|X) = Σ_v p(v)·H(Y|X=v), where p(v) estimated on the data is precisely n_v/n: the two expressions coincide term by term, hence IG(Y, X) = I(Y; X). Information gain does not "resemble" mutual information: it is mutual information on the empirical distribution. A tree is then a greedy algorithm that at each node picks the question with the largest mutual information with the label — which also explains its limitation, because greedy means "largest one step ahead", and a pair of variables can be informative when neither is on its own. Let us check it against scikit-learn, on the same sixteen examples.

import numpy as np
from sklearn.tree import DecisionTreeClassifier

# reuses canale, etichetta, rumore and mutua_per_kl defined above
X = np.column_stack([canale, rumore]).astype(float)
y = np.array(etichetta)

albero = DecisionTreeClassifier(criterion="entropy", max_depth=1, random_state=0).fit(X, y)
impurita = albero.tree_.impurity           # entropy of each node, in bits
numerosita = albero.tree_.n_node_samples

print(f"column chosen for the split: {albero.tree_.feature[0]}  (0 = channel, 1 = noise)")
print(f"threshold                  : {albero.tree_.threshold[0]:.1f}")
print(f"entropy of the root        : {impurita[0]:.6f} bit  ({numerosita[0]} examples)")
print(f"entropy of the left child   : {impurita[1]:.6f} bit  ({numerosita[1]} examples)")
print(f"entropy of the right child  : {impurita[2]:.6f} bit  ({numerosita[2]} examples)")

guadagno = impurita[0] - sum(
    (numerosita[figlio] / numerosita[0]) * impurita[figlio] for figlio in (1, 2)
)
print(f"information gain of the tree : {guadagno:.6f} bit")
print(f"mutual information by hand   : {mutua_per_kl(canale, etichetta):.6f} bit")

Output obtained (NumPy 2.3.5, scikit-learn 1.8.0):

column chosen for the split: 0  (0 = channel, 1 = noise)
threshold                  : 0.5
entropy of the root        : 1.000000 bit  (16 examples)
entropy of the left child   : 0.543564 bit  (8 examples)
entropy of the right child  : 0.543564 bit  (8 examples)
information gain of the tree : 0.456436 bit
mutual information by hand   : 0.456436 bit

What the code does. We train a depth-1 tree — a single split — with the entropy criterion: tree_.impurity holds the entropy of each node in the order root, left child, right child, and tree_.n_node_samples the sample counts. The same number, 0.456436 bits, comes out three times by three independent routes: as a reduction of entropy, as a KL divergence between joint and product, and as the internal gain of a library that knew nothing of the other two. The tree picks canale and ignores rumore, whose gain is exactly zero. It is also the reason scikit-learn works in bits: that number ends up in a chart a human being has to read.

Implementing them without getting hurt

The logarithm of zero

The fastest way to fill a training run with nan is to compute log(0). It happens with a probability that underflows, with a class never seen in a minibatch, or with an already normalised vector to which the softmax is applied again. The result is not an error: it is a -inf which, multiplied by zero, becomes nan, propagates into the gradient and wipes out the weights in one step.

import math

try:
    math.log(0.0)
except ValueError as errore:
    print("math.log(0.0) ->", errore)

print("0.0 * (-inf) ->", 0.0 * float("-inf"))
print(f"with the clip at 1e-12, the surprise stops at {-math.log(1e-12):.6f} nat")
print(f"that is {-math.log2(1e-12):.6f} bit: large, but finite")

Output obtained (Python 3.14.4):

math.log(0.0) -> expected a positive input, got 0.0
0.0 * (-inf) -> nan
with the clip at 1e-12, the surprise stops at 27.631021 nat
that is 39.863137 bit: large, but finite

What the code does. math.log(0.0) raises an exception — the wording of the message changed in recent Python versions, but what matters is that it is an exception and not a value — whereas NumPy and PyTorch return -inf with a warning; in both cases 0 · (−inf) gives nan, and that is where the trouble starts. The last part shows the most widespread fix, clipping: bounding the probabilities to [ε, 1−ε] with ε = 1e-12 keeps the maximum surprise at 27.6 nats, large but finite, and the gradient stays computable.

A real trap: clipping masks the symptom, not the cause. If the model assigns 1e-12 to the correct class, the problem is not the logarithm: it is the model. Clipping exists so you do not lose a training run over one row, not so you can live forever with impossible predictions. Put a counter on how often it kicks in: if it grows, the diagnosis lies elsewhere.

logsumexp, derived

The second obstacle is the opposite: the exponential that explodes. To go from raw scores (logits) to probabilities you use the softmax, softmax(z)ᵢ = e^{zᵢ} / Σⱼ e^{zⱼ}, and with logits of the order of hundreds e^z leaves the range of float64, which stops at around 1.8 · 10³⁰⁸ (lesson on floating-point numbers).

The fix is derived in one line. For every constant c we have log Σⱼ e^{zⱼ} = log ( e^c · Σⱼ e^{zⱼ−c} ) = c + log Σⱼ e^{zⱼ−c}: adding and removing c does not change the mathematical result, but it radically changes the numerical one. Choosing c = max_j zⱼ, the largest exponential becomes e⁰ = 1 and the others lie in (0, 1]: no overflow is possible, and underflow on the tiniest terms is harmless because they contribute negligibly to the sum. This is logsumexp, and log_softmax(z) = z − logsumexp(z) follows.

import math

def logsumexp(valori):
    massimo = max(valori)
    return massimo + math.log(sum(math.exp(v - massimo) for v in valori))

logit = [1000.0, 1001.0, 1002.0]
try:
    [math.exp(z) for z in logit]
except OverflowError as errore:
    print("naive softmax, math.exp(1000.0) ->", errore)

lse = logsumexp(logit)
log_prob = [z - lse for z in logit]
print(f"stable logsumexp  : {lse:.6f}")
print("log_softmax       :", " ".join(f"{v:.6f}" for v in log_prob))
print("softmax           :", " ".join(f"{math.exp(v):.6f}" for v in log_prob))

# the opposite failure: a probability that underflows
logit_estremo = [0.0, -800.0]
lse2 = logsumexp(logit_estremo)
probabilita = [math.exp(z - lse2) for z in logit_estremo]
print("softmax(x)        :", " ".join(f"{v:.6e}" for v in probabilita))
print("log(softmax(x))[1] : -inf   (the probability became 0.0: information lost)")
print(f"log_softmax(x)[1]  : {logit_estremo[1] - lse2:.6f}   (exact, never exponentiating)")

Output obtained:

naive softmax, math.exp(1000.0) -> math range error
stable logsumexp  : 1002.407606
log_softmax       : -2.407606 -1.407606 -0.407606
softmax           : 0.090031 0.244728 0.665241
softmax(x)        : 1.000000e+00 0.000000e+00
log(softmax(x))[1] : -inf   (the probability became 0.0: information lost)
log_softmax(x)[1]  : -800.000000   (exact, never exponentiating)

What the code does. The first half shows the overflow and the fix: with logits around 1000 the direct exponential fails, whereas logsumexp returns 1002.407606 and the correct probabilities, which depend only on the differences between the logits. The second half shows the symmetric and more insidious failure: with logits [0, −800] the softmax produces an exact zero and log(softmax(x)) loses the information by returning -inf, whereas log_softmax, which never exponentiates, gives the right value, −800.

The rule to take away. torch.nn.CrossEntropyLoss is not API laziness: it fuses log_softmax and the negative log-likelihood into a single operation precisely so as never to materialise the probabilities. Hence the most common mistake in applied deep learning: passing it a tensor on which you already called softmax. The code does not complain, the loss goes down, the model learns something — but you are minimising the cross-entropy of a softmax of a softmax, a different and much flatter function. If the loss starts around ln K and falls extremely slowly, check this first.

from scipy.special import logsumexp
from scipy.stats import entropy

p = [0.75, 0.125, 0.125]
q = [0.5, 0.25, 0.25]

print(f"entropy(p, base=2)        : {entropy(p, base=2):.6f} bit")
print(f"entropy(p, q, base=2) = KL: {entropy(p, q, base=2):.6f} bit")
print(f"entropy(p) in nat         : {entropy(p):.6f} nat")
print(f"logsumexp([1000,1001,1002]): {logsumexp([1000.0, 1001.0, 1002.0]):.6f}")

Output obtained (SciPy 1.16.3):

entropy(p, base=2)        : 1.061278 bit
entropy(p, q, base=2) = KL: 0.188722 bit
entropy(p) in nat         : 0.735622 nat
logsumexp([1000,1001,1002]): 1002.407606

What the code does. scipy.stats.entropy surprises anyone who only reads its name: with one argument it computes the entropy, with two the KL divergence. SciPy has no function for H(p, q): if you need the cross-entropy you get it by adding the two calls, 1.061278 + 0.188722 = 1.250000 bits. The library's logsumexp agrees with our hand-written one down to the last digit printed, and the 0.188722 on the second line is the same number as in the asymmetry snippet: two independent implementations of the same definition.

The experiment: three ways, one single number

Let us put everything together on a synthetic three-class problem, built so that the entropy of the data is not zero. Twelve examples in three groups of four; inside each group the inputs are identical but the labels are not, so the conditional distribution is genuinely uncertain: in group A two examples out of four are of class 0 and one each of the other two, and so on by rotation. This is the situation of any dataset with noisy labelling or overlapping classes, that is, the case in which H(p, q) = H(p) + KL(p‖q) stops being academic.

import numpy as np

# three groups of four examples; within a group the input is the same,
# the label is not. Numbers written by hand to get exact values.
P = np.array([[0.50, 0.25, 0.25],
              [0.25, 0.50, 0.25],
              [0.25, 0.25, 0.50]])
etichette = np.array([[0, 0, 1, 2],
                      [1, 1, 0, 2],
                      [2, 2, 0, 1]])

def interpola(P, t):
    # model: a mixture of the uniform (t = 0) and the truth (t = 1)
    return (1 - t) / 3 + t * P

Q = interpola(P, 0.5)

# (a) average negative log-likelihood, example by example
nll = -np.mean([np.log(Q[g, y]) for g in range(3) for y in etichette[g]])

# (b) average cross-entropy, group by group
entropia_incrociata = np.mean(-np.sum(P * np.log(Q), axis=1))

# (c) entropy of the data plus KL divergence
entropia_dati = np.mean(-np.sum(P * np.log(P), axis=1))
divergenza = np.mean(np.sum(P * np.log(P / Q), axis=1))

scarti = [abs(nll - entropia_incrociata), abs(nll - entropia_dati - divergenza)]
print(f"(a) average negative log-likelihood: {nll:.9f} nat")
print(f"(b) average cross-entropy          : {entropia_incrociata:.9f} nat")
print(f"(c) H(p) + KL(p || q)              : {entropia_dati + divergenza:.9f} nat")
print(f"maximum gap below 1e-12: {max(scarti) < 1e-12}")
print(f"    H(p), depends only on the data     : {entropia_dati:.9f} nat")
print(f"    KL(p || q), depends on the model   : {divergenza:.9f} nat")

Output obtained (NumPy 2.3.5):

(a) average negative log-likelihood: 1.053806209 nat
(b) average cross-entropy          : 1.053806209 nat
(c) H(p) + KL(p || q)              : 1.053806209 nat
maximum gap below 1e-12: True
    H(p), depends only on the data     : 1.039720771 nat
    KL(p || q), depends on the model   : 0.014085438 nat

What the code does. Three routes to the same number: the first walks the twelve examples and sums −ln q[y], as a loss would during training; the second works on the three empirical group distributions; the third adds entropy and divergence computed separately. The three values agree to the ninth decimal, with a discrepancy below 1e-12: the identity is not an approximation, it is algebra. The interesting part is the decomposition: of the 1.053806 nats of loss, 1.039721 are entropy of the data and only 0.014085 are the model's fault. Perfect training would bring the loss to 1.039721, not to zero: if you are waiting for zero, you are waiting for something that cannot happen.

What actually goes down when the loss goes down

A three-class model producing logits z and passing them through a softmax has a cross-entropy gradient of suspicious simplicity: ∂H(p, q)/∂z = q − p, the prediction minus the truth. Since H(p) does not depend on z, that is also the gradient of the KL divergence: gradient descent, whatever the step and whatever the optimiser, moves only the KL component. Let us check it by measuring the decomposition on four models of increasing quality, instead of relying on a trajectory that would depend on the step chosen.

import numpy as np

# reuses P and interpola from the previous snippet
def scomposizione(Q):
    h_p = np.mean(-np.sum(P * np.log2(P), axis=1))
    h_pq = np.mean(-np.sum(P * np.log2(Q), axis=1))
    return h_p, h_pq, h_pq - h_p

modelli = {
    "swapped": P[[2, 0, 1]],          # predicts another group's distribution
    "uniform": interpola(P, 0.0),
    "intermediate": interpola(P, 0.5),
    "perfect": interpola(P, 1.0),
}

print(f"{'model':<12}{'H(p)':>10}{'H(p,q)':>11}{'KL(p||q)':>11}")
for nome, previsioni in modelli.items():
    h_p, h_pq, kl = scomposizione(previsioni)
    print(f"{nome:<12}{h_p:>10.6f}{h_pq:>11.6f}{kl:>11.6f}")

gradiente = interpola(P, 0.0)[0] - P[0]
print("gradient q - p on group A, uniform model:", " ".join(f"{g:+.6f}" for g in gradiente))
print(f"sum of the gradient: {gradiente.sum():+.6f}  (stays on the simplex)")

Output obtained:

model             H(p)     H(p,q)   KL(p||q)
swapped       1.500000   1.750000   0.250000
uniform       1.500000   1.584963   0.084963
intermediate  1.500000   1.520321   0.020321
perfect       1.500000   1.500000   0.000000
gradient q - p on group A, uniform model: -0.166667 +0.083333 +0.083333
sum of the gradient: -0.000000  (stays on the simplex)

What the code does. The four models go from the worst — which predicts the distribution of a different group — to the perfect one. The H(p) column is a column of constants, 1.5 bits for all four; H(p,q) falls from 1.75 to 1.5 and KL from 0.25 to 0: the two columns fall by exactly the same amount, because they differ by a constant. When you look at a loss curve going down you are looking at a KL curve shifted upwards by a value no optimiser can touch. The gradient sums to -0.000000, negative zero again: the correction moves mass between classes without ever leaving the simplex.

Common mistakes (symptom, cause, fix)

1. nan after a few epochs. Symptom: the loss is sensible for a few hundred steps, then becomes nan and the weights follow. Cause: log(0), from an underflowing probability or from a logarithm applied to an already computed softmax. Fix: use the fused version of the loss on logits, never on probabilities, and in your own code stay in log_softmax.
2. Bits confused with nats. Symptom: your implementation and the library give different losses, but with a constant ratio. Cause: you use log₂, the library ln; they differ by the factor ln 2 ≈ 0.693147. Fix: declare the unit in the variable name (perdita_nat, guadagno_bit) and convert once only, at the boundary.
3. KL used as a symmetric distance. Symptom: two measures of "distance between distributions" that ought to agree and do not, or a clustering that changes result when you swap the arguments. Cause: KL(p‖q) ≠ KL(q‖p) and the triangle inequality does not hold. Fix: if you need a metric use Jensen-Shannon (its square root is one) or Wasserstein; otherwise choose the direction deliberately.
4. Probabilities passed to a loss that applies the softmax again. Symptom: the loss starts around ln K and falls with inexplicable slowness, accuracy improves but stays below expectations. Cause: the incoming tensor is already normalised and the internal softmax flattens it again. Fix: check that the incoming values can be negative and do not sum to 1; if they sum to 1, you have found the bug.
5. The loss rises while accuracy rises. Symptom: two curves diverging at the end of training, and the temptation to trust the more reassuring one. Cause: not a paradox — accuracy looks at which class has the maximum, cross-entropy at how much probability sits on the right class, and a model that becomes more confident both when right and when wrong improves the former and worsens the latter. Fix: measure calibration, and decide in advance which metric picks the model to save.

The five mistakes are the same family of failures seen from different angles: units, arguments, normalisations applied twice. Four produce wrong but plausible numbers, and only the first announces itself with a nan. That is why it pays to print the floor H(p) and the unit next to the loss: an out-of-scale value is noticed at once.

Best practice

• Declare the unit everywhere. Bits or nats belongs in the variable name and in the chart label: it is the most banal source of numbers that do not agree between two implementations.
• Never materialise the probabilities. Stay in log_softmax from the network to the loss: more stable, faster, and no opportunity to call log on a zero.
• Print the floor before training. Compute H(p) of the empirical label distribution: an untrained model starts around ln K, and the loss will not go below the conditional entropy of the data.
• Choose the direction of the KL in writing. When a specification says "minimise the divergence between the two distributions", ask which one goes on the left: forward and reverse give different models, and on multimodal data the difference is visible.
• Verify every implementation with a second method. A hand-written formula can be checked in thirty seconds against scipy.stats.entropy: in this lesson the same number came out three times by three independent routes.
• Use rational probabilities in your tests. Powers of 1/2 and simple fractions give exact entropies (1.75 bits, 2 bits) verifiable in your head: a test with random numbers says the code runs, not that it is right.

The six rules are all low-cost checks against numbers that do not add up: the first and the sixth avoid confusion of units and of verification, the second and the third stabilise and interpret the loss, the fourth fixes the direction of the KL, the fifth asks for a second independent method. If you have to choose one, choose the fifth.

Minimal glossary

• Surprise (surprisal, self-information): −log p, the information carried by a single outcome of probability p.
• Entropy: H(p) = −Σ pᵢ log pᵢ, the average surprise of a distribution. It lies between 0 and log n.
• Cross-entropy: H(p, q) = −Σ pᵢ log qᵢ, the average cost of using the code optimal for q when reality is p.
• KL divergence (Kullback-Leibler divergence): KL(p‖q) = H(p, q) − H(p) ≥ 0, the waste due to the wrong model. It is not a distance.
• Mutual information: I(X; Y) = H(Y) − H(Y|X) = KL(joint ‖ product), how much observing one variable reduces the uncertainty about the other.
• Jensen's inequality: for φ convex, φ(E[X]) ≤ E[φ(X)]. It is the tool that proves KL ≥ 0 and that generates the ELBO.
• Gibbs' inequality: H(p) ≤ H(p, q) with equality only if q = p. It is KL ≥ 0 rewritten.
• logsumexp: log Σ e^{zᵢ} computed by subtracting the maximum, to avoid the overflow of the exponential. It is the heart of log_softmax.

The eight terms are read in sequence as a chain of definitions: each is built with the previous one, and from the fourth onwards all are measured in the same unit chosen at the start. If an entry does not add up for you, go back to surprise: the whole glossary descends from −log p.

Mini-project

Build a script that measures the "floor" of a dataset with noisy labels and checks that a real loss does not cross it.

1. Generate by hand a synthetic dataset with K = 4 classes and G groups of identical inputs, assigning each group a different label distribution with rational probabilities; compute and print the average conditional entropy H(p) in nats;
2. implement a cross-entropy loss that accepts logits, using logsumexp and never computing the probabilities explicitly; check it against a second, naive implementation on small values, and print the maximum difference;
3. optimise the logits (one vector per group) by any method, printing at every step the loss and the quantity loss − H(p); document what happens when a group contains a single label — the limiting case in which H(p) = 0 and the loss really can tend to zero, with the logits diverging.

The three requests should be carried out in order, because each produces the oracle for the next: first the theoretical floor, then the loss written with logsumexp and checked against the naive one, finally the optimisation, where the difference between loss and floor must fall towards zero without ever crossing it.

Deliverable: a script that prints the theoretical floor, the final loss and the residual KL divergence, plus two lines of comment explaining why the difference between the last two is exactly the floor.

Challenge

• Easy. Compute by hand, in bits, the entropy of a coin with p = 0.9 and that of a fair six-sided die. Which of the two sources "surprises" more on average, and by how much? Then say: is there a single outcome of the coin that surprises more than any outcome of the die?
• Medium. Prove that H(p) ≤ log n for every distribution over n outcomes, with equality only on the uniform one, using Gibbs' inequality instead of Lagrange multipliers. Hint: write KL(p‖u) with u uniform and simplify.
• Hard. Prove Jensen's inequality for n points by induction, writing out every step, and deduce KL(p‖q) ≥ 0. Then characterise the equality case: where exactly is strict convexity needed?
• Expert. Show numerically that for q close to p we have KL(p‖q) ≈ ½·(p−q)ᵀ F (p−q), with F the Fisher information matrix of the categorical family, that is F = diag(1/pᵢ). Take p = (0.5, 0.3, 0.2), perturb it by ε·d with d = (1, −1, 0) and ε ∈ {10⁻¹, 10⁻², 10⁻³}, and check that the ratio between the exact KL and the quadratic form tends to 1 with order ε.
• Enterprise. Take a classification model in production in your organisation. Compute the distribution of the predicted classes over the training period and over the last month, and measure the KL between the two in bits. Choose an alert threshold documenting how you chose it: the defensible route is to estimate the distribution of the KL over historical windows in which nothing happened and set the threshold at a high quantile, not to pick a round number. Also write what this indicator does not detect: drift that leaves the class frequencies unchanged.

The five levels climb from calculation by hand to checking a property on the system you manage: the first two consolidate entropy and Gibbs, the third demands the proof by induction, the fourth connects the KL to the geometry of Fisher information, the last measures a real drift. Do first the ones that make you get the calculations wrong.

The coding exercises for this lesson are in a single file, esercizi.py (about 12 KB): entropy in bits and in nats, the KL that is not a distance and the mutual information that coincides with the information gain of a split. It requires only Python 3.12 or later, with no external dependencies. Run it with python3 esercizi.py: it works even with the parts to complete left empty and tells you which ones are missing.

Recap

Mental map. Surprise = −log p, the only additive function that zeroes out the certain. Entropy H(p) = average surprise, between 0 and log n. Cross-entropy H(p, q) = cost with the wrong code. KL(p‖q) = H(p,q) − H(p) ≥ 0 by Jensen: the waste. Negative log-likelihood = H(p, q) with p one-hot. Information gain = I(Y; X) = H(Y) − H(Y|X).

Cheat sheet.

3 concepts: surprise is −log p because it is the only additive function that zeroes out the certain; cross-entropy is the negative log-likelihood, it does not merely resemble it; KL is a non-negative waste, not a distance. 3 mistakes: log(0) producing nan; bits mistaken for nats with a factor 0.693 of difference; already normalised probabilities passed to a loss that applies the softmax again. 3 best practices: always declare the unit; stay in log_softmax all the way to the loss; print the floor H(p) before you start training. 1 exercise: the mini-project of this lesson.

Next lesson: Bayes applied to parameters, where it turns out that the regularisation you have already used was a belief written in formulae.

Questions to take further

• If KL is not a distance, is there a symmetric quantity that is one and that can be built from it? (The Jensen-Shannon divergence and its square root: Cover and Thomas, chapter 2.)
• Is cross-entropy the right loss even when the classes are heavily imbalanced, or is it better to weight it? (Lesson on classification metrics, module 03.)
• What happens to the KL divergence when the distributions are continuous and their supports do not overlap? (It is the problem that motivates the Wasserstein distance in generative adversarial networks, module 07.)

The three questions are hooks to the following lessons: the first leads to Jensen-Shannon, the second to the metrics for imbalanced classes, the third to continuous distributions with disjoint supports. They do not have to be solved now; they serve to decide what to read next.

The same series also includes: 26 - Constrained Optimisation: Lagrange Multipliers, Duality and the KKT Conditions (the previous lesson), 28 - Bayesian Statistics: Priors, Posteriors, Conjugates and Why Regularisation Is a Belief (the next lesson), 95 - Attention, Derived in Full: Query, Key, Value and the Transformer Block Written From Scratch (which refers back to this lesson).