The classification loss measures how surprised the model is
When you train a classifier with cross-entropy, on each example the model pays -\ln q_y, where q_y is the probability it assigned to the right class: if it gives it 90% it pays 0.105, if it gives it 10% it pays 2.30. The loss is the logarithm of the surprise, and understanding why it is precisely the logarithm, and what ties it to entropy, KL divergence and the perplexity of language models, keeps you from using these quantities as formulas to copy. There is a single thread: the information a model lacks to describe the data is measured in extra bits (or nats) of code compared with the minimum possible.
Notation, following that of the article on probability and statistics: p is the true distribution, q the model's, H the entropy. The logarithm is base 2 for bits and natural for nats; the book by Goodfellow, Bengio and Courville always uses the natural one, and it is the convention of all deep learning libraries. Going from one unit to the other is a multiplication: 1 nat = 1/\ln 2 \approx 1.4427 bits.
Information of an event: surprise as the logarithm of the probability
The information of an event of probability P(x) is I(x) = -\log_2 P(x): the larger the rarer the event. A head on a fair coin is worth 1 bit, a 4 on a fair die \log_2 6 \approx 2.585 bits, a one-in-1024 event exactly 10 bits, a 99% event only 0.0145 bits. The logarithm is not an arbitrary choice: two independent events have probability P(x)P(y) and the logarithm turns the product into a sum, -\log(P(x)P(y)) = I(x) + I(y), so two heads in a row carry 1 + 1 = 2 bits. Shannon (1948) defines the bit with this base: N two-state devices store N bits because \log_2 2^N = N.
import numpy as np
def surprise_bits(p):
return -np.log2(p)
for name, p in [("fair coin: heads", 1 / 2), ("fair die: a 4", 1 / 6), ("1-in-1024 event", 1 / 1024), ("almost certain event", 0.99)]:
print(f"{name:22s} P = {p:.4f} -> {surprise_bits(p):.4f} bits")
print("two independent tosses (heads, heads):", surprise_bits(1 / 2 * 1 / 2), "bits = 1 + 1")
print("1 nat in bits:", round(1 / np.log(2), 4))
# Output:
# fair coin: heads P = 0.5000 -> 1.0000 bits
# fair die: a 4 P = 0.1667 -> 2.5850 bits
# 1-in-1024 event P = 0.0010 -> 10.0000 bits
# almost certain event P = 0.9900 -> 0.0145 bits
# two independent tosses (heads, heads): 2.0 bits = 1 + 1
# 1 nat in bits: 1.4427
The listing is a direct computation of -\log_2 p for four events, and it shows that surprise grows slowly: going from probability 1/2 to 1/1024 multiplies the rarity by 512 but the information only by ten. The last line verifies the additive property and the conversion factor between nats and bits, which will be needed when we compare the numbers of a run in nats with the hand ones in bits.
Entropy: the average surprise of a distribution
Entropy is the expected value of the information, that is, the average surprise before the outcome is known:
with the convention 0\log 0 = 0. A few hand cases. A fair coin: H = -2 \cdot \tfrac{1}{2}\log_2\tfrac{1}{2} = 1 bit. A coin with 99% heads: -0.99\log_2 0.99 - 0.01\log_2 0.01 \approx 0.0144 + 0.0664 = 0.0808 bits: almost no uncertainty. A fair die: \log_2 6 \approx 2.585 bits. A die where the 3 comes up half the time and the other five numbers 10% each: 0.5 \cdot 1 + 5 \cdot 0.1 \cdot 3.3219 \approx 2.161 bits, less than the fair die.
Entropy has an operational meaning: it is the minimum number of bits per symbol, on average, of a lossless code. Let p = (1/2, 1/4, 1/8, 1/8) over the symbols a,b,c,d. The prefix code 0, 10, 110, 111 has lengths 1, 2, 3, 3 and average length \tfrac{1}{2} + \tfrac{2}{4} + \tfrac{3}{8} + \tfrac{3}{8} = 1.75 bits, which coincides with H(p). A code that ignores the frequencies and uses 2 bits per symbol costs 2 bits per symbol: 0.25 bits more, which is exactly the divergence we will define shortly.
import numpy as np
def entropy_bits(p):
p = np.asarray(p, dtype=float)
p = p[p > 0] # by convention 0 * log 0 = 0
return -np.sum(p * np.log2(p))
print(f"fair coin H = {entropy_bits([0.5, 0.5]):.4f} bits")
print(f"coin with P(H) = 0.99 H = {entropy_bits([0.99, 0.01]):.4f} bits")
print(f"fair die H = {entropy_bits([1 / 6] * 6):.4f} bits (log2 6 = {np.log2(6):.4f})")
print(f"die with a 3 at 50% H = {entropy_bits([0.1, 0.1, 0.5, 0.1, 0.1, 0.1]):.4f} bits")
# operational meaning: average length of a prefix code
p = np.array([1 / 2, 1 / 4, 1 / 8, 1 / 8])
code_len = np.array([1, 2, 3, 3]) # code 0, 10, 110, 111
print(f"\nH(p) = {entropy_bits(p):.4f} bits; average code length (0, 10, 110, 111) = {np.sum(p * code_len):.4f} bits")
print(f"uniform code with 2 bits per symbol: average length = {np.sum(p * 2):.4f} bits = H(p) + {np.sum(p * 2) - entropy_bits(p):.4f}")
# Output:
# fair coin H = 1.0000 bits
# coin with P(H) = 0.99 H = 0.0808 bits
# fair die H = 2.5850 bits (log2 6 = 2.5850)
# die with a 3 at 50% H = 2.1610 bits
#
# H(p) = 1.7500 bits; average code length (0, 10, 110, 111) = 1.7500 bits
# uniform code with 2 bits per symbol: average length = 2.0000 bits = H(p) + 0.2500
The listing repeats the hand calculations and compares them: the entropy of the rigged coin is 0.0808 bits, that of the rigged die 2.1610, and the average length of the prefix code is 1.7500, equal to the entropy. The line p = p[p > 0] implements the convention 0\log 0 = 0, without which np.log2(0) would give -\infty and the product nan. There are three properties to keep:
- H(X) \geq 0 always, with H(X) = 0 only when one outcome has probability 1
- H(X) is maximal, equal to \log K, when the K outcomes are equiprobable
- for continuous variables the formula with the integral (differential entropy) loses some of these properties, for example it can be negative
The last item is a warning that gets forgotten: everything that holds for discrete entropy, such as being always non-negative, does not automatically carry over to the continuous case (the book by Goodfellow and colleagues notes this for the information of an event with unit density). The first two say that entropy measures how much uncertainty remains: none if the outcome is already decided, maximal when no outcome is favored.
Cross-entropy: the cost of using the wrong code
The cross-entropy H(p, q) = -\sum_x p(x)\log q(x) is the average code length when the data come from p but the code is built for q. In the example above the uniform code corresponds to q = (1/4, 1/4, 1/4, 1/4) and H(p, q) = 2 bits against H(p) = 1.75. In classification p is the one-hot label (probability 1 on the true class y) and q is the softmax output: the sum reduces to a single term, L = -\log q_y. For two classes it becomes the binary cross-entropy -[y\log \hat y + (1-y)\log(1-\hat y)], the same one that in the article on probability coincided with the negative log-likelihood of a Bernoulli.
Hand calculations: if the true class receives 0.9 the loss is -\ln 0.9 = 0.1054 nat (0.152 bits); with 0.5 it is 0.6931 nat, exactly 1 bit; with 0.1 it is 2.3026 nat (3.32 bits). With three logits z = (2, 1, 0.1) and true class 0, the softmax assigns q_0 \approx 0.659 and the loss is -\ln 0.659 = 0.417; it can be computed without going through the probabilities as \ln\sum_j e^{z_j} - z_0 = \ln 11.21 - 2 = 0.417. The gradient with respect to the logits is surprisingly simple: q - p = (0.659 - 1;\ 0.2424;\ 0.0986).
import numpy as np
z = np.array([2.0, 1.0, 0.1]) # logits; the true class is 0
y = 0
q = np.exp(z) / np.exp(z).sum() # softmax
print("softmax:", np.round(q, 4))
print(f"-log q[0] = {-np.log(q[y]):.4f} nat")
print(f"log(sum(exp(z))) - z[0] = {np.log(np.exp(z).sum()) - z[y]:.4f} nat")
for qy in (0.9, 0.5, 0.1):
print(f"probability assigned to the true class {qy}: loss {-np.log(qy):.4f} nat = {-np.log2(qy):.4f} bits")
def ce(z, y): # cross-entropy with one-hot = log-sum-exp - z[y]
m = z.max()
return m + np.log(np.exp(z - m).sum()) - z[y]
onehot = np.eye(3)[y]
num = np.array([(ce(z + e, y) - ce(z - e, y)) / 2e-6 for e in 1e-6 * np.eye(3)])
print("numerical gradient dL/dz :", np.round(num, 4))
print("softmax - one-hot :", np.round(q - onehot, 4))
big = np.array([1000.0, 0.0, 0.0])
print("large logits, stable form:", ce(big, 0), " naive form:", -np.log(np.exp(big[0]) / np.exp(big).sum()))
# Output:
# softmax: [0.659 0.2424 0.0986]
# -log q[0] = 0.4170 nat
# log(sum(exp(z))) - z[0] = 0.4170 nat
# probability assigned to the true class 0.9: loss 0.1054 nat = 0.1520 bits
# probability assigned to the true class 0.5: loss 0.6931 nat = 1.0000 bits
# probability assigned to the true class 0.1: loss 2.3026 nat = 3.3219 bits
# numerical gradient dL/dz : [-0.341 0.2424 0.0986]
# softmax - one-hot : [-0.341 0.2424 0.0986]
# large logits, stable form: 0.0 naive form: nan
The listing does four things. It shows that the two ways of writing the loss coincide; it lists the cost of a confident prediction (0.9), an uncertain one (0.5) and a wrong one (0.1) in nats and bits; it compares the gradient computed with finite differences against the formula q - p and finds them equal to the fourth digit, a result that the discussion on loss functions uses to explain why cross-entropy with softmax does not suffer from the vanishing gradient of squared loss with sigmoid, described in the article on differential calculus. Finally it shows why libraries compute the loss directly from the logits: with a logit of 1000 the naive softmax overflows (nan), while subtracting the maximum before exponentiating gives the correct value, 0. The np.clip(q, 1e-15, ...) cut often seen in tutorials avoids the log(0) but is a band-aid: it does not solve the overflow and distorts extreme values.
KL divergence: how many extra bits the approximation costs
The Kullback–Leibler divergence measures the gap between the cross-entropy and the entropy:
The code with wrong probabilities costs D_{\mathrm{KL}} bits more than the optimal code. It is always non-negative (Gibbs' inequality) and zero only if p = q. It is not symmetric, so it is not a distance in the strict sense (Goodfellow and colleagues observe this explicitly), and if q assigns probability zero where p assigns a positive one it equals +\infty. Since H(p) does not depend on the model, minimizing the cross-entropy with respect to q is equivalent to minimizing the KL, and it is also equivalent to maximizing the log-likelihood of the data, because with N samples the cross-entropy against the empirical distribution is the mean negative log-likelihood.
Numbers: p uniform over four classes and q = (0.3;\ 0.2;\ 0.3;\ 0.2) give D_{\mathrm{KL}}(p\|q) = 0.25\,[2\ln(0.25/0.3) + 2\ln(0.25/0.2)] = 0.25\,[2(-0.1823) + 2(0.2231)] \approx 0.0204 nat. The cross-entropy is 1.4067 nat and H(p) = \ln 4 = 1.3863: the difference is precisely 0.0204. In generative models with latent variables (VAE) a closed-form term appears, the KL between \mathcal{N}(\mu, \sigma^2) and \mathcal{N}(0, 1): \tfrac{1}{2}(\mu^2 + \sigma^2 - \ln\sigma^2 - 1). By hand: with \mu = 1 and \sigma^2 = 1 it equals 0.5, with \mu = 0 and \sigma^2 = 1 it is zero, because the two distributions coincide.
import numpy as np
from scipy import integrate
def kl(p, q):
p, q = np.asarray(p, float), np.asarray(q, float)
if np.any((q == 0) & (p > 0)):
return np.inf # p puts mass where q puts none
m = p > 0
return np.sum(p[m] * np.log(p[m] / q[m]))
p = np.full(4, 0.25)
q = np.array([0.3, 0.2, 0.3, 0.2])
print(f"KL(p||q) = {kl(p, q):.5f} nat")
print(f"KL(q||p) = {kl(q, p):.5f} nat (not symmetric)")
H = lambda a: -np.sum(a * np.log(a))
CE = -np.sum(p * np.log(q))
print(f"H(p) + KL(p||q) = {H(p) + kl(p, q):.5f} H(p, q) = {CE:.5f}")
a, b = np.array([0.5, 0.5, 0.0]), np.full(3, 1 / 3)
print(f"different supports: KL(a||b) = {kl(a, b):.4f} (ln 1.5 = {np.log(1.5):.4f}) KL(b||a) = {kl(b, a)}")
# KL between N(mu, s2) and N(0, 1): closed formula against numerical integration
mu, s2 = 0.5, 0.6
closed = 0.5 * (mu ** 2 + s2 - np.log(s2) - 1)
pdf = lambda x, m, v: np.exp(-(x - m) ** 2 / (2 * v)) / np.sqrt(2 * np.pi * v)
numeric = integrate.quad(lambda x: pdf(x, mu, s2) * np.log(pdf(x, mu, s2) / pdf(x, 0, 1)), -20, 20)[0]
print(f"KL(N({mu}, {s2}) || N(0, 1)): formula = {closed:.6f}, integral = {numeric:.6f}")
print("case mu = 1, s2 = 1:", 0.5 * (1 + 1 - 0 - 1), " case mu = 0, s2 = 1:", 0.5 * (0 + 1 - 0 - 1))
# Output:
# KL(p||q) = 0.02041 nat
# KL(q||p) = 0.02014 nat (not symmetric)
# H(p) + KL(p||q) = 1.40671 H(p, q) = 1.40671
# different supports: KL(a||b) = 0.4055 (ln 1.5 = 0.4055) KL(b||a) = inf
# KL(N(0.5, 0.6) || N(0, 1)): formula = 0.180413, integral = 0.180413
# case mu = 1, s2 = 1: 0.5 case mu = 0, s2 = 1: 0.0
The listing implements the KL with an explicit check: if q = 0 where p > 0 it returns inf instead of hiding the case behind a mask, as many implementations do, and the case with different supports shows the behavior: D_{\mathrm{KL}}(a\|b) = \ln 1.5 = 0.4055 while D_{\mathrm{KL}}(b\|a) = \infty. For Gaussians it compares the closed formula with the numerical integral of scipy.integrate.quad on one example (\mu = 0.5, \sigma^2 = 0.6): the two outputs agree to the sixth digit. This article does not go into the construction of VAEs, which is a topic of its own; the formula is here because it is a good case where the KL is computed by hand and verified.
Mutual information: how much one variable says about another
Mutual information I(X;Y) = \sum_{x,y} P(x,y)\log\frac{P(x,y)}{P(x)P(y)} is the KL between the joint distribution and the product of the marginals, that is, how far the joint moves away from independence. It equals H(X) - H(X \mid Y): the reduction of uncertainty about X obtained by knowing Y. It is zero if and only if X and Y are independent. Example: two binary variables with joint \begin{pmatrix} 0.4 & 0.1 \\ 0.1 & 0.4 \end{pmatrix} have marginals (0.5; 0.5) and I = 2 \cdot 0.4\ln(0.4/0.25) + 2 \cdot 0.1\ln(0.1/0.25) = 0.376 - 0.1833 \approx 0.1927 nat, that is 0.278 bits.
import numpy as np
from sklearn.metrics import mutual_info_score
P = np.array([[0.4, 0.1],
[0.1, 0.4]]) # joint distribution P(x, y)
Px, Py = P.sum(axis=1), P.sum(axis=0)
mi = np.sum(P * np.log(P / np.outer(Px, Py)))
Hx = -np.sum(Px * np.log(Px))
Hx_given_y = -np.sum(P * np.log(P / Py)) # H(X|Y) = -sum P(x,y) log P(x|y)
print(f"I(X;Y) = {mi:.5f} nat = {mi / np.log(2):.4f} bits")
print(f"H(X) - H(X|Y) = {Hx - Hx_given_y:.5f} nat (H(X) = {Hx:.4f}, H(X|Y) = {Hx_given_y:.4f})")
print("scikit-learn:", round(mutual_info_score(None, None, contingency=P * 100), 5))
# Output:
# I(X;Y) = 0.19274 nat = 0.2781 bits
# H(X) - H(X|Y) = 0.19274 nat (H(X) = 0.6931, H(X|Y) = 0.5004)
# scikit-learn: 0.19274
The block computes mutual information in two ways, as a KL and as a difference of entropies (H(X) = \ln 2 = 0.6931 minus H(X \mid Y) = 0.5004), compares them with scikit-learn's mutual_info_score and finds the same number. In practice the direct calculation requires a table of joint probabilities: with continuous variables or many categories it must be estimated from the data, and the estimates are biased with few samples. It is the tool behind feature selection by mutual information and the evaluation of a clustering against reference labels.
Perplexity: cross-entropy read as a number of choices
The perplexity of a language model on N tokens is the exponential of the mean cross-entropy per token, \mathrm{PPL} = \exp\left(-\frac{1}{N}\sum_i \ln q(w_i \mid w_{<i})\right). It reads as the number of equiprobable choices that would give the same uncertainty: a uniform model over k options has perplexity k. Example: if the model assigns to the three observed tokens the probabilities 0.5; 0.25; 0.25, the mean cross-entropy is 1.1552 nat and the perplexity is \exp(1.1552) = 3.175, which is also (1/(0.5 \cdot 0.25 \cdot 0.25))^{1/3} = 32^{1/3}.
import numpy as np
probs = np.array([0.5, 0.25, 0.25]) # probabilities the model assigns to the 3 observed tokens
ce = -np.mean(np.log(probs))
print(f"mean cross-entropy = {ce:.4f} nat perplexity = exp(ce) = {np.exp(ce):.4f}")
print(f"check: (1 / product)^(1/3) = {(1 / np.prod(probs)) ** (1 / 3):.4f} (32^(1/3))")
for k in (2, 10, 50_000):
print(f"uniform model over {k:6d} options: perplexity = {np.exp(-np.log(1 / k)):.1f}")
# Output:
# mean cross-entropy = 1.1552 nat perplexity = exp(ce) = 3.1748
# check: (1 / product)^(1/3) = 3.1748 (32^(1/3))
# uniform model over 2 options: perplexity = 2.0
# uniform model over 10 options: perplexity = 10.0
# uniform model over 50000 options: perplexity = 50000.0
The listing verifies the equivalence between the two ways of writing it and that a uniform model over 50000 options has perplexity 50000, the starting point of a model that has not yet learned anything about a vocabulary of that size. A lower perplexity is better, but the values are comparable only between models evaluated on the same data and with the same way of splitting the text into tokens: changing the tokenizer changes the meaning of «per token», and the comparison becomes meaningless.
What you can do now
You can compute by hand the surprise of an event, the entropy of a distribution and the length of a code that achieves it, and you can write the cross-entropy of a classifier as -\ln q_y and its gradient as q - p. You know why minimizing the cross-entropy, minimizing the KL and maximizing the likelihood are the same problem, and you can avoid the two tutorial traps: the log(0) hidden by a mask or a clip, and the comparison of perplexity between models with different tokenizers. The next article, on PCA, goes back to linear algebra: the same idea of «how much information remains» becomes how much variance is kept when dimensions are discarded.
How I checked the listings: I ran each one on its own on 2026-11-21 with Python 3.14.4, NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.8.0, and the decimal figures quoted in the text match the ones produced. With other library versions the last digits may change.
Sources
- Goodfellow, Bengio, Courville, Deep Learning, chapter 3 (§3.13 Information Theory) - use of the natural logarithm and of nat units, definition of entropy, cross-entropy and KL, asymmetry of the KL; chapter read, accessed on 2026-11-12
- Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, 1948 (copy of the paper) - definition of the bit as a unit with logarithm base 2; copy reached, excerpt read, accessed on 2026-11-12
- Kullback, Leibler, On Information and Sufficiency, Annals of Mathematical Statistics 22, 1951 - original paper of the divergence that bears their name; page reached, text not read, accessed on 2026-11-12
- scikit-learn: sklearn.metrics.mutual_info_score - definition of mutual information used for the numerical comparison, accessed on 2026-11-12
Commenti
Caricamento commenti...
Sign in to comment