The loss decides what "wrong" means
The model does not know what you want: it only knows how to reduce the number you give it. If the loss treats an error of one euro and one of a thousand in the same way, the model cannot tell a rounding from a disaster; if it weighs a rare case and a common one equally, on imbalanced data the common one wins. This is why the choice of loss is not an implementation detail: it defines the problem the model is really solving. The article goes through the losses you need most often, each with its formula, a hand-worked calculation and a NumPy check, and ends with how to write your own without getting the gradient wrong.
Notation: y is the true value or class, \hat y the prediction, z the logit (the number before the sigmoid or the softmax), n the number of samples. The link with what came before is close: cross-entropy is the loss derived in the article on information theory, its relation to maximum likelihood is in the article on probability and statistics, and the gradients are those of differential calculus.
Regression: MSE, MAE and Huber
The mean squared error L_{\text{MSE}} = \frac{1}{n}\sum_i (y_i - \hat y_i)^2 penalises quadratically, and its derivative with respect to the prediction, \frac{2}{n}(\hat y_i - y_i), is proportional to the error: the more wrong you are, the harder the gradient pushes. The mean absolute error L_{\text{MAE}} = \frac{1}{n}\sum_i |y_i - \hat y_i| has a gradient of constant magnitude (it does not slow down as it approaches the minimum) and is not differentiable at zero. The Huber loss with threshold \delta is quadratic for small errors and linear for large ones: \frac{1}{2}e^2 if |e| \leq \delta, otherwise \delta|e| - \frac{1}{2}\delta^2. The two branches join at |e| = \delta with the same value and the same slope.
Hand calculation on five points with errors 0.1; 0.2; 0.2; 0.1; 0.1: MSE = (0.01 + 0.04 + 0.04 + 0.01 + 0.01)/5 = 0.022, MAE = 0.7/5 = 0.14, Huber with \delta = 1 equal to half the MSE, 0.011, because all the errors lie on the quadratic branch. Adding a sixth point with error 14: MSE = (0.11 + 196)/6 \approx 32.685, MAE = (0.7 + 14)/6 = 2.45, Huber = (0.055 + 13.5)/6 \approx 2.259, where 13.5 = 1 \cdot 14 - \tfrac{1}{2}.
import numpy as np
mse = lambda y, p: np.mean((y - p) ** 2)
mae = lambda y, p: np.mean(np.abs(y - p))
def huber(y, p, delta=1.0):
e = np.abs(y - p)
return np.mean(np.where(e <= delta, 0.5 * e ** 2, delta * e - 0.5 * delta ** 2))
y = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
p = np.array([1.1, 2.2, 2.8, 4.1, 4.9]) # errors 0.1, 0.2, 0.2, 0.1, 0.1
print(f"no outlier: MSE {mse(y, p):.4f} MAE {mae(y, p):.4f} Huber {huber(y, p):.4f}")
y2, p2 = np.append(y, 6.0), np.append(p, 20.0) # an error of 14
print(f"with outlier: MSE {mse(y2, p2):.4f} MAE {mae(y2, p2):.4f} Huber {huber(y2, p2):.4f}")
# which constant minimises each loss? (data with an anomalous value)
d = np.array([1.0, 2.0, 3.0, 4.0, 100.0])
cs = np.linspace(0, 110, 1101)
best_mse = cs[np.argmin([mse(d, c) for c in cs])]
best_mae = cs[np.argmin([mae(d, c) for c in cs])]
print(f"MSE minimum at c = {best_mse:.1f} (mean = {d.mean()}); MAE minimum at c = {best_mae:.1f} (median = {np.median(d)})")
# Output:
# no outlier: MSE 0.0220 MAE 0.1400 Huber 0.0110
# with outlier: MSE 32.6850 MAE 2.4500 Huber 2.2592
# MSE minimum at c = 22.0 (mean = 22.0); MAE minimum at c = 3.0 (median = 3.0)
The listing confirms the three calculations. A single large error raises the MSE by three orders of magnitude (from 0.022 to 32.7) and the MAE and Huber by only one: the MSE is dominated by outliers. The last part shows the statistical meaning of the choice: among all the constant values c you could predict for five data points with one anomalous value (1, 2, 3, 4, 100), the MSE is minimal at the mean, 22, the MAE at the median, 3. With MSE the model estimates what the response is worth on average; with MAE, what it is worth typically. If the outliers are measurement errors, the median is more appropriate; if they are real and costly cases, the mean is what you care about. The MSE is the negative log-likelihood of a model with Gaussian noise, as Deisenroth and colleagues show in chapter 9 for linear regression.
Classification: binary and multiclass cross-entropy
With a sigmoid \hat y = \sigma(z), the binary cross-entropy is L = -[y\ln\hat y + (1-y)\ln(1-\hat y)]. Its gradient with respect to the logit simplifies to \partial L/\partial z = \hat y - y: the sigmoid and the logarithm cancel, and the signal does not fade when the model is confident and wrong. That is the difference from the squared error on a sigmoid, whose gradient carries the factor \sigma'(z) that vanishes at saturation, as in the article on differential calculus. The multiclass case with softmax is identical: with a one-hot label, L = -\ln\hat y_c and the gradient with respect to the logits is \hat{\mathbf y} - \mathbf y, as seen in the article on information theory. By hand, with z = 0.8 and y = 1: \hat y = \sigma(0.8) \approx 0.69, L = -\ln 0.69 \approx 0.371 and gradient 0.69 - 1 = -0.31.
import numpy as np
sigmoid = lambda z: 1 / (1 + np.exp(-z))
def bce_logit(z, y): # stable: max(z,0) - z*y + log(1 + exp(-|z|))
return np.maximum(z, 0) - z * y + np.log1p(np.exp(-np.abs(z)))
z, y = 0.8, 1.0
q = sigmoid(z)
print(f"y_hat = {q:.4f} loss = {-np.log(q):.4f} (stable: {bce_logit(z, y):.4f})")
e = 1e-6
print(f"numerical dL/dz = {(bce_logit(z + e, y) - bce_logit(z - e, y)) / (2 * e):.5f} y_hat - y = {q - y:.5f}")
zz = 40.0 # a large logit, y = 0: a confident wrong prediction
with np.errstate(divide="ignore"):
naive = -(0 * np.log(sigmoid(zz)) + 1 * np.log(1 - sigmoid(zz)))
print(f"y = 0, z = 40: naive form = {naive} stable form = {bce_logit(zz, 0.0):.1f} (z - 0 = {zz})")
# Output:
# y_hat = 0.6900 loss = 0.3711 (stable: 0.3711)
# numerical dL/dz = -0.31003 y_hat - y = -0.31003
# y = 0, z = 40: naive form = inf stable form = 40.0 (z - 0 = 40.0)
The listing checks two things. The gradient computed with finite differences coincides with \hat y - y to the fifth digit. And it shows a problem that tutorials often hide: with a logit of 40 and y = 0 (a confident, wrong prediction) the naive form -log(1 - sigmoid(z)) gives inf, because \sigma(40) in floating point is exactly 1. The stable form, \max(z, 0) - zy + \ln(1 + e^{-|z|}), never goes through the sigmoid and returns 40, the correct value (the loss grows linearly with the logit, it does not explode). This is why libraries offer a version of the loss that takes the logits, not the probabilities, as its argument.
Imbalanced data: focal loss
In a problem with very few positives and a great many easy negatives, the sum of the losses is dominated by the negatives: each contributes little, but there are many of them. The focal loss (Lin et al., 2017) multiplies the cross-entropy by (1 - p_t)^\gamma, where p_t is the probability assigned to the true class: L_{\text{FL}} = -(1 - p_t)^\gamma\ln p_t. For \gamma = 2 an example with p_t = 0.9 is weighted 0.1^2 = 0.01 (a hundred times less), one with p_t = 0.5 is weighted 0.25, one with p_t = 0.1 is weighted 0.81. With \gamma = 0 you recover the cross-entropy. In the original paper, on object detection, \gamma = 2 works best in their experiments, and the optimal balancing factor \alpha was 0.25 with \gamma = 2: values specific to that problem, not universal.
import numpy as np
gamma = 2.0
for pt in (0.9, 0.5, 0.1):
w = (1 - pt) ** gamma
print(f"p_t = {pt}: CE = {-np.log(pt):.4f} factor (1-p_t)^2 = {w:.2f} focal = {w * -np.log(pt):.4f}")
# 1000 easy negatives (p_t = 0.95) and 10 hard positives (p_t = 0.1)
pt = np.concatenate([np.full(1000, 0.95), np.full(10, 0.1)])
ce = -np.log(pt)
fl = (1 - pt) ** gamma * ce
for name, l in (("cross-entropy", ce), ("focal gamma=2", fl)):
print(f"{name:14s} total = {l.sum():7.3f} share of the 1000 easy = {l[:1000].sum() / l.sum():.1%} share of the 10 hard = {l[1000:].sum() / l.sum():.1%}")
# check: with gamma = 0 the focal loss coincides with the cross-entropy
print("gamma = 0 -> identical to CE:", np.allclose((1 - pt) ** 0 * ce, ce))
# Output:
# p_t = 0.9: CE = 0.1054 factor (1-p_t)^2 = 0.01 focal = 0.0011
# p_t = 0.5: CE = 0.6931 factor (1-p_t)^2 = 0.25 focal = 0.1733
# p_t = 0.1: CE = 2.3026 factor (1-p_t)^2 = 0.81 focal = 1.8651
# cross-entropy total = 74.319 share of the 1000 easy = 69.0% share of the 10 hard = 31.0%
# focal gamma=2 total = 18.779 share of the 1000 easy = 0.7% share of the 10 hard = 99.3%
# gamma = 0 -> identical to CE: True
The listing simulates the typical case, with 1000 easy negatives (p_t = 0.95) and 10 hard positives (p_t = 0.1). With the cross-entropy, the thousand easy ones account for 69.0\% of the total loss and the ten hard ones for 31.0\%: the gradient is driven by the cases the model already knows how to solve. With the focal loss the easy ones drop to 0.7\% and the hard ones rise to 99.3\%. The practical consequence is that the loss is no longer the negative log-likelihood of the data; as an alternative, to correct the imbalance there are class weights in the cross-entropy, which do not change the shape of the loss, and the choice between the two routes should be made with a metric on the problem (precision, recall) and not with the loss itself.
Margin losses and losses for embeddings: hinge, contrastive and triplet
Some losses do not measure a probability but a distance or a margin. The hinge loss of SVMs, \max(0, 1 - yf(x)) with y \in \{-1, +1\}, is zero when the point is correctly classified with a margin of at least 1: for y = +1 and score 1.5 the loss is 0, for 0.3 it is 0.7, for -0.5 it is 1.5. The contrastive loss works on pairs of embeddings with distance d: for similar pairs it is \frac{1}{2}d^2 (it pulls them together), for different pairs \frac{1}{2}\max(0, m - d)^2 (it pushes them apart up to the margin m). The triplet loss uses triplets (anchor, positive, negative): \max(0, \|f(a) - f(p)\|^2 - \|f(a) - f(n)\|^2 + m) asks that the positive be closer to the anchor than the negative by at least m in squared distance.
import numpy as np
hinge = lambda y, f: np.maximum(0, 1 - y * f)
print("hinge, y = +1:", {f: float(hinge(1, f)) for f in (1.5, 0.3, -0.5)})
def contrastive(d, similar, m=1.0): # similar: 1 if the pair is similar
return 0.5 * d ** 2 if similar else 0.5 * np.maximum(0, m - d) ** 2
print("contrastive, similar pair d = 0.4:", contrastive(0.4, True))
print("contrastive, different pair d = 0.4 / d = 1.2 (m = 1):", contrastive(0.4, False), contrastive(1.2, False))
def triplet(a, p, n, m=1.0):
return max(0.0, np.sum((a - p) ** 2) - np.sum((a - n) ** 2) + m)
a, p = np.array([0.0, 0.0]), np.array([1.0, 0.0])
print("triplet, far negative (0, 2):", triplet(a, p, np.array([0.0, 2.0])))
print("triplet, near negative (0, 1):", triplet(a, p, np.array([0.0, 1.0])))
# Output:
# hinge, y = +1: {1.5: 0.0, 0.3: 0.7, -0.5: 1.5}
# contrastive, similar pair d = 0.4: 0.08000000000000002
# contrastive, different pair d = 0.4 / d = 1.2 (m = 1): 0.18 0.0
# triplet, far negative (0, 2): 0.0
# triplet, near negative (0, 1): 1.0
The values in the listing can be checked by hand. Contrastive with margin 1 and d = 0.4: similar pair \frac{1}{2}\cdot 0.16 = 0.08; different pair \frac{1}{2}(1 - 0.4)^2 = 0.18; with d = 1.2 the different pair is already beyond the margin and costs zero. Triplet with anchor (0,0), positive (1,0) and negative (0,2): 1 - 4 + 1 = -2, so loss 0 (the triplet is already satisfied); with negative (0,1): 1 - 1 + 1 = 1. The fact that many triplets contribute zero is why, in real systems, the choice of the hard triplets to learn from is what matters.
Temperature: changing the confidence without changing the answer
Dividing the logits by a temperature T > 0 before the softmax, \text{softmax}(z_k/T), does not change which class has the maximum probability, but changes its confidence: T > 1 flattens the distribution, T < 1 makes it sharper. For z = (2, 1, 0.1) class 0 gets 0.864 with T = 0.5, 0.659 with T = 1, 0.502 with T = 2. Guo and colleagues (2017) show that modern deep networks are often poorly calibrated, that is, overconfident, and that temperature scaling, a single-parameter variant of Platt scaling, is surprisingly effective at correcting it. The parameter T is searched on a validation set by minimising the negative log-likelihood, and it does not change the accuracy.
import numpy as np
softmax = lambda z, T=1.0: np.exp(z / T) / np.exp(z / T).sum()
z = np.array([2.0, 1.0, 0.1])
for T in (0.5, 1.0, 2.0):
print(f"T = {T}: {np.round(softmax(z, T), 4)} predicted class {softmax(z, T).argmax()}")
# calibration: synthetic data with an overconfident classifier
rng = np.random.default_rng(0)
n, K = 5000, 3
y = rng.integers(0, K, n)
signal = np.eye(K)[y] * 1.5 + rng.normal(0, 1.0, (n, K)) # weak hint about the true class
logits = 4.0 * signal # inflated logits: overconfident
def probs(T):
s = logits / T
e = np.exp(s - s.max(axis=1, keepdims=True))
return e / e.sum(axis=1, keepdims=True)
nll = lambda T: -np.mean(np.log(probs(T)[np.arange(n), y]))
conf = lambda T: probs(T).max(axis=1).mean()
acc = np.mean(logits.argmax(1) == y)
Ts = np.linspace(0.5, 10, 951)
T_best = Ts[np.argmin([nll(T) for T in Ts])]
print(f"accuracy = {acc:.3f} (does not depend on T)")
print(f"T = 1.00: NLL = {nll(1.0):.3f} mean confidence = {conf(1.0):.3f}")
print(f"T = {T_best:.2f}: NLL = {nll(T_best):.3f} mean confidence = {conf(T_best):.3f}")
# Output:
# T = 0.5: [0.8638 0.1169 0.0193] predicted class 0
# T = 1.0: [0.659 0.2424 0.0986] predicted class 0
# T = 2.0: [0.5017 0.3043 0.194 ] predicted class 0
# accuracy = 0.766 (does not depend on T)
# T = 1.00: NLL = 0.834 mean confidence = 0.906
# T = 2.64: NLL = 0.571 mean confidence = 0.763
The second part of the listing is synthetic: it generates a three-class classifier with inflated logits, which guesses the class in 76.6% of cases but declares a mean confidence of 90.6%. The search for the T that minimises the negative log-likelihood finds T \approx 2.64, for which the loss falls from 0.834 to 0.571 and the mean confidence from 0.906 to 0.763, almost equal to the accuracy. The result depends on the way the data were built, it is not a measurement of a real network; but it shows the mechanism: calibration is a problem of logit scale, not of classification, and it must be done on data not used for training.
Writing your own loss, and checking its gradient
When the standard losses do not represent the real cost, for example because underestimating a demand costs more than overestimating it, you write your own. The risk is the gradient: with an automatic differentiation framework you get it for free, but if you write it by hand (or use a loss that is not differentiable at some point) a single wrong sign is enough to train in the opposite direction. The example is a squared error that weighs underestimates k = 3 times more, e = \hat y - y < 0: L = \frac{1}{n}\sum w_i e_i^2 with w_i = 3 if e_i < 0 and 1 otherwise. The gradient is \partial L/\partial\hat y_i = \frac{2}{n}w_i e_i. With y = (10, 12, 9, 15) and \hat y = (11, 9, 9.5, 14) the errors are (1, -3, 0.5, -1) and the loss is (1 + 27 + 0.25 + 3)/4 = 7.8125, against 2.8125 for the ordinary MSE.
import numpy as np
# Custom loss: squared error that penalises underestimation (y_hat < y) 3 times more
def asym_mse(y, y_hat, k=3.0):
e = y_hat - y
return np.mean(np.where(e < 0, k, 1.0) * e ** 2)
def asym_mse_grad(y, y_hat, k=3.0): # analytical derivative with respect to y_hat
e = y_hat - y
return 2 * np.where(e < 0, k, 1.0) * e / len(y)
y = np.array([10.0, 12.0, 9.0, 15.0])
y_hat = np.array([11.0, 9.0, 9.5, 14.0])
print("loss:", round(asym_mse(y, y_hat), 4), " ordinary MSE:", round(np.mean((y - y_hat) ** 2), 4))
g = asym_mse_grad(y, y_hat)
eps = 1e-6
num = np.array([(asym_mse(y, y_hat + eps * np.eye(4)[i]) - asym_mse(y, y_hat - eps * np.eye(4)[i])) / (2 * eps) for i in range(4)])
print("analytical gradient:", np.round(g, 4))
print("numerical gradient :", np.round(num, 4))
print("maximum gap:", f"{np.abs(g - num).max():.1e}")
# Output:
# loss: 7.8125 ordinary MSE: 2.8125
# analytical gradient: [ 0.5 -4.5 0.25 -1.5 ]
# numerical gradient : [ 0.5 -4.5 0.25 -1.5 ]
# maximum gap: 3.4e-09
The check is the same as in differential calculus: compare the analytical gradient with centred finite differences. Here the maximum gap is of the order of 10^{-9}, so the derivative is right. A flipped sign or a forgotten factor of 2 would give a gap of the same order as the gradient. The check is done on a small example, once, and takes a few lines: it costs less than a training run that does not converge.
How to choose
There is no best loss in absolute terms; you start from the type of output and adjust according to the data. The indications that follow are a starting point, not rules:
- regression: MSE if large errors must weigh a lot and the data are clean, MAE or Huber if there are outliers you do not want to chase (with Huber, \delta must be chosen on the scale of the errors)
- binary or multiclass classification: cross-entropy, preferably computed on the logits
- very imbalanced classes: cross-entropy with class weights or focal loss, evaluating with precision and recall
- embeddings and similarity: contrastive or triplet, paying attention to the choice of pairs and triplets
The first two lines are the base case, the others are used when a property of the data (imbalance) or of the objective (similarity rather than a label) makes the standard loss the wrong one. Whatever the choice, the loss must be chosen together with the evaluation metric, because minimising one and judging with the other can lead to excellent scores on the loss and disappointing results on the task.
What you can do now
You can say what each loss estimates (the mean with MSE, the median with MAE), why the gradient of the cross-entropy on the logits is \hat y - y, and how the focal loss shifts the weight onto the hard cases, measuring by how much. You can avoid the two numerical traps of naive implementations, and you can write a custom loss while checking its gradient. The next step looks at the result of a training run: the article on inferential statistics shows how to establish whether a difference between two models or two variants is real or just noise.
How I checked the listings: I ran each one on its own on 2026-11-21 with Python 3.14.4, NumPy 2.3.5, and the decimal figures quoted in the text match the ones produced. With other library versions the last digits may change.
Sources
- Lin, Goyal, Girshick, He, Dollár, Focal Loss for Dense Object Detection (arXiv 1708.02002) - definition of the focal loss, effect of the factor (1-p_t)^gamma, values gamma = 2 and alpha = 0.25 in their experiments; PDF read, consulted on 2026-11-12
- Guo, Pleiss, Sun, Weinberger, On Calibration of Modern Neural Networks (arXiv 1706.04599) - temperature scaling, parameter optimised on a validation set, accuracy unchanged; PDF read, consulted on 2026-11-12
- Goodfellow, Bengio, Courville, Deep Learning, chapters 3 and 6 - cross-entropy as a loss and its link with maximum likelihood, consulted on 2026-11-12
- Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapter 9 - the squared loss as negative log-likelihood with Gaussian noise; PDF read, consulted on 2026-11-12
Commenti
Caricamento commenti...
Sign in to comment