Skip to main content

07 - Loss Functions: MSE, Cross-Entropy, Focal Loss, and Custom Loss

MSE, MAE, Huber, cross-entropy, focal loss and temperature, with hand calculations and the gradient of a custom loss checked numerically.

What do you think about this article?

At a Glance

MSE, MAE, Huber, cross-entropy, focal loss and temperature, with hand calculations and the gradient of a custom loss checked numerically.

The loss decides what "wrong" means

The model does not know what you want: it only knows how to reduce the number you give it. If the loss treats an error of one euro and one of a thousand in the same way, the model cannot tell a rounding from a disaster; if it weighs a rare case and a common one equally, on imbalanced data the common one wins. This is why the choice of loss is not an implementation detail: it defines the problem the model is really solving. The article goes through the losses you need most often, each with its formula, a hand-worked calculation and a NumPy check, and ends with how to write your own without getting the gradient wrong.

Notation: y is the true value or class, \hat y the prediction, z the logit (the number before the sigmoid or the softmax), n the number of samples. The link with what came before is close: cross-entropy is the loss derived in the article on information theory, its relation to maximum likelihood is in the article on probability and statistics, and the gradients are those of differential calculus.

Regression: MSE, MAE and Huber

The mean squared error L_{\text{MSE}} = \frac{1}{n}\sum_i (y_i - \hat y_i)^2 penalises quadratically, and its derivative with respect to the prediction, \frac{2}{n}(\hat y_i - y_i), is proportional to the error: the more wrong you are, the harder the gradient pushes. The mean absolute error L_{\text{MAE}} = \frac{1}{n}\sum_i |y_i - \hat y_i| has a gradient of constant magnitude (it does not slow down as it approaches the minimum) and is not differentiable at zero. The Huber loss with threshold \delta is quadratic for small errors and linear for large ones: \frac{1}{2}e^2 if |e| \leq \delta, otherwise \delta|e| - \frac{1}{2}\delta^2. The two branches join at |e| = \delta with the same value and the same slope.

Hand calculation on five points with errors 0.1; 0.2; 0.2; 0.1; 0.1: MSE = (0.01 + 0.04 + 0.04 + 0.01 + 0.01)/5 = 0.022, MAE = 0.7/5 = 0.14, Huber with \delta = 1 equal to half the MSE, 0.011, because all the errors lie on the quadratic branch. Adding a sixth point with error 14: MSE = (0.11 + 196)/6 \approx 32.685, MAE = (0.7 + 14)/6 = 2.45, Huber = (0.055 + 13.5)/6 \approx 2.259, where 13.5 = 1 \cdot 14 - \tfrac{1}{2}.

import numpy as np

mse = lambda y, p: np.mean((y - p) ** 2)
mae = lambda y, p: np.mean(np.abs(y - p))
def huber(y, p, delta=1.0):
    e = np.abs(y - p)
    return np.mean(np.where(e <= delta, 0.5 * e ** 2, delta * e - 0.5 * delta ** 2))

y = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
p = np.array([1.1, 2.2, 2.8, 4.1, 4.9])             # errors 0.1, 0.2, 0.2, 0.1, 0.1
print(f"no outlier:   MSE {mse(y, p):.4f}  MAE {mae(y, p):.4f}  Huber {huber(y, p):.4f}")
y2, p2 = np.append(y, 6.0), np.append(p, 20.0)      # an error of 14
print(f"with outlier: MSE {mse(y2, p2):.4f}  MAE {mae(y2, p2):.4f}  Huber {huber(y2, p2):.4f}")

# which constant minimises each loss? (data with an anomalous value)
d = np.array([1.0, 2.0, 3.0, 4.0, 100.0])
cs = np.linspace(0, 110, 1101)
best_mse = cs[np.argmin([mse(d, c) for c in cs])]
best_mae = cs[np.argmin([mae(d, c) for c in cs])]
print(f"MSE minimum at c = {best_mse:.1f} (mean = {d.mean()});  MAE minimum at c = {best_mae:.1f} (median = {np.median(d)})")

# Output:
# no outlier:   MSE 0.0220  MAE 0.1400  Huber 0.0110
# with outlier: MSE 32.6850  MAE 2.4500  Huber 2.2592
# MSE minimum at c = 22.0 (mean = 22.0);  MAE minimum at c = 3.0 (median = 3.0)

The listing confirms the three calculations. A single large error raises the MSE by three orders of magnitude (from 0.022 to 32.7) and the MAE and Huber by only one: the MSE is dominated by outliers. The last part shows the statistical meaning of the choice: among all the constant values c you could predict for five data points with one anomalous value (1, 2, 3, 4, 100), the MSE is minimal at the mean, 22, the MAE at the median, 3. With MSE the model estimates what the response is worth on average; with MAE, what it is worth typically. If the outliers are measurement errors, the median is more appropriate; if they are real and costly cases, the mean is what you care about. The MSE is the negative log-likelihood of a model with Gaussian noise, as Deisenroth and colleagues show in chapter 9 for linear regression.

Classification: binary and multiclass cross-entropy

With a sigmoid \hat y = \sigma(z), the binary cross-entropy is L = -[y\ln\hat y + (1-y)\ln(1-\hat y)]. Its gradient with respect to the logit simplifies to \partial L/\partial z = \hat y - y: the sigmoid and the logarithm cancel, and the signal does not fade when the model is confident and wrong. That is the difference from the squared error on a sigmoid, whose gradient carries the factor \sigma'(z) that vanishes at saturation, as in the article on differential calculus. The multiclass case with softmax is identical: with a one-hot label, L = -\ln\hat y_c and the gradient with respect to the logits is \hat{\mathbf y} - \mathbf y, as seen in the article on information theory. By hand, with z = 0.8 and y = 1: \hat y = \sigma(0.8) \approx 0.69, L = -\ln 0.69 \approx 0.371 and gradient 0.69 - 1 = -0.31.

import numpy as np

sigmoid = lambda z: 1 / (1 + np.exp(-z))

def bce_logit(z, y):                                # stable: max(z,0) - z*y + log(1 + exp(-|z|))
    return np.maximum(z, 0) - z * y + np.log1p(np.exp(-np.abs(z)))

z, y = 0.8, 1.0
q = sigmoid(z)
print(f"y_hat = {q:.4f}  loss = {-np.log(q):.4f}  (stable: {bce_logit(z, y):.4f})")
e = 1e-6
print(f"numerical dL/dz = {(bce_logit(z + e, y) - bce_logit(z - e, y)) / (2 * e):.5f}   y_hat - y = {q - y:.5f}")

zz = 40.0                                           # a large logit, y = 0: a confident wrong prediction
with np.errstate(divide="ignore"):
    naive = -(0 * np.log(sigmoid(zz)) + 1 * np.log(1 - sigmoid(zz)))
print(f"y = 0, z = 40: naive form = {naive}  stable form = {bce_logit(zz, 0.0):.1f}  (z - 0 = {zz})")

# Output:
# y_hat = 0.6900  loss = 0.3711  (stable: 0.3711)
# numerical dL/dz = -0.31003   y_hat - y = -0.31003
# y = 0, z = 40: naive form = inf  stable form = 40.0  (z - 0 = 40.0)

The listing checks two things. The gradient computed with finite differences coincides with \hat y - y to the fifth digit. And it shows a problem that tutorials often hide: with a logit of 40 and y = 0 (a confident, wrong prediction) the naive form -log(1 - sigmoid(z)) gives inf, because \sigma(40) in floating point is exactly 1. The stable form, \max(z, 0) - zy + \ln(1 + e^{-|z|}), never goes through the sigmoid and returns 40, the correct value (the loss grows linearly with the logit, it does not explode). This is why libraries offer a version of the loss that takes the logits, not the probabilities, as its argument.

Imbalanced data: focal loss

In a problem with very few positives and a great many easy negatives, the sum of the losses is dominated by the negatives: each contributes little, but there are many of them. The focal loss (Lin et al., 2017) multiplies the cross-entropy by (1 - p_t)^\gamma, where p_t is the probability assigned to the true class: L_{\text{FL}} = -(1 - p_t)^\gamma\ln p_t. For \gamma = 2 an example with p_t = 0.9 is weighted 0.1^2 = 0.01 (a hundred times less), one with p_t = 0.5 is weighted 0.25, one with p_t = 0.1 is weighted 0.81. With \gamma = 0 you recover the cross-entropy. In the original paper, on object detection, \gamma = 2 works best in their experiments, and the optimal balancing factor \alpha was 0.25 with \gamma = 2: values specific to that problem, not universal.

import numpy as np

gamma = 2.0
for pt in (0.9, 0.5, 0.1):
    w = (1 - pt) ** gamma
    print(f"p_t = {pt}: CE = {-np.log(pt):.4f}   factor (1-p_t)^2 = {w:.2f}   focal = {w * -np.log(pt):.4f}")

# 1000 easy negatives (p_t = 0.95) and 10 hard positives (p_t = 0.1)
pt = np.concatenate([np.full(1000, 0.95), np.full(10, 0.1)])
ce = -np.log(pt)
fl = (1 - pt) ** gamma * ce
for name, l in (("cross-entropy", ce), ("focal gamma=2", fl)):
    print(f"{name:14s} total = {l.sum():7.3f}   share of the 1000 easy = {l[:1000].sum() / l.sum():.1%}   share of the 10 hard = {l[1000:].sum() / l.sum():.1%}")

# check: with gamma = 0 the focal loss coincides with the cross-entropy
print("gamma = 0 -> identical to CE:", np.allclose((1 - pt) ** 0 * ce, ce))

# Output:
# p_t = 0.9: CE = 0.1054   factor (1-p_t)^2 = 0.01   focal = 0.0011
# p_t = 0.5: CE = 0.6931   factor (1-p_t)^2 = 0.25   focal = 0.1733
# p_t = 0.1: CE = 2.3026   factor (1-p_t)^2 = 0.81   focal = 1.8651
# cross-entropy  total =  74.319   share of the 1000 easy = 69.0%   share of the 10 hard = 31.0%
# focal gamma=2  total =  18.779   share of the 1000 easy = 0.7%   share of the 10 hard = 99.3%
# gamma = 0 -> identical to CE: True

The listing simulates the typical case, with 1000 easy negatives (p_t = 0.95) and 10 hard positives (p_t = 0.1). With the cross-entropy, the thousand easy ones account for 69.0\% of the total loss and the ten hard ones for 31.0\%: the gradient is driven by the cases the model already knows how to solve. With the focal loss the easy ones drop to 0.7\% and the hard ones rise to 99.3\%. The practical consequence is that the loss is no longer the negative log-likelihood of the data; as an alternative, to correct the imbalance there are class weights in the cross-entropy, which do not change the shape of the loss, and the choice between the two routes should be made with a metric on the problem (precision, recall) and not with the loss itself.

Margin losses and losses for embeddings: hinge, contrastive and triplet

Some losses do not measure a probability but a distance or a margin. The hinge loss of SVMs, \max(0, 1 - yf(x)) with y \in \{-1, +1\}, is zero when the point is correctly classified with a margin of at least 1: for y = +1 and score 1.5 the loss is 0, for 0.3 it is 0.7, for -0.5 it is 1.5. The contrastive loss works on pairs of embeddings with distance d: for similar pairs it is \frac{1}{2}d^2 (it pulls them together), for different pairs \frac{1}{2}\max(0, m - d)^2 (it pushes them apart up to the margin m). The triplet loss uses triplets (anchor, positive, negative): \max(0, \|f(a) - f(p)\|^2 - \|f(a) - f(n)\|^2 + m) asks that the positive be closer to the anchor than the negative by at least m in squared distance.

import numpy as np

hinge = lambda y, f: np.maximum(0, 1 - y * f)
print("hinge, y = +1:", {f: float(hinge(1, f)) for f in (1.5, 0.3, -0.5)})

def contrastive(d, similar, m=1.0):                  # similar: 1 if the pair is similar
    return 0.5 * d ** 2 if similar else 0.5 * np.maximum(0, m - d) ** 2
print("contrastive, similar pair d = 0.4:", contrastive(0.4, True))
print("contrastive, different pair d = 0.4 / d = 1.2 (m = 1):", contrastive(0.4, False), contrastive(1.2, False))

def triplet(a, p, n, m=1.0):
    return max(0.0, np.sum((a - p) ** 2) - np.sum((a - n) ** 2) + m)
a, p = np.array([0.0, 0.0]), np.array([1.0, 0.0])
print("triplet, far negative (0, 2):", triplet(a, p, np.array([0.0, 2.0])))
print("triplet, near negative (0, 1):", triplet(a, p, np.array([0.0, 1.0])))

# Output:
# hinge, y = +1: {1.5: 0.0, 0.3: 0.7, -0.5: 1.5}
# contrastive, similar pair d = 0.4: 0.08000000000000002
# contrastive, different pair d = 0.4 / d = 1.2 (m = 1): 0.18 0.0
# triplet, far negative (0, 2): 0.0
# triplet, near negative (0, 1): 1.0

The values in the listing can be checked by hand. Contrastive with margin 1 and d = 0.4: similar pair \frac{1}{2}\cdot 0.16 = 0.08; different pair \frac{1}{2}(1 - 0.4)^2 = 0.18; with d = 1.2 the different pair is already beyond the margin and costs zero. Triplet with anchor (0,0), positive (1,0) and negative (0,2): 1 - 4 + 1 = -2, so loss 0 (the triplet is already satisfied); with negative (0,1): 1 - 1 + 1 = 1. The fact that many triplets contribute zero is why, in real systems, the choice of the hard triplets to learn from is what matters.

Temperature: changing the confidence without changing the answer

Dividing the logits by a temperature T > 0 before the softmax, \text{softmax}(z_k/T), does not change which class has the maximum probability, but changes its confidence: T > 1 flattens the distribution, T < 1 makes it sharper. For z = (2, 1, 0.1) class 0 gets 0.864 with T = 0.5, 0.659 with T = 1, 0.502 with T = 2. Guo and colleagues (2017) show that modern deep networks are often poorly calibrated, that is, overconfident, and that temperature scaling, a single-parameter variant of Platt scaling, is surprisingly effective at correcting it. The parameter T is searched on a validation set by minimising the negative log-likelihood, and it does not change the accuracy.

import numpy as np

softmax = lambda z, T=1.0: np.exp(z / T) / np.exp(z / T).sum()
z = np.array([2.0, 1.0, 0.1])
for T in (0.5, 1.0, 2.0):
    print(f"T = {T}: {np.round(softmax(z, T), 4)}  predicted class {softmax(z, T).argmax()}")

# calibration: synthetic data with an overconfident classifier
rng = np.random.default_rng(0)
n, K = 5000, 3
y = rng.integers(0, K, n)
signal = np.eye(K)[y] * 1.5 + rng.normal(0, 1.0, (n, K))      # weak hint about the true class
logits = 4.0 * signal                                          # inflated logits: overconfident
def probs(T):
    s = logits / T
    e = np.exp(s - s.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)
nll = lambda T: -np.mean(np.log(probs(T)[np.arange(n), y]))
conf = lambda T: probs(T).max(axis=1).mean()
acc = np.mean(logits.argmax(1) == y)
Ts = np.linspace(0.5, 10, 951)
T_best = Ts[np.argmin([nll(T) for T in Ts])]
print(f"accuracy = {acc:.3f} (does not depend on T)")
print(f"T = 1.00: NLL = {nll(1.0):.3f}  mean confidence = {conf(1.0):.3f}")
print(f"T = {T_best:.2f}: NLL = {nll(T_best):.3f}  mean confidence = {conf(T_best):.3f}")

# Output:
# T = 0.5: [0.8638 0.1169 0.0193]  predicted class 0
# T = 1.0: [0.659  0.2424 0.0986]  predicted class 0
# T = 2.0: [0.5017 0.3043 0.194 ]  predicted class 0
# accuracy = 0.766 (does not depend on T)
# T = 1.00: NLL = 0.834  mean confidence = 0.906
# T = 2.64: NLL = 0.571  mean confidence = 0.763

The second part of the listing is synthetic: it generates a three-class classifier with inflated logits, which guesses the class in 76.6% of cases but declares a mean confidence of 90.6%. The search for the T that minimises the negative log-likelihood finds T \approx 2.64, for which the loss falls from 0.834 to 0.571 and the mean confidence from 0.906 to 0.763, almost equal to the accuracy. The result depends on the way the data were built, it is not a measurement of a real network; but it shows the mechanism: calibration is a problem of logit scale, not of classification, and it must be done on data not used for training.

Writing your own loss, and checking its gradient

When the standard losses do not represent the real cost, for example because underestimating a demand costs more than overestimating it, you write your own. The risk is the gradient: with an automatic differentiation framework you get it for free, but if you write it by hand (or use a loss that is not differentiable at some point) a single wrong sign is enough to train in the opposite direction. The example is a squared error that weighs underestimates k = 3 times more, e = \hat y - y < 0: L = \frac{1}{n}\sum w_i e_i^2 with w_i = 3 if e_i < 0 and 1 otherwise. The gradient is \partial L/\partial\hat y_i = \frac{2}{n}w_i e_i. With y = (10, 12, 9, 15) and \hat y = (11, 9, 9.5, 14) the errors are (1, -3, 0.5, -1) and the loss is (1 + 27 + 0.25 + 3)/4 = 7.8125, against 2.8125 for the ordinary MSE.

import numpy as np

# Custom loss: squared error that penalises underestimation (y_hat < y) 3 times more
def asym_mse(y, y_hat, k=3.0):
    e = y_hat - y
    return np.mean(np.where(e < 0, k, 1.0) * e ** 2)

def asym_mse_grad(y, y_hat, k=3.0):                # analytical derivative with respect to y_hat
    e = y_hat - y
    return 2 * np.where(e < 0, k, 1.0) * e / len(y)

y = np.array([10.0, 12.0, 9.0, 15.0])
y_hat = np.array([11.0, 9.0, 9.5, 14.0])
print("loss:", round(asym_mse(y, y_hat), 4), "   ordinary MSE:", round(np.mean((y - y_hat) ** 2), 4))

g = asym_mse_grad(y, y_hat)
eps = 1e-6
num = np.array([(asym_mse(y, y_hat + eps * np.eye(4)[i]) - asym_mse(y, y_hat - eps * np.eye(4)[i])) / (2 * eps) for i in range(4)])
print("analytical gradient:", np.round(g, 4))
print("numerical gradient :", np.round(num, 4))
print("maximum gap:", f"{np.abs(g - num).max():.1e}")

# Output:
# loss: 7.8125    ordinary MSE: 2.8125
# analytical gradient: [ 0.5  -4.5   0.25 -1.5 ]
# numerical gradient : [ 0.5  -4.5   0.25 -1.5 ]
# maximum gap: 3.4e-09

The check is the same as in differential calculus: compare the analytical gradient with centred finite differences. Here the maximum gap is of the order of 10^{-9}, so the derivative is right. A flipped sign or a forgotten factor of 2 would give a gap of the same order as the gradient. The check is done on a small example, once, and takes a few lines: it costs less than a training run that does not converge.

How to choose

There is no best loss in absolute terms; you start from the type of output and adjust according to the data. The indications that follow are a starting point, not rules:

  • regression: MSE if large errors must weigh a lot and the data are clean, MAE or Huber if there are outliers you do not want to chase (with Huber, \delta must be chosen on the scale of the errors)
  • binary or multiclass classification: cross-entropy, preferably computed on the logits
  • very imbalanced classes: cross-entropy with class weights or focal loss, evaluating with precision and recall
  • embeddings and similarity: contrastive or triplet, paying attention to the choice of pairs and triplets

The first two lines are the base case, the others are used when a property of the data (imbalance) or of the objective (similarity rather than a label) makes the standard loss the wrong one. Whatever the choice, the loss must be chosen together with the evaluation metric, because minimising one and judging with the other can lead to excellent scores on the loss and disappointing results on the task.

What you can do now

You can say what each loss estimates (the mean with MSE, the median with MAE), why the gradient of the cross-entropy on the logits is \hat y - y, and how the focal loss shifts the weight onto the hard cases, measuring by how much. You can avoid the two numerical traps of naive implementations, and you can write a custom loss while checking its gradient. The next step looks at the result of a training run: the article on inferential statistics shows how to establish whether a difference between two models or two variants is real or just noise.

How I checked the listings: I ran each one on its own on 2026-11-21 with Python 3.14.4, NumPy 2.3.5, and the decimal figures quoted in the text match the ones produced. With other library versions the last digits may change.

Sources

Try These Tools

Free developer tools related to this topic.

Commenti

Caricamento commenti...

Sign in to comment

New articles and tools on Telegram too

The Telegram channel flags the site's new articles and tools.

Did you like this article?

Explore other content on the blog or discover my projects