Skip to main content

03 - Probability and Statistics: Distributions, Bayesian Inference, and Hypothesis Testing

Bayes' theorem, key distributions, Naive Bayes, MLE and MAP, the central limit theorem and hypothesis testing, each with a hand calculation and a check.

What do you think about this article?

At a Glance

Bayes' theorem, key distributions, Naive Bayes, MLE and MAP, the central limit theorem and hypothesis testing, each with a hand calculation and a check.

Reasoning under uncertainty: why a positive test is not enough

A medical test with 90% sensitivity and 5% false positives comes back positive: how likely is it that you are ill? If the disease affects one person in a hundred, the answer is about 15%, not 90%. This error of intuition is the same one that makes you overrate a classifier with high accuracy on a rare class, and it is corrected with a single tool, Bayes' theorem. The article builds it starting from conditional probability, moves on to the distributions that really matter in machine learning, shows that training a model means estimating parameters (maximum likelihood, then maximum a posteriori) and closes with hypothesis testing, the other way of deciding whether an effect is real. Every formula has a hand calculation and a check in code.

In the notation of the series, P(\cdot) is the probability, \mathbb{E}[\cdot] the expected value and \theta the model parameters, as in the article on linear algebra; the observed data are D. The derivative of the log-likelihood will use what was seen in the article on differential calculus.

Conditional probability, independence and Bayes' theorem

The probability of A knowing that B has occurred is P(A \mid B) = P(A \cap B)/P(B): the universe is restricted to the cases in which B is true. Two events are independent if knowing B does not change P(A), that is P(A \cap B) = P(A)P(B). Swapping the roles of A and B in the definition gives Bayes' theorem, which inverts the conditioning: from «how likely is the data if the hypothesis is true» to «how likely is the hypothesis given the data».

P(\theta \mid D) = \frac{P(D \mid \theta)\,P(\theta)}{P(D)}

In the language of inference, the four quantities have names of their own:

  • P(\theta) is the prior, the belief about the hypothesis before seeing the data
  • P(D \mid \theta) is the likelihood, how plausible the observed data are if the hypothesis is true
  • P(\theta \mid D) is the posterior, the belief updated after the data
  • P(D) is the evidence, a constant that makes the posterior a probability: it is obtained by summing P(D \mid \theta)P(\theta) over all hypotheses

The first three items are the heart of the formula: the posterior is proportional to the product of likelihood and prior. The evidence does not depend on the hypothesis, so to compare two hypotheses it is enough to compare the numerators; this is what is done in classification. The prior is the item that gets forgotten, and it is the one that decides the result in the medical test.

The calculation. With prevalence P(M) = 0.01, sensitivity P(+ \mid M) = 0.90 and false positives P(+ \mid \text{not } M) = 0.05, the total probability of a positive result is P(+) = 0.90 \cdot 0.01 + 0.05 \cdot 0.99 = 0.009 + 0.0495 = 0.0585. So P(M \mid +) = 0.009/0.0585 \approx 0.1538. In other words, out of a thousand people only 10 are ill and 9 of them test positive, but so do 49.5 of the 990 healthy ones: one positive in six is really ill.

import numpy as np

prev, sens, fpr = 0.01, 0.90, 0.05          # P(M), P(+|M), P(+|not M)
p_pos = sens * prev + fpr * (1 - prev)       # P(+) by the law of total probability
post = sens * prev / p_pos                   # Bayes: P(M|+)
print(f"P(+) = {p_pos:.4f}   P(M|+) = {post:.4f}")

rng = np.random.default_rng(0)
n = 1_000_000
ill = rng.random(n) < prev
pos = np.where(ill, rng.random(n) < sens, rng.random(n) < fpr)
print(f"simulation: {pos.sum()} positives out of {n}, of which ill {np.mean(ill[pos]):.4f}")

for prev2 in (0.001, 0.01, 0.1, 0.5):
    p2 = sens * prev2 + fpr * (1 - prev2)
    print(f"prevalence {prev2:5.3f} -> P(M|+) = {sens * prev2 / p2:.4f}")

# Output:
# P(+) = 0.0585   P(M|+) = 0.1538
# simulation: 58261 positives out of 1000000, of which ill 0.1539
# prevalence 0.001 -> P(M|+) = 0.0177
# prevalence 0.010 -> P(M|+) = 0.1538
# prevalence 0.100 -> P(M|+) = 0.6667
# prevalence 0.500 -> P(M|+) = 0.9474

The first two lines of the listing repeat the hand calculation. The simulation generates a million people, assigns each one the disease and the test result, and measures the fraction of ill people among the positives: 0.1539, which matches 0.1538 up to sampling noise. The final loop shows the effect of the prior: with prevalence 0.1% the same test gives a posterior of just 1.8%, with prevalence 50% it rises to 94.7%. The same logic holds for a fraud or spam classifier: its accuracy says nothing about the probability that an alarm is true until you specify how rare the class is.

Expected value, variance and the distributions that matter in machine learning

The expected value of a random variable X is the weighted mean of its values, \mathbb{E}[X] = \sum_x x\,P(X = x) (for continuous variables the sum becomes an integral with the density). The variance measures dispersion, \mathrm{Var}(X) = \mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2] - \mathbb{E}[X]^2, and the standard deviation \sigma = \sqrt{\mathrm{Var}(X)} has the same unit as X. For a fair die \mathbb{E}[X] = (1 + \dots + 6)/6 = 3.5 and \mathbb{E}[X^2] = 91/6, so \mathrm{Var}(X) = 91/6 - 3.5^2 = 35/12 \approx 2.9167.

Three distributions cover most practical cases. The Bernoulli describes a binary outcome: P(X = k) = p^k(1-p)^{1-k} for k \in \{0, 1\}, with mean p and variance p(1-p); the output of a sigmoid neuron is the parameter p of a Bernoulli. The categorical generalizes it to K classes with probabilities p_1, \dots, p_K summing to 1; the output of a softmax is a categorical distribution. The Gaussian has density

f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)

with mean \mu and variance \sigma^2. For heights with \mu = 170 cm and \sigma = 10 cm, about 68.3% of the population lies within one standard deviation (160–180 cm) and 95.4% within two (150–190 cm). The Gaussian appears everywhere thanks to the central limit theorem, covered further on.

import numpy as np
from scipy import stats

faces = np.arange(1, 7)
mean = faces.mean()                          # E[X] = (1+...+6)/6
var = (faces ** 2).mean() - mean ** 2        # E[X^2] - E[X]^2
print(f"die: E = {mean}, Var = {var:.4f}  (35/12 = {35 / 12:.4f})")

rng = np.random.default_rng(1)
rolls = rng.integers(1, 7, size=100_000)
print(f"die, 100000 rolls: mean {rolls.mean():.3f}, variance {rolls.var():.3f}")

p = 0.7
print(f"Bernoulli(0.7): mean {p}, variance p(1-p) = {p * (1 - p):.2f}, empirical {stats.bernoulli(p).rvs(100_000, random_state=2).var():.3f}")

g = stats.norm(loc=170, scale=10)
print(f"P(160 < X < 180) = {g.cdf(180) - g.cdf(160):.4f}   P(150 < X < 190) = {g.cdf(190) - g.cdf(150):.4f}")

z = np.array([2.0, 1.0, 0.1])                # three scores -> categorical distribution
q = np.exp(z) / np.exp(z).sum()
draws = rng.choice(3, size=100_000, p=q)
print("softmax:", np.round(q, 4), " frequencies:", np.round(np.bincount(draws) / draws.size, 4))

# Output:
# die: E = 3.5, Var = 2.9167  (35/12 = 2.9167)
# die, 100000 rolls: mean 3.497, variance 2.921
# Bernoulli(0.7): mean 0.7, variance p(1-p) = 0.21, empirical 0.209
# P(160 < X < 180) = 0.6827   P(150 < X < 190) = 0.9545
# softmax: [0.659  0.2424 0.0986]  frequencies: [0.6597 0.2421 0.0981]

The listing compares each formula with a sample: exact die variance 2.9167 against 2.921 from a hundred thousand rolls, Bernoulli variance p(1-p) = 0.21 against 0.209, and the Gaussian probabilities 0.6827 and 0.9545 computed with the cumulative distribution function (cdf). The three empirical softmax frequencies, drawn with rng.choice, match the probabilities to the third digit: this is the check that «softmax output» and «categorical distribution» are the same thing. default_rng is used with an explicit seed so that the random numbers are reproducible.

Naive Bayes: classifying with Bayes' formula

The Naive Bayes classifier applies Bayes' theorem with a simplifying assumption: given the classes, the features are independent. In this way P(\text{class} \mid x_1, \dots, x_d) \propto P(\text{class}) \prod_i P(x_i \mid \text{class}) and only one distribution per feature needs to be estimated. Hand example with two words: prior 0.5; «free» appears in 80% of spam and in 10% of legitimate messages; «offer» in 60% of spam and 20% of legitimate ones. A message with both has spam score 0.5 \cdot 0.8 \cdot 0.6 = 0.24 and legitimate score 0.5 \cdot 0.1 \cdot 0.2 = 0.01; normalizing, P(\text{spam}) = 0.24/0.25 = 0.96.

import numpy as np
from sklearn.naive_bayes import GaussianNB

# 1) by hand: prior 0.5, P(free|spam)=0.8, P(free|ham)=0.1, P(offer|spam)=0.6, P(offer|ham)=0.2
spam = 0.5 * 0.8 * 0.6
ham = 0.5 * 0.1 * 0.2
print(f"spam ~ {spam:.3f}  ham ~ {ham:.3f}  P(spam | free, offer) = {spam / (spam + ham):.2f}")

# 2) the same scheme with continuous features (word counts) and Gaussian likelihood
X = np.array([[3, 2, 0], [4, 3, 1], [0, 0, 3], [1, 0, 4], [5, 4, 0], [0, 1, 2]], dtype=float)
y = np.array([1, 1, 0, 0, 1, 0])             # 1 = spam
x_new = np.array([2.0, 1.0, 2.0])

eps = 1e-9 * X.var(axis=0).max()             # same regularization as scikit-learn
def log_post(c):
    Xc = X[y == c]
    mu, var = Xc.mean(axis=0), Xc.var(axis=0) + eps
    ll = -0.5 * np.sum(np.log(2 * np.pi * var) + (x_new - mu) ** 2 / var)
    return np.log(len(Xc) / len(X)) + ll
lp = np.array([log_post(0), log_post(1)])
prob = np.exp(lp - lp.max()); prob /= prob.sum()   # normalization: subtracting the maximum avoids underflow
print("by hand      P(ham), P(spam):", np.round(prob, 6))
print("scikit-learn P(ham), P(spam):", np.round(GaussianNB().fit(X, y).predict_proba([x_new])[0], 6))

# Output:
# spam ~ 0.240  ham ~ 0.010  P(spam | free, offer) = 0.96
# by hand      P(ham), P(spam): [0.991832 0.008168]
# scikit-learn P(ham), P(spam): [0.991832 0.008168]

The second half of the listing is the same scheme with word counts and Gaussian likelihood, implemented in full and compared with scikit-learn's GaussianNB: the two probabilities agree to the sixth digit. The technical point is working with logarithms: a product of many small probabilities underflows, a sum of logarithms does not, and to normalize you subtract the maximum before exponentiating (the same trick as the softmax). The independence assumption is rarely true, and the counts of «free» and «offer» tend to appear together; the classifier works anyway as a ranker, but the scikit-learn guide warns that it is a poor probability estimator: the outputs of predict_proba should be treated as scores, not as calibrated probabilities. With only three samples per class, moreover, the estimated variances are very unstable: you need a few dozen examples to trust it.

Maximum likelihood: the parameters that make the data most probable

Maximum likelihood estimation (MLE) chooses the parameters \theta that maximize the probability of the observed data. For independent observations the likelihood is the product \mathcal{L}(\theta) = \prod_i P(x_i \mid \theta); we work with its logarithm, which turns the product into a sum and does not change the maximum point: \ell(\theta) = \sum_i \log P(x_i \mid \theta). The derivative is set to zero. For Gaussian data the result is the sample mean \hat\mu = \frac{1}{n}\sum x_i and the variance \hat\sigma^2 = \frac{1}{n}\sum (x_i - \hat\mu)^2.

With ten heights in cm, 168, 172, 175, 170, 173, 169, 171, 174, 176, 170, the sum is 1718 and the mean 171.8. The squared deviations add up to 63.6, so the MLE variance is 63.6/10 = 6.36. It is a biased estimate: on average it underestimates the true variance, and the unbiased estimator divides by n-1 and gives 63.6/9 \approx 7.0667. With a lot of data the difference vanishes, with little it does not, and this is why np.var has the ddof parameter.

import numpy as np

data = np.array([168, 172, 175, 170, 173, 169, 171, 174, 176, 170], dtype=float)
n = len(data)
mu = data.mean()
print(f"mean (MLE of mu)             = {mu}")
print(f"MLE variance (divides by n)  = {data.var():.4f}   (63.6 / 10)")
print(f"unbiased variance (n-1)      = {data.var(ddof=1):.4f}   (63.6 / 9)")

# the mean maximizes the log-likelihood: check on a grid
grid = np.linspace(165, 178, 13001)
sigma2 = data.var()
ll = -0.5 * n * np.log(2 * np.pi * sigma2) - ((data[:, None] - grid) ** 2).sum(axis=0) / (2 * sigma2)
print(f"maximum of the log-likelihood at mu = {grid[ll.argmax()]:.3f}")

# Bernoulli: labels [1, 1, 0, 1]; mean negative log-likelihood = binary cross-entropy
labels = np.array([1, 1, 0, 1])
bce = lambda p: -np.mean(labels * np.log(p) + (1 - labels) * np.log(1 - p))
ps = np.linspace(0.01, 0.99, 981)
print(f"cross-entropy with p = 0.8 : {bce(0.8):.4f}   with p = 0.75 : {bce(0.75):.4f}")
print(f"p that minimizes the cross-entropy: {ps[np.argmin([bce(q) for q in ps])]:.2f}  (k/n = 3/4)")

# Output:
# mean (MLE of mu)             = 171.8
# MLE variance (divides by n)  = 6.3600   (63.6 / 10)
# unbiased variance (n-1)      = 7.0667   (63.6 / 9)
# maximum of the log-likelihood at mu = 171.800
# cross-entropy with p = 0.8 : 0.5697   with p = 0.75 : 0.5623
# p that minimizes the cross-entropy: 0.75  (k/n = 3/4)

The listing verifies two things. The first is that the mean really is the maximum: the log-likelihood on a fine grid peaks at 171.8. The second is the link that makes MLE relevant for learning: for binary labels, the mean negative log-likelihood of a Bernoulli is exactly the binary cross-entropy, -\frac{1}{n}\sum [y_i \log p + (1-y_i)\log(1-p)]. With labels (1, 1, 0, 1) and p = 0.8 it equals (3 \cdot 0.2231 + 1.6094)/4 \approx 0.5697; with p = 0.75 it drops to 0.5623 and this is the minimum, in agreement with \hat p = k/n = 3/4. Minimizing the cross-entropy is therefore the same thing as maximizing the likelihood, and the article on loss functions exploits the consequences.

Maximum a posteriori: the prior becomes regularization

The MAP estimate adds to the likelihood a prior on the parameters: \hat\theta_{\text{MAP}} = \arg\max_\theta [\log P(D \mid \theta) + \log P(\theta)]. With a Gaussian prior P(\theta) = \mathcal{N}(0, \sigma_p^2) the term -\log P(\theta) equals \theta^2/(2\sigma_p^2) plus a constant: it is a quadratic penalty on the weights, that is, L2 regularization (Ridge) with coefficient 1/(2\sigma_p^2) with respect to the negative log-likelihood. It is the probabilistic reading of a technique often introduced only as a trick (see §9.2.3 of Deisenroth and colleagues). A small prior, a narrow \sigma_p, means a strong penalty.

For the mean of a Gaussian with known variance \sigma^2 and prior \mathcal{N}(\mu_0, \sigma_p^2) the result is in closed form, a weighted mean between the data and the prior with weights equal to the «precisions» n/\sigma^2 and 1/\sigma_p^2:

\hat\mu_{\text{MAP}} = \frac{(n/\sigma^2)\,\overline{x} + (1/\sigma_p^2)\,\mu_0}{n/\sigma^2 + 1/\sigma_p^2}

With \sigma^2 = 6.36, \mu_0 = 170, \sigma_p^2 = 25 and the ten heights, n/\sigma^2 = 1.572 and 1/\sigma_p^2 = 0.04: the numerator is 1.572 \cdot 171.8 + 0.04 \cdot 170 \approx 276.93, the denominator 1.612, and \hat\mu_{\text{MAP}} \approx 171.76. With a single observation, x = 176, the weights are 0.157 and 0.04 and the estimate becomes 34.47/0.197 \approx 174.78, much closer to the prior than the data are.

import numpy as np
from scipy.optimize import minimize_scalar

data = np.array([168, 172, 175, 170, 173, 169, 171, 174, 176, 170], dtype=float)
sigma2 = data.var()                           # known variance (here: the MLE estimate)
mu0, sp2 = 170.0, 25.0                        # Gaussian prior on the mean: N(170, 5^2)

def map_mean(x):                              # closed form: weighted mean between data and prior
    prec_data, prec_prior = len(x) / sigma2, 1 / sp2
    return (prec_data * x.mean() + prec_prior * mu0) / (prec_data + prec_prior)

def neg_log_post(mu, x):
    return np.sum((x - mu) ** 2) / (2 * sigma2) + (mu - mu0) ** 2 / (2 * sp2)

for x in (data, np.array([176.0])):           # 10 observations, then just one (176)
    num = minimize_scalar(neg_log_post, args=(x,), bounds=(150, 190), method="bounded", options={"xatol": 1e-10}).x
    print(f"n = {len(x):2d}: data mean = {x.mean():.2f}  MAP formula = {map_mean(x):.4f}  numerical MAP = {num:.4f}")

# Output:
# n = 10: data mean = 171.80  MAP formula = 171.7553  numerical MAP = 171.7553
# n =  1: data mean = 176.00  MAP formula = 174.7832  numerical MAP = 174.7832

The listing compares the closed formula with a numerical minimization (minimize_scalar) of the negative log-posterior and the two agree to the fourth digit. The behavior that matters is the dependence on n: with ten data points the prior shifts the estimate by only 0.05 cm, with a single one by 1.2 cm. The more data there are, the less the prior weighs; regularization acts when the data are few relative to the parameters, which is the case of overfitting. The choice of the prior (here 170 with deviation 5) is subjective, and a wrong, strong prior, with little data, moves the estimate to the wrong place.

Central limit theorem: why the Gaussian is everywhere

The central limit theorem says that the mean \overline{X}_n of n independent and identically distributed variables with mean \mu and finite variance \sigma^2 tends, as n grows, to a Gaussian \mathcal{N}(\mu, \sigma^2/n), whatever the starting distribution. The standard deviation of the mean falls as 1/\sqrt{n}: you need to quadruple the data to halve the uncertainty. Example: exponential variables with mean 1 and deviation 1, highly skewed; the mean of 30 observations should have deviation 1/\sqrt{30} \approx 0.183.

import numpy as np

rng = np.random.default_rng(3)
# exponential with mean 1: very skewed (skewness 2). Means of n = 30 observations.
n, reps = 30, 100_000
means = rng.exponential(1.0, size=(reps, n)).mean(axis=1)

skew = lambda v: np.mean(((v - v.mean()) / v.std()) ** 3)
single = rng.exponential(1.0, size=reps)
print(f"single observation: mean {single.mean():.3f}  std {single.std():.3f}  skewness {skew(single):.2f}")
print(f"mean of 30:         mean {means.mean():.3f}  std {means.std():.3f}  (1/sqrt(30) = {30 ** -0.5:.3f})  skewness {skew(means):.2f}")
print(f"share within 1.96 std of the mean: {np.mean(np.abs(means - 1) < 1.96 * 30 ** -0.5):.4f}  (Gaussian: 0.95)")

# Output:
# single observation: mean 1.005  std 1.004  skewness 2.03
# mean of 30:         mean 1.001  std 0.183  (1/sqrt(30) = 0.183)  skewness 0.38
# share within 1.96 std of the mean: 0.9510  (Gaussian: 0.95)

A hundred thousand means of 30 observations give deviation 0.183, as predicted, and the skewness falls from 2.03 to 0.38: the distribution is not yet perfectly symmetric, but the share of means within 1.96 standard deviations of the mean is 0.951, almost the 0.95 of the Gaussian. The result justifies the use of Gaussian confidence intervals for means (article on inferential statistics). It requires finite variance and independence, and with small n or heavy tails the approximation can be terrible: the residual skewness here is the proof that n = 30 is not a magic threshold.

Hypothesis testing: the same question with a different logic

A hypothesis test answers a different question than Bayes: not «how likely is the hypothesis» but «how surprising would these data be if the null hypothesis were true». The p-value is the probability, computed under the null hypothesis, of obtaining a result at least as extreme as the one observed. It is not the probability that the null hypothesis is true. Example: a coin gives 9 heads in 10 tosses; null hypothesis p = 0.5. The cases with 9 or 10 heads are \binom{10}{9} + \binom{10}{10} = 11 out of 2^{10} = 1024, so P(X \geq 9) = 11/1024 \approx 0.0107; counting also the opposite extremes (0 or 1 heads) the two-sided p-value doubles and equals 0.0215.

import numpy as np
from scipy import stats

# Exact binomial test: 9 heads in 10 tosses, null hypothesis p = 0.5
k, n = 9, 10
p_one = sum(stats.binom.pmf(j, n, 0.5) for j in range(k, n + 1))
print(f"P(X >= 9 | p = 0.5) = {p_one:.4f}   (11/1024 = {11 / 1024:.4f})")
print(f"two-sided p-value (scipy) = {stats.binomtest(k, n, 0.5).pvalue:.4f}")

# Same data, Bayesian reading: uniform prior Beta(1,1) -> posterior Beta(1+9, 1+1)
post = stats.beta(1 + k, 1 + n - k)
print(f"P(p > 0.5 | data) = {1 - post.cdf(0.5):.4f}   posterior mean = {post.mean():.4f}")
print(f"95% credible interval: {np.round(post.interval(0.95), 3)}")

# Output:
# P(X >= 9 | p = 0.5) = 0.0107   (11/1024 = 0.0107)
# two-sided p-value (scipy) = 0.0215
# P(p > 0.5 | data) = 0.9941   posterior mean = 0.8333
# 95% credible interval: [0.587 0.977]

The listing computes the p-value twice, by summing binomial probabilities and with binomtest, and then looks at the same data from a Bayesian angle: a uniform prior \mathrm{Beta}(1,1) becomes, after 9 heads and 1 tail, a posterior \mathrm{Beta}(10, 2), for which P(p > 0.5 \mid \text{data}) = 0.9941 and the 95% credible interval is [0.587;\ 0.977]. The two numbers, 0.0215 and 0.9941, answer different questions and must not be confused: the first does not say that the coin is rigged with probability 97.85\%. Thresholds like 0.05 are conventions, not laws, and with many tests run some will cross the threshold by chance; confidence intervals and the A/B test are developed in the next article on inferential statistics.

What you can do now

You can invert a conditioning with Bayes' theorem and check the result with a simulation, you can recognize a categorical distribution in a softmax and a negative log-likelihood in a cross-entropy, and you can read L2 regularization as the Gaussian prior of a MAP estimate whose weight decreases as the data grow. You can also tell a p-value from a posterior. The next step is to use the gradient to search for these parameters: the article on optimization starts from gradient descent on a negative log-likelihood.

Sources

Try These Tools

Free developer tools related to this topic.

Commenti

Caricamento commenti...

Sign in to comment

New articles and tools on Telegram too

The Telegram channel flags the site's new articles and tools.

Did you like this article?

Explore other content on the blog or discover my projects