Skip to main content

08 - Inferential Statistics: Confidence Intervals and A/B Testing

Confidence intervals, hypothesis tests, power and the sample size of an A/B test, with hand calculations, simulations and the traps of repeated looks.

What do you think about this article?

At a Glance

Confidence intervals, hypothesis tests, power and the sample size of an A/B test, with hand calculations, simulations and the traps of repeated looks.

Is model B really better, or did you just get lucky?

You trained two versions of a model: the first has a mean accuracy of 0.909 over fifteen runs, the second 0.933. The difference is a little over two points; is it real or is it sampling noise? Inferential statistics answers this question in three moves: estimate how uncertain a mean is (standard error and confidence interval), decide whether a difference is compatible with chance (hypothesis testing), and plan an experiment large enough to notice an effect that exists (power). The article goes through the three moves with hand calculations and checks in NumPy and SciPy, and shows two mistakes made in practice: stopping an A/B test as soon as the p-value drops below 0.05, and using the wrong formula for the sample size.

It builds on the article on probability and statistics, from which it takes the central limit theorem and the first hypothesis test. When two training configurations are being compared, for example two optimizers as in the article on optimization, you need several runs with different seeds: a single run cannot tell merit from chance. Notation: \overline{x} is the sample mean, s the sample standard deviation (divided by n-1), \mu the true mean.

Standard error and confidence interval

The mean of a sample changes from one sample to another. By the central limit theorem it has mean \mu and standard deviation \sigma/\sqrt{n}, called the standard error (SE) and estimated by s/\sqrt{n}. The factor \sqrt{n} means that to halve the uncertainty you need four times the data. The 95\% confidence interval is \overline{x} \pm t_{0.975,\,n-1}\cdot s/\sqrt{n}, where t is the quantile of Student's t distribution with n-1 degrees of freedom (Student, 1908). You use t and not the Gaussian value 1.96 because \sigma is estimated with s, and with little data the estimate is itself uncertain: the intervals must be wider.

Hand calculation with ten accuracies, 0.92; 0.89; 0.91; 0.93; 0.90; 0.88; 0.91; 0.94; 0.90; 0.92. The sum is 9.10, so \overline{x} = 0.910. The squared deviations add up to 0.003, hence s^2 = 0.003/9, s \approx 0.01826 and \text{SE} = s/\sqrt{10} \approx 0.00577. With t_{0.975,9} = 2.2622 the half-width is 0.01306 and the interval is [0.8969;\ 0.9231]. With 1.96 you would get [0.8987;\ 0.9213], narrower than the data allow.

import numpy as np
from scipy import stats

acc = np.array([0.92, 0.89, 0.91, 0.93, 0.90, 0.88, 0.91, 0.94, 0.90, 0.92])   # 10 runs
n, mean = len(acc), acc.mean()
s = acc.std(ddof=1)                                  # sample standard deviation (divides by n-1)
se = s / np.sqrt(n)
t_crit = stats.t.ppf(0.975, df=n - 1)
print(f"mean = {mean:.4f}  s = {s:.5f}  SE = {se:.5f}  t(0.975, 9) = {t_crit:.4f}")
print(f"95% CI with t: [{mean - t_crit * se:.4f}, {mean + t_crit * se:.4f}]")
print(f"95% CI with z (1.96): [{mean - 1.96 * se:.4f}, {mean + 1.96 * se:.4f}]   <- narrower")
print("scipy.stats.t.interval:", np.round(stats.t.interval(0.95, df=n - 1, loc=mean, scale=stats.sem(acc)), 4))

# What "95%" means: repeat the experiment 100000 times with a known true mean
rng = np.random.default_rng(0)
mu, sigma, reps = 0.91, 0.02, 100_000
x = rng.normal(mu, sigma, size=(reps, n))
m, sd = x.mean(axis=1), x.std(axis=1, ddof=1)
half_t, half_z = t_crit * sd / np.sqrt(n), 1.96 * sd / np.sqrt(n)
print(f"coverage of the interval with t: {np.mean(np.abs(m - mu) < half_t):.4f}")
print(f"coverage of the interval with z: {np.mean(np.abs(m - mu) < half_z):.4f}")

# Output:
# mean = 0.9100  s = 0.01826  SE = 0.00577  t(0.975, 9) = 2.2622
# 95% CI with t: [0.8969, 0.9231]
# 95% CI with z (1.96): [0.8987, 0.9213]   <- narrower
# scipy.stats.t.interval: [0.8969 0.9231]
# coverage of the interval with t: 0.9508
# coverage of the interval with z: 0.9191

The first part repeats the calculation and compares it with scipy.stats.t.interval. The second answers the question of what "95%" means. The listing repeats 100000 times an experiment with a known true mean \mu = 0.91, builds the interval each time and counts how many times it contains \mu: with t the coverage is 95.1\%, with 1.96 only 91.9\% because with n = 10 the interval is too narrow. This frequency is the meaning of the confidence level: the 95\% is a property of the repeated procedure, not the probability that this interval contains the true value, which either is inside or is not. Reading it as a probability on the parameter requires a Bayesian approach, such as the credible interval of the earlier article.

The hypothesis test, type I and II errors and power

A test compares the data with a null hypothesis H_0 (no effect). It computes a statistic, for a mean t = (\overline{x} - \mu_0)/(s/\sqrt{n}), and its p-value: the probability, if H_0 is true, of observing a value at least as extreme. You reject H_0 if p < \alpha, usually \alpha = 0.05. Two errors are possible. The type I error is rejecting a true null hypothesis (false positive), and its rate is \alpha by construction. The type II error, of probability \beta, is failing to reject a false null hypothesis (false negative); the power 1 - \beta is the probability of noticing an effect that exists. The American Statistical Association (ASA, 2016) points out that the p-value does not measure the probability that the hypothesis under study is true, nor the size of the effect.

On the ten values from before, with H_0: mean = 0.90, the statistic is t = (0.910 - 0.900)/0.00577 = 1.732 (=\sqrt{3}) with 9 degrees of freedom, and the two-sided p-value is 0.117: a mean accuracy of 0.910 is compatible with 0.90 at that sample size. The same difference of one point, with more data, would become significant.

import numpy as np
from scipy import stats

acc = np.array([0.92, 0.89, 0.91, 0.93, 0.90, 0.88, 0.91, 0.94, 0.90, 0.92])
t, p = stats.ttest_1samp(acc, popmean=0.90)
print(f"H0: mean = 0.90:  t = {t:.4f}   p (two-sided) = {p:.4f}")

# Type I error: if H0 is true, how often does the test get it wrong?
rng = np.random.default_rng(1)
x = rng.normal(0.90, 0.02, size=(50_000, 10))
pv = stats.ttest_1samp(x, popmean=0.90, axis=1).pvalue
print(f"H0 true: fraction of rejections at alpha = 0.05: {np.mean(pv < 0.05):.4f}")

# Power: if the true mean is 0.91 (effect 0.01), how often does the test notice?
y = rng.normal(0.91, 0.02, size=(50_000, 10))
pw = stats.ttest_1samp(y, popmean=0.90, axis=1).pvalue
print(f"true mean 0.91, n = 10: power = {np.mean(pw < 0.05):.4f}")

# Output:
# H0: mean = 0.90:  t = 1.7321   p (two-sided) = 0.1173
# H0 true: fraction of rejections at alpha = 0.05: 0.0499
# true mean 0.91, n = 10: power = 0.2962

The listing verifies that the test errs as promised. If you simulate the true null hypothesis (mean 0.90) 50000 times, the test rejects in 4.99\% of cases: this is the type I error rate, equal to \alpha. If the true mean is 0.91 (a real effect of one point) with only 10 samples, the test notices it in just 29.6\% of cases: the power is low, and 70.4\% of the time the real effect goes unnoticed. A "non-significant" result with little data does not demonstrate the absence of an effect, but the inability to see it.

Comparing two models: Welch's t-test and the effect size

For two independent groups the statistic is t = \dfrac{\overline{x}_1 - \overline{x}_2}{\sqrt{s_1^2/n_1 + s_2^2/n_2}}. This form, without assuming equal variances, is Welch's test. In SciPy you have to ask for it: ttest_ind has equal_var=True by default, and runs Student's test with a common variance; you get Welch with equal_var=False. A small hand example: A = (0.90; 0.92; 0.91) and B = (0.93; 0.94; 0.95) have means 0.91 and 0.94 and sample variances 0.0001 each; the standard error of the difference is \sqrt{0.0001/3 + 0.0001/3} \approx 0.008165 and t = -0.03/0.008165 = -3.674, with p-value 0.0213 (4 degrees of freedom).

The p-value does not say whether the difference is large. The standardised measure is Cohen's d, d = (\overline{x}_1 - \overline{x}_2)/s_{\text{pooled}} with s_{\text{pooled}} = \sqrt{(s_1^2 + s_2^2)/2}: the difference expressed in standard deviations. The thresholds 0.2, 0.5 and 0.8 attributed to Cohen for "small", "medium" and "large" are conventions of convenience and not a source I have verified here: what matters is the comparison with the effect size that matters in your own problem.

import numpy as np
from scipy import stats

a = np.array([0.90, 0.92, 0.91])
b = np.array([0.93, 0.94, 0.95])
va, vb = a.var(ddof=1), b.var(ddof=1)
se = np.sqrt(va / len(a) + vb / len(b))
t = (a.mean() - b.mean()) / se
print(f"A by hand: mean {a.mean():.2f}, s^2 {va:.4f};  B: mean {b.mean():.2f}, s^2 {vb:.4f};  SE = {se:.6f};  t = {t:.4f}")
print("scipy Welch:", np.round(stats.ttest_ind(a, b, equal_var=False), 4))

# 15 runs each
A = np.array([0.92, 0.89, 0.91, 0.93, 0.90, 0.88, 0.91, 0.94, 0.90, 0.92, 0.91, 0.89, 0.93, 0.90, 0.91])
B = np.array([0.94, 0.93, 0.95, 0.92, 0.94, 0.91, 0.93, 0.95, 0.93, 0.94, 0.92, 0.93, 0.94, 0.93, 0.94])
r = stats.ttest_ind(A, B, equal_var=False)
sp = np.sqrt((A.var(ddof=1) + B.var(ddof=1)) / 2)
print(f"means {A.mean():.4f} and {B.mean():.4f}; Welch t = {r.statistic:.3f}, p = {r.pvalue:.2e}; Cohen's d = {(B.mean() - A.mean()) / sp:.2f}")

# Significant does not mean large: two groups of a million, difference 0.01 with deviation 1
r2 = stats.ttest_ind_from_stats(0.01, 1.0, 1_000_000, 0.0, 1.0, 1_000_000)
print(f"n = 1000000 per group, difference 0.01, d = 0.01:  t = {r2.statistic:.2f}, p = {r2.pvalue:.1e}")

# Output:
# A by hand: mean 0.91, s^2 0.0001;  B: mean 0.94, s^2 0.0001;  SE = 0.008165;  t = -3.6742
# scipy Welch: [-3.6742  0.0213]
# means 0.9093 and 0.9333; Welch t = -4.637, p = 1.01e-04; Cohen's d = 1.69
# n = 1000000 per group, difference 0.01, d = 0.01:  t = 7.07, p = 1.5e-12

The listing repeats the hand calculation (same t and p-value) and then applies it to the two models with fifteen runs each: means 0.9093 and 0.9333, t = -4.64, p = 1.0 \cdot 10^{-4}, d = 1.69, a difference much larger than the dispersion. The last line shows the opposite case: with a million samples per group, a difference of 0.01 standard deviations (d = 0.01) gives t = 7.07 and p \approx 10^{-12}. It is statistically significant and practically irrelevant. The numbers for the runs are synthetic and chosen for the example; runs that share the same test set are not independent, and this has to be kept in mind with real data.

How many samples you need: the formula with the factor two

Before the experiment you decide how large it must be, because power depends on the number of samples. To compare the means of two independent groups with an effect d, level \alpha and power 1 - \beta, the number per group is n = 2\left(\dfrac{z_{1-\alpha/2} + z_{1-\beta}}{d}\right)^2. The factor 2 comes from the variance of the difference of two means, which is twice that of one mean. For d = 0.4, \alpha = 0.05 and power 0.80: z_{0.975} + z_{0.80} = 1.960 + 0.842 = 2.802, so n = 2 \cdot (2.802/0.4)^2 = 2 \cdot 49.06 = 98.1, that is 99 per group. Without the factor 2 you get 50, which is the formula valid for comparing one mean with a known value, not for an A/B test.

import numpy as np
from scipy import stats

def n_per_group(d, alpha=0.05, power=0.80):         # two independent groups
    z = stats.norm.ppf(1 - alpha / 2) + stats.norm.ppf(power)
    return 2 * (z / d) ** 2

d = 0.4
print(f"d = {d}: n per group = 2 * ((z_a/2 + z_b) / d)^2 = {n_per_group(d):.2f}  -> {int(np.ceil(n_per_group(d)))}")
print(f"(one-group formula: {n_per_group(d) / 2:.2f}  -> {int(np.ceil(n_per_group(d) / 2))})")

rng = np.random.default_rng(2)
def power(n, d, reps=20_000):
    a = rng.normal(0, 1, size=(reps, n))
    b = rng.normal(d, 1, size=(reps, n))
    return np.mean(stats.ttest_ind(a, b, axis=1).pvalue < 0.05)
for n in (50, 99):
    print(f"n = {n:3d} per group, true effect d = {d}: simulated power = {power(n, d):.3f}")

# Output:
# d = 0.4: n per group = 2 * ((z_a/2 + z_b) / d)^2 = 98.11  -> 99
# (one-group formula: 49.06  -> 50)
# n =  50 per group, true effect d = 0.4: simulated power = 0.506
# n =  99 per group, true effect d = 0.4: simulated power = 0.795

The listing proves it by simulation, which trusts no formula: with 50 samples per group and a true effect of d = 0.4 the power is 0.506, that is a coin toss; with 99 it is 0.795, close to the target of 0.80. An experiment sized with the wrong formula would not be wrong in the sense of giving false results, but would have a one in two chance of missing an effect that exists. The formula is an approximation with the Gaussian: for small groups the exact value with the t is slightly higher.

Stopping a test as soon as it is significant: the error of repeated looks

The p-value is valid only if the sample size is decided beforehand. If instead you look at the result every day and stop the experiment at the first p < 0.05, the probability of a false positive grows with every look. You measure it with an A/A test, in which the two variants are identical and every "win" is by definition an error.

import numpy as np
from scipy import stats

rng = np.random.default_rng(3)
exps, n_max, looks = 5000, 1000, 10
a = rng.normal(0, 1, size=(exps, n_max))             # A/A test: no real difference
b = rng.normal(0, 1, size=(exps, n_max))

checkpoints = np.arange(n_max // looks, n_max + 1, n_max // looks)
cs_a, cs_b = a.cumsum(axis=1), b.cumsum(axis=1)
z = np.empty((exps, looks))
for j, k in enumerate(checkpoints):
    diff = cs_a[:, k - 1] / k - cs_b[:, k - 1] / k
    z[:, j] = diff / np.sqrt(2 / k)                   # known standard deviation = 1
p = 2 * stats.norm.sf(np.abs(z))
print(f"a single look at the end:               false positives = {np.mean(p[:, -1] < 0.05):.3f}")
print(f"{looks} looks, stop at the first p < 0.05: false positives = {np.mean((p < 0.05).any(axis=1)):.3f}")
print(f"{looks} looks with Bonferroni threshold 0.05/{looks}: false positives = {np.mean((p < 0.05 / looks).any(axis=1)):.3f}")

# Output:
# a single look at the end:               false positives = 0.051
# 10 looks, stop at the first p < 0.05: false positives = 0.190
# 10 looks with Bonferroni threshold 0.05/10: false positives = 0.027

With a single look at the end the false positives are 5.1\%, as promised. With ten equally spaced looks and stopping at the first p < 0.05 they are 19.0\%: almost one in five experiments with no real difference at all "wins". Dividing the threshold by the number of looks (Bonferroni, 0.05/10) brings the error back to 2.7\%, below 5\% because the successive looks largely use the same data and so are not independent: the correction is conservative. The correct alternatives are to fix the sample and look only once, or to use a sequential method designed for interim looks; the ASA puts transparency about the number of analyses performed among its principles.

Many tests at once: Bonferroni and Benjamini-Hochberg

With m independent tests and no real effect, the probability of at least one false positive at \alpha = 0.05 is 1 - 0.95^m: for m = 10 it is 0.4013, and it increases with every additional model compared. Bonferroni uses the threshold \alpha/m, which brings the overall error probability back below \alpha (0.0489 for m = 10) at the cost of losing power. Benjamini and Hochberg instead control a different quantity, the expected fraction of discoveries that are false (FDR): you sort the p-values, p_{(1)} \leq \dots \leq p_{(m)}, and reject all the hypotheses up to the largest i with p_{(i)} \leq (i/m)\,\alpha.

import numpy as np
from scipy import stats

m = 10
print(f"{m} independent tests with alpha = 0.05 and no real effect: P(at least one false positive) = {1 - 0.95 ** m:.4f}")
print(f"with Bonferroni (alpha / m = {0.05 / m}): {1 - (1 - 0.05 / m) ** m:.4f}")

p = np.array([0.001, 0.011, 0.015, 0.032, 0.60])
mm = len(p)
print("\nBonferroni (p < 0.05 / 5 = 0.01):", p < 0.05 / mm)
thr = np.arange(1, mm + 1) / mm * 0.05                # Benjamini-Hochberg thresholds: i/m * 0.05
passa = p <= thr
k = np.max(np.where(passa)[0]) + 1 if passa.any() else 0
print("BH thresholds i/m*0.05:", np.round(thr, 3), "-> the first", k, "are rejected")
adj = stats.false_discovery_control(p)
print("adjusted p (scipy, bh):", np.round(adj, 4), " rejections at 0.05:", adj < 0.05)

# Output:
# 10 independent tests with alpha = 0.05 and no real effect: P(at least one false positive) = 0.4013
# with Bonferroni (alpha / m = 0.005): 0.0489
#
# Bonferroni (p < 0.05 / 5 = 0.01): [ True False False False False]
# BH thresholds i/m*0.05: [0.01 0.02 0.03 0.04 0.05] -> the first 4 are rejected
# adjusted p (scipy, bh): [0.005 0.025 0.025 0.04  0.6  ]  rejections at 0.05: [ True  True  True  True False]

With five p-values 0.001; 0.011; 0.015; 0.032; 0.60 the Benjamini-Hochberg thresholds are 0.01; 0.02; 0.03; 0.04; 0.05, and the first four satisfy them (the fourth: 0.032 \leq 0.04); Bonferroni, with threshold 0.01, keeps only one. SciPy's check with false_discovery_control gives the same four rejections. The choice depends on what an error costs: if a false positive is very costly (a drug), Bonferroni; if you are exploring and accept a small fraction of false leads (feature selection), Benjamini-Hochberg.

What you can do now

You can build a confidence interval by hand with the t and explain that the 95\% describes the procedure, not the individual interval. You can choose between Student's test and Welch's (in SciPy, with equal_var=False), accompany the p-value with the effect size, size an experiment with the right factor 2 and check it by simulation. You can also recognise that looking at the results at every step, or trying many comparisons, inflates the false positives, and correct for it. The next article returns to the algebra from the start of the series and applies it to an architecture: the mathematics of Transformers.

Sources

Try These Tools

Free developer tools related to this topic.

Commenti

Caricamento commenti...

Sign in to comment

New articles and tools on Telegram too

The Telegram channel flags the site's new articles and tools.

Did you like this article?

Explore other content on the blog or discover my projects