06 - Test and Learn: Controlled Experiments and A/B Tests

Machine Learning Engineer course · Module 00 · Data-driven mindset · Lesson 6

Goal. Explain why the controlled experiment is the tool that solves confounding (lesson 05), and give the reader everything needed to design one: the test on the difference of two proportions (fully derived), the computation of the sample size (derived), and the most widespread trap, "checking the results every day" (peeking), measured with a simulation.

False myth. "As soon as the test shows p < 0.05, you can stop and declare the winner." By checking ten times, the false positive rate is no longer 5%.

Introduction

In lesson 05 we saw that the comparison between two groups can deceive when the groups differ in something else. The most robust remedy is called randomisation: deciding with a coin who gets what. When the assignment is random, by construction no characteristic can be correlated with the choice, so any systematic difference in the results is due to the treatment (or to chance, which statistics quantifies).

Roadmap. (1) Why the coin solves confounding. (2) The test: from the observed difference to the p-value, with all the steps. (3) How many users are needed. (4) The simulation of peeking. (5) Multiple comparisons. (6) How to work in practice.

What it is, why it exists, when NOT to use it

Test and learn is the cycle: formulate a hypothesis, design an experiment, run, measure, decide, learn, repeat. An A/B test is its simplest form: two versions (A = control, B = variant), random assignment, one metric.

Analogy: comparing two fertilisers. If you give fertiliser B to the flower beds in the sun and A to those in the shade, you do not know whether the difference is due to the fertiliser or to the sun. If you assign the fertilisers to the beds by lot, the sun is distributed by chance equally between the two groups and can contribute only noise, not a systematic distortion.

When NOT to run an A/B test:

Preparation

pip install numpy

We will use math for the error function (the CDF of the normal) and NumPy for the simulations.

First example: randomisation balances what you do not see

Let us return to Simpson's paradox: "severity" confounded the comparison. We simulate 20,000 people, 30% with a severe case. We assign the treatment at random.

import numpy as np

rng = np.random.default_rng(0)
n = 20_000
grave = rng.random(n) < 0.30                  # 30% severe cases
trattamento_B = rng.random(n) < 0.5           # coin: no relation with "grave"

print("share of severe cases in group A:", round(float(grave[~trattamento_B].mean()), 3))
print("share of severe cases in group B:", round(float(grave[trattamento_B].mean()), 3))

# Output (Python 3.14.4, seed 0, two identical runs):
# share of severe cases in group A: 0.299
# share of severe cases in group B: 0.286

How to read the code. grave is a true/false vector: rng.random(n) < 0.30 is true with probability 0.30, so about 30% of the users are "severe". trattamento_B is the coin: a second, independently drawn vector, true for about half of the users. Since the coin does not look at grave, the share of severe cases should be the same in the two groups. grave[~trattamento_B] selects the severe cases among those who did not get B (the operator ~ negates the boolean vector) and .mean() computes their fraction (the mean of true/false is the proportion of trues). The two values are close to 0.30 up to small random oscillations: this balance, obtained even for the variables we did not measure, is why randomisation works.

The two shares come out at 0.299 and 0.286 (seed 0): both close to 0.30, with a difference due only to noise. This is why the experiment works: the weights w_g of lesson 05 are the same in the two groups, so the paradox cannot appear. With randomisation you get balance even for the variables you did not measure or do not know: it is its unique advantage over "correcting afterwards".

Trap: randomisation has a unit (user, session, shop). If you randomise by session but the same user appears in both groups, the groups are not independent and the user sees two versions. Choose the unit that coincides with the one on which you compute the metric.

A second, contrasting example: without the coin the comparison deceives

To see what randomisation is really worth, we build a case in which the variant has no effect at all and compare two ways of forming the groups. In the first (self-selection) the more active users choose the new version more often, as happens when a feature is offered as "try it if you like". In the second (randomisation) the coin decides.

import math
import numpy as np

def phi(z):
    return 0.5 * (1 + math.erf(z / math.sqrt(2)))

def z_test(x_a, n_a, x_b, n_b):
    p_a, p_b = x_a / n_a, x_b / n_b
    p = (x_a + x_b) / (n_a + n_b)
    se = math.sqrt(p * (1 - p) * (1 / n_a + 1 / n_b))
    z = (p_b - p_a) / se
    return p_b - p_a, z, 2 * (1 - phi(abs(z)))

rng = np.random.default_rng(2)
n = 40_000
impegno = rng.random(n)                               # how engaged the user already is, between 0 and 1
conversione = rng.random(n) < 0.02 + 0.10 * impegno   # depends ONLY on engagement: the variant does nothing

scelte = {
    "self-selection (active users try B)": rng.random(n) < 0.2 + 0.6 * impegno,
    "randomisation (coin)": rng.random(n) < 0.5,
}
for nome, in_B in scelte.items():
    diff, z, p = z_test(int(conversione[~in_B].sum()), int((~in_B).sum()), int(conversione[in_B].sum()), int(in_B.sum()))
    print(f"{nome:<38}: B - A = {diff:+.4f} | z = {z:+6.2f} | p-value = {p:.4f}")

Output obtained:

self-selection (active users try B)   : B - A = +0.0167 | z =  +6.50 | p-value = 0.0000
randomisation (coin)                  : B - A = +0.0009 | z =  +0.35 | p-value = 0.7227

How to read the code. phi is the cumulative distribution function of the standard normal (via erf), z_test implements the test of the next section (the formula is derived in full further on). The conversion of each user depends only on impegno (engagement): there is no effect of the variant, by construction. In the first scenario the most engaged users enter group B more often, and since they convert more, B "wins" with a tiny p-value: an apparently solid and completely false result. In the second scenario the coin distributes engagement equally in the two groups and the difference is compatible with noise. It is the confounding of the previous lesson, seen in action: the statistical test is not at fault, it correctly answered a different question ("do the two groups have different rates?", yes, because they are made of different people) from the one we cared about ("does the variant change the rate?").

Theory: from data to decision

The model

Group A has n users, with x_A conversions. Group B has n users, with x_B conversions. We estimate the conversion probabilities:

p̂_A = x_A / n_A, p̂_B = x_B / n_B.

We want to know whether the observed difference D = p̂_B − p̂_A is a real effect or only sampling noise.

Step 1: the distribution of a proportion

Each user converts with probability p, independently. Let Xᵢ ∈ {0, 1} be their outcome. Then E[Xᵢ] = p and Var(Xᵢ) = p(1 − p) (the variance of a Bernoulli: E[X²] − E[X]² = p − p², because X² = X).

The sample proportion is the mean p̂ = (1/n) Σ Xᵢ. By the properties of the variance of sums of independent variables,

Var(p̂) = (1/n²) · n · p(1 − p) = p(1 − p) / n.

By the central limit theorem, for large n p̂ is approximately normal with mean p and variance p(1 − p)/n.

Step 2: the distribution of the difference

The two groups are independent, so the variances add:

Var(D) = p_A(1 − p_A)/n_A + p_B(1 − p_B)/n_B.

Step 3: the null hypothesis and the standard error

The null hypothesis H₀ says: there is no real difference, p_A = p_B = p. Under H₀ the best estimate of p uses all the data together (the pooled estimate):

p̂ = (x_A + x_B) / (n_A + n_B),

and the standard error of the difference under H₀ is

SE = sqrt( p̂(1 − p̂) · (1/n_A + 1/n_B) ).

Step 4: the z statistic and the p-value

z = (p̂_B − p̂_A) / SE.

Under H₀, z is approximately standard normal. The p-value is the probability, assuming H₀ true, of observing a difference at least as large as the one seen, in absolute value:

p-value = 2 · (1 − Φ(|z|))

where Φ is the cumulative distribution function of the standard normal (the "2" is because the test is two-sided: a difference in one direction or the other). H₀ is rejected if the p-value is below the threshold α (typically 0.05).

What the p-value is NOT. It is not the probability that H₀ is true, nor the probability that the result is due to chance. It is the probability of the data, given H₀. A p-value of 0.02 means: "if there were no difference at all, seeing a result this extreme would happen 2 times out of 100". It does not say how large or important the effect is.

Code

import math

def phi(z):
    """CDF of the standard normal, via the error function."""
    return 0.5 * (1 + math.erf(z / math.sqrt(2)))

def z_test_proporzioni(x_a, n_a, x_b, n_b):
    p_a, p_b = x_a / n_a, x_b / n_b
    p_comb = (x_a + x_b) / (n_a + n_b)
    se = math.sqrt(p_comb * (1 - p_comb) * (1 / n_a + 1 / n_b))
    z = (p_b - p_a) / se
    p_value = 2 * (1 - phi(abs(z)))
    return z, p_value

# A: 200 conversions out of 4000 (5.0%). B: 248 out of 4000 (6.2%).
z, p = z_test_proporzioni(200, 4000, 248, 4000)
print("z =", round(z, 3), "| p-value =", round(p, 4))

# Output:
# z = 2.334 | p-value = 0.0196

Output: z = 2.334, p-value = 0.0196. It passes the 0.05 threshold. But beware: "significant" does not mean "large" nor "worthwhile". The observed effect is +1.2 percentage points, and whether the variant's cost is worth it is decided by lesson 04, not by the p-value.

What "not significant" means

The test that follows says whether the observed difference is "significant". Before seeing it, it is worth fixing what it means not to be: the test does not prove that the versions are equal, it only says that with this sample you cannot tell them apart from noise. To have an informative test even in case of a null result you need the power, that is the probability of noticing a real effect of a given size: this is what is computed in the section on the sample.

How many users are needed: derivation

Before starting the experiment the sample is decided. Three numbers are needed:

• α: accepted probability of a false positive (typically 0.05).
• Power 1 − β: probability of detecting a real effect of the size you care about (typically 0.80).
• δ: the minimum effect worth detecting (a business decision!).

The first two are statistical conventions, the third is not: it is the one number that no formula supplies, because it depends on how much an improvement is worth to you. If you cannot say how much the conversion must improve for the variant to repay the work, you cannot size the test, and the calculation that follows stops here. Derivation, assuming p_A = p₀ and p_B = p₀ + δ, n per group.

1. Under the alternative hypothesis, D = p̂_B − p̂_A is approximately normal with mean δ and variance V/n, where V = p₀(1−p₀) + p₁(1−p₁) and p₁ = p₀ + δ. Let σ = sqrt(V/n).
2. We reject H₀ when |D| > z_{α/2} · σ (approximating the standard error under H₀ with σ).
3. The probability of rejecting when the effect is δ (the power) is, neglecting the wrong tail, P(D > z_{α/2} σ) = P( Z > z_{α/2} − δ/σ ), with Z standard normal.
4. We want this to be 1 − β, that is z_{α/2} − δ/σ = −z_β.
5. Hence δ/σ = z_{α/2} + z_β, that is σ = δ / (z_{α/2} + z_β).
6. Squaring and substituting σ² = V/n:

The six steps are a single idea repeated: the rejection threshold lies z_{α/2} standard errors from zero, the real distribution of the difference is centred on δ, and the power is the share of that distribution that exceeds the threshold. Requiring that share to be 80% fixes the distance between δ and the threshold, and n comes out of that distance.

n = (z_{α/2} + z_β)² · [ p₀(1−p₀) + p₁(1−p₁) ] / δ²

per group.

Read it: n grows with the square of the inverse of δ. Halving the effect you want to detect quadruples the necessary users. This is why micro-optimisations are extremely expensive to measure.

def z_quantile(q):
    """Inverse of the normal CDF, by bisection (no external library)."""
    lo, hi = -10.0, 10.0
    for _ in range(100):
        mid = (lo + hi) / 2
        if phi(mid) < q:
            lo = mid
        else:
            hi = mid
    return (lo + hi) / 2

def n_per_gruppo(p0, delta, alpha=0.05, potenza=0.80):
    za = z_quantile(1 - alpha / 2)
    zb = z_quantile(potenza)
    p1 = p0 + delta
    varianza = p0 * (1 - p0) + p1 * (1 - p1)
    return math.ceil((za + zb) ** 2 * varianza / delta ** 2)

print("z_alpha/2 =", round(z_quantile(0.975), 3), "| z_beta =", round(z_quantile(0.80), 3))
print("n per group, p0=5%, +1 point:   ", n_per_gruppo(0.05, 0.01))
print("n per group, p0=5%, +0.5 points:", n_per_gruppo(0.05, 0.005))

# Output (run after the previous blocks, which define phi and z_test):
# z_alpha/2 = 1.96 | z_beta = 0.842
# n per group, p0=5%, +1 point:    8155
# n per group, p0=5%, +0.5 points: 31231

Output: z_alpha/2 = 1.96, z_beta = 0.842; to detect from 5% to 6% you need 8155 users per group; to detect +0.5 points you need 31231, that is 3.8 times as many (not exactly 4× because the variance V also changes slightly). The formula is an approximation (normal, standard error under H₀ replaced): let us check it.

Check by simulation

import numpy as np

rng = np.random.default_rng(3)

def tasso_di_rifiuto(p0, p1, n, sims=2000):
    rifiuti = 0
    for _ in range(sims):
        xa = rng.binomial(n, p0)
        xb = rng.binomial(n, p1)
        _, pv = z_test_proporzioni(xa, n, xb, n)
        rifiuti += pv < 0.05
    return rifiuti / sims

n = n_per_gruppo(0.05, 0.01)
print("simulated power (real effect +1 point):", tasso_di_rifiuto(0.05, 0.06, n))
print("simulated false positives (A/A, no effect):", tasso_di_rifiuto(0.05, 0.05, n))

# Output (seed 3, after the formula block):
# simulated power (real effect +1 point): 0.7925
# simulated false positives (A/A, no effect): 0.043

How to read the code. tasso_di_rifiuto repeats 2000 simulated experiments: in each one it draws the conversions of the two groups from two binomials (rng.binomial(n, p) counts the successes in n trials with probability p) and applies the test; then it counts the fraction of experiments in which the p-value is below 0.05. If p0 ≠ p1 that fraction is the simulated power; if p0 = p1 it is the false positive rate. In the design of an experiment these very simulations are often run, before touching real traffic.

Output obtained (seed 3): simulated power 0.7925 (target 0.80) and false positives 0.043 (target ≤ 0.05; the value fluctuates because of the finite number of simulations, about ±0.01). The formula holds.

The inverse problem: which effect can I detect with the traffic I have?

The sample formula can be read backwards. Often the constraint is not "how much do I want to detect" but "how many users do I have". With a fixed traffic, what minimum δ can be distinguished (with 80% power)? For small δ, the variance is about V ≈ 2·p₀(1−p₀), and solving the formula for δ: δ ≈ (z_{α/2} + z_β) · sqrt(2·p₀(1−p₀)/n).

import math

za, zb = 1.959964, 0.841621             # normal quantiles for α = 0.05 two-sided and power 0.80
p0 = 0.05
print("users per group | minimum detectable effect (percentage points) | in relative terms")
for n in (2_000, 8_000, 32_000, 128_000):
    delta = (za + zb) * math.sqrt(2 * p0 * (1 - p0) / n)
    print(f"{n:>15,} | {delta * 100:>45.2f} | {delta / p0:>18.1%}")

Output obtained:

users per group | minimum detectable effect (percentage points) | in relative terms
          2,000 |                                          1.93 |              38.6%
          8,000 |                                          0.97 |              19.3%
         32,000 |                                          0.48 |               9.7%
        128,000 |                                          0.24 |               4.8%

How to read the code. For four sample sizes (each time four times the previous one) it computes the minimum detectable effect. Each quadrupling of the data halves the detectable effect: it is the 1/√n law, the same that governs the precision of the sample mean (probability lesson). With 2000 users per group you only see a relative effect of almost 40% with respect to a 5% conversion; to distinguish effects of a few relative percentage points you need hundreds of thousands of users. A table like this, shown to the client before the experiment, avoids the most common misunderstanding: "we ran the test and nothing came out", when the test could not, by construction, see the effect that was expected.

The most widespread trap: peeking

Analogy: the rigged photo finish. If you watch a race at every metre and stop everything at the instant "your horse is in the lead", you will declare it the winner even if the race, completed, sees it last. The race is not rigged: it is the stopping rule.

The p-value is computed under the hypothesis of looking at the result only once, at the end, with a sample fixed in advance. If you look several times and stop at the first occasion on which p < 0.05, you are giving chance many occasions to look like an effect. The false positive rate grows with the number of checks (Armitage, McPherson and Rowe 1969 quantified it for repeated tests).

Let us measure it. We run an A/A test: the two groups have the same conversion probability (5%), so any "victory" is a false positive. We compare: a single check at the end against ten equally spaced checks, stopping at the first p < 0.05.

def peeking(p, n_max=5000, controlli=10, sims=2000):
    falsi = 0
    punti = np.linspace(n_max / controlli, n_max, controlli).astype(int)
    for _ in range(sims):
        a = rng.random(n_max) < p
        b = rng.random(n_max) < p              # same p: no real effect
        ca, cb = np.cumsum(a), np.cumsum(b)
        for k in punti:
            _, pv = z_test_proporzioni(ca[k - 1], k, cb[k - 1], k)
            if pv < 0.05:
                falsi += 1
                break                           # "I saw the significant result: I stop"
    return falsi / sims

print("a single check :", peeking(0.05, controlli=1))
print("ten checks     :", peeking(0.05, controlli=10))

# Output (same generator, run after the previous blocks):
# a single check : 0.0565
# ten checks     : 0.1895

Output obtained (same generator, executed after the previous block): a single check 0.0565; ten checks 0.1895. The nominal level is 0.05, but with ten checks about one experiment in five with no effect at all is declared a winner. (With 2000 simulations each value oscillates by about ±0.01.)

How to read the peeking code. punti contains the equally spaced checks (every 500 users, 10 times). For each of the 2000 simulations it generates two series of 5000 users with the same probability, computes the cumulative conversions with np.cumsum (the cumulative sum gives, for each k, the number of conversions among the first k users) and applies the test at the checkpoints. As soon as a check gives p < 0.05, the simulation "declares the winner" (falsi += 1) and leaves the loop with break. The fraction of experiments declared winners in the absence of any effect is the real false positive rate of that procedure.

Remedies.

1. Fix the sample in advance and look only at the end (the simplest remedy).
2. Sequential tests designed for repeated checks (the tradition starts with Wald, 1945): they correct the threshold at every check.
3. Stricter thresholds for intermediate checks.

Trade-off. Waiting for the end has a cost: if the variant is much worse, you show it to many users for longer. A guardrail monitoring (errors, crashes, complaints) with safety thresholds is allowed and recommended; what is incorrect is using the significance test of the benefit to stop early on success.

Multiple comparisons

If you test 20 metrics or 20 variants with threshold 0.05, you expect about one "significant" result by pure chance even without any effect (with independence, the probability of at least one is 1 − 0.95²⁰ ≈ 0.64: 0.95²⁰ = 0.358). Simplest remedy: Bonferroni correction: use α/m for each test over m comparisons. More conservative than others, but easy to explain and apply. Even better: a single primary metric declared in advance.

A health check: the assignment ratio

One of the most common mistakes, in real systems, is that randomisation does not really produce the planned share (a bug in the assignment code, a filter that discards more users from one group). The check is simple: the observed sizes in the two groups must be compatible with the expected share, for example 50/50. Under H₀ (correct assignment) the number of users in B is a binomial with p = 0.5, so z = (observed − expected)/√(n·0.5·0.5).

import math

def srm(n_a, n_b, quota_a=0.5):
    n = n_a + n_b
    attesi_a = n * quota_a
    z = (n_a - attesi_a) / math.sqrt(n * quota_a * (1 - quota_a))
    p = 2 * (1 - 0.5 * (1 + math.erf(abs(z) / math.sqrt(2))))
    return z, p

for n_a, n_b in ((50_100, 49_900), (50_800, 49_200)):
    z, p = srm(n_a, n_b)
    print(f"A = {n_a}, B = {n_b} -> z = {z:+.2f}, p-value = {p:.4f}")

Output obtained:

A = 50100, B = 49900 -> z = +0.63, p-value = 0.5271
A = 50800, B = 49200 -> z = +5.06, p-value = 0.0000

How to read the code. srm compares the size of group A with the expected one. A difference of 200 users out of 100,000 (50,100 versus 49,900) is compatible with chance: an experiment like that is fine. A difference of 1600 users (50,800 versus 49,200, 1.6% of the total) has a tiny p-value: the assignment is not 50/50, there is a defect in the system, and any result of that experiment must be discarded before even looking at the metric. This is why large experimentation systems make this check the first traffic light (it is called sample ratio mismatch, see Kohavi et al. in the sources).

A minimal vocabulary

• Randomisation: assignment of treatments by lot.
• Null hypothesis H₀: assumption of no real difference.
• p-value: probability, under H₀, of a result at least as extreme as the observed one.
• Power: probability of detecting a real effect of size δ.
• Minimum detectable effect: the smallest difference distinguishable from noise with a given sample.
• Peeking: checking results several times and stopping at the first "significant" one.
• A/A test: experiment with two identical groups, to verify the system.
• Sample ratio mismatch: observed shares different from the planned ones.

The eight entries are ordered along the life cycle of an experiment: first you assign (randomisation), then you formulate and size (null hypothesis, power, minimum effect), then you observe and decide (p-value, peeking), and meanwhile you check the instruments (A/A, SRM). If one of the terms is unclear, go back to the point of the lesson where it appeared.

Typical mistakes

Best practices

• Pre-registration: hypothesis, primary metric, δ, α, power, duration, written before starting.
• A single primary metric, one or two guardrails.
• Randomise on the right unit, check the balance.
• Compute the sample, and if traffic is not enough, give up or change the design.
• Document null results too: an experiment that finds nothing is information.
• Report the confidence interval, not just the p-value: the interval says how large the effect can be.

The first two rules apply before you see the data and prevent choosing the metric afterwards. The next two check that the design is valid, the last two that the result is communicated well. If time is short, pre-register: it is the rule that costs one page and protects against peeking and multiple comparisons at once.

Mini-project

Design an A/B test for an imaginary website with a base conversion of 4%.

1. Decide δ (the minimum effect that repays the costs) and compute n per group with n_per_gruppo.
2. Given a traffic of 1500 visitors per day split in two, compute the days needed.
3. Simulate the experiment 1000 times with a real effect equal to δ and check the power.
4. Add to the code the computation of the 95% confidence interval for D: D ± 1.96 · sqrt(p̂_A(1−p̂_A)/n_A + p̂_B(1−p̂_B)/n_B).

The four steps reproduce an entire experiment, from the decision on δ to the estimate with its uncertainty. Step 3 is the one that gives you confidence in the formula, because you try it on a world whose truth you know.

Challenge

• Easy. Check by hand the computation of z for 200/4000 against 248/4000, step by step.
• Medium. Plot n as a function of δ for p₀ = 5%. What shape does it have? Why?
• Hard. Implement the Bonferroni correction in the peeking code: use 0.05/10 at each check. Does the false positive rate return within 0.05? Does it lose power?
• Expert. Prove that the variance of a Bernoulli is maximal for p = 0.5 and discuss the consequences for n: why do metrics with probability around 0.5 require more users at equal δ?
• Enterprise. Write the pre-registration document template for your organisation (one page) and a written rule on who can stop an experiment and with which criteria.

Hard is the challenge worth the most: it makes you check first-hand that correcting for peeking reduces false positives but has a price in power. The other five challenges (plot, proof, document) consolidate the method, but this is the only one that subjects it to a result you do not know in advance.

Recap

Mind map. Coin → balances confounders → difference D → standard error → z → p-value. Before starting: α, power, δ → n. Never stop at the first "significant".

Cheat sheet.

3 concepts: randomisation; p-value (what it is and what it is not); power and sample. 3 mistakes: peeking; multiple metrics without correction; sample not computed. 3 best practices: pre-registration; one primary metric; confidence interval. 1 exercise: the mini-project of this lesson.

Next lesson: the cloud: where data and models live, and how to reason about costs.

The same series also includes: 05 - Data Culture (the previous lesson), 07 - Cloud and Artificial Intelligence (the next lesson), 28 - Bayesian Statistics: Priors, Posteriors, Conjugates and Why Regularisation Is a Belief (which refers back to this lesson).