17 - Combinatorics: Counting Without Listing

Machine Learning Engineer course · Module 02 · Mathematics and statistics · Lesson 2

Goal. Learn to count in how many ways something can happen without listing them one by one, with four tools (multiplication principle, permutations, arrangements, combinations) that we will derive step by step. At the end you will be able to compute how many hyperparameter combinations a search contains, how many training/test splits exist, and why in a group of 23 people it is more likely than not that two share a birthday.

Myth to dispel. "If the cases are many, the computer tries them all." Combinatorial numbers grow faster than any computer. A deck of 52 cards can be ordered in a number of ways that has 68 digits: no computer will ever be able to list them. Counting with a formula is not a mathematical elegance, it is the only way to know in advance whether a brute-force strategy is possible.

Introduction

In the previous lesson we met the numbers 2ⁿ (the subsets of a set) and the Cartesian product (the combinations of hyperparameters). Here we give a general method. Combinatorics is also the basis of "classical" probability: if all outcomes are equally possible, the probability of an event is favourable cases / possible cases, and to obtain those two numbers you must know how to count.

The path starts from the multiplication principle, goes through permutations, arrangements and combinations, reaches Pascal's triangle and the counts with repetitions, and closes with the birthday paradox, the numbers too large for a float and the applications to machine learning. You only need what you saw about sets: Sets and logic explains why a set of n elements has 2ⁿ subsets.

The multiplication principle

The analogy: the menu

A restaurant offers 3 first courses (soup, pasta, rice) and 2 second courses (fish, meat). How many complete menus? For each first course you have 2 possible second courses: 3 × 2 = 6. The idea can be drawn as a tree: from the root start as many branches as there are choices for the first step; from each branch start as many branches as there are choices for the second; the final cases are the leaves.

Multiplication principle. If a result is obtained with k consecutive choices, and the first can be made in n₁ ways, the second in n₂ ways whatever the first was, and so on, then the possible results are

n₁ × n₂ × … × n_k.

The proof is the tree itself: the leaves after the first step are n₁; each branches into n₂, so the leaves after two steps are n₁·n₂; one proceeds by induction. The important condition is in the words in bold: the number of choices at the second step must not depend on the choice made at the first (the elements may change, their number may not).

The code: counting with the formula and with the list

We verify the formula on a small case and then use it on a huge one, where listing is impossible.

from itertools import product

primi = ["soup", "pasta", "rice"]
secondi = ["fish", "meat"]

menu = list(product(primi, secondi))        # lists all the (first, second) pairs
print(len(menu), "menus listed |", len(primi) * len(secondi), "with the formula")

# Password of 8 characters chosen among 62 symbols (26 lowercase + 26 uppercase + 10 digits):
spazio = 62 ** 8
tentativi_al_secondo = 1e9                  # declared assumption: one billion attempts per second
giorni = spazio / tentativi_al_secondo / 86400
print(f"possible passwords: {spazio:,}")
print(f"time to try them all at 10^9 per second: about {giorni:.1f} days")

Output obtained:

6 menus listed | 6 with the formula
possible passwords: 218,340,105,584,896
time to try them all at 10^9 per second: about 2.5 days

What the code does. product(primi, secondi) produces all the pairs: the first part of the print shows that the list and the formula coincide (6 and 6). The second part applies the same idea to a password: each position has 62 choices, independent of the others, so 62 · 62 · … · 62 = 62⁸. Dividing by an assumption on speed (which we declare: it is not a measurement) we obtain the order of magnitude of the time of a brute-force attack. The point is not the exact number, but that adding one character multiplies the attacker's effort by 62: the length counts more than the "complexity".

The addition principle, and the double-counting trap

If a result is obtained alternatively through cases that do not overlap (either you choose a first course, or you choose a dessert, never both), the ways are added: n₁ + n₂. When the cases overlap, the sum counts twice those in common: it is the inclusion-exclusion of the previous lesson. Remember the practical rule: "and" (one choice after the other) → multiply; "or" (disjoint alternative cases) → add.

Permutations: ordering all

The analogy: the queue

You have n people and n places in a queue. Who takes the first place? Any of the n. The second? One of the n − 1 remaining. The third: n − 2... until the last place, where only one person is left. By the multiplication principle the different queues (the permutations of n objects) are

n! = n · (n−1) · (n−2) · … · 2 · 1

(read "n factorial"). By convention 0! = 1: there is only one way to order zero objects (the empty queue), and it is consistent with the rule n! = n · (n−1)! applied to n = 1, which gives 1! = 1 · 0!.

The growth is dizzying, and the code makes it tangible.

import math

for n in (5, 10, 20, 52):
    print(f"{n:>2}! = {math.factorial(n):,}")

print("digits of 52!:", len(str(math.factorial(52))))

Output obtained:

 5! = 120
10! = 3,628,800
20! = 2,432,902,008,176,640,000
52! = 80,658,175,170,943,878,571,660,636,856,403,766,975,289,505,440,883,277,824,000,000,000,000
digits of 52!: 68

What the code does. math.factorial computes with exact integers (Python has no size limit for integers), so 52! is printed in full: it has 68 digits. The permutations of a deck of cards are therefore about 8 × 10⁶⁷: if you shuffle well, with practically certain probability you produce an order that nobody, at any time, has ever seen before.

Stirling: estimating a factorial

For large n there is a famous estimate, Stirling's formula (1730):

n! ≈ √(2πn) · (n/e)ⁿ

We do not prove it (it requires tools of calculus that we will see later), but we check how precise it is.

for n in (5, 10, 20, 50):
    esatto = math.factorial(n)
    stirling = math.sqrt(2 * math.pi * n) * (n / math.e) ** n
    errore = (stirling / esatto - 1) * 100
    print(f"n = {n:>2}: relative error of the Stirling estimate = {errore:6.3f} %")

Output obtained:

n =  5: relative error of the Stirling estimate = -1.651 %
n = 10: relative error of the Stirling estimate = -0.830 %
n = 20: relative error of the Stirling estimate = -0.416 %
n = 50: relative error of the Stirling estimate = -0.167 %

What the code does. For each n it compares the exact value with the estimate and prints the percentage gap. The error falls as n grows: already at n = 10 the estimate is off by less than 1%, at n = 50 by less than 0.2%. It is the reason why one can work with log n! without computing the whole factorial, as we will see in the section on huge numbers.

Arrangements: ordering only some

If instead of ordering all n objects you choose k of them and the order counts (a podium: first, second, third), the places are k and the candidates decrease by one at each step:

n · (n−1) · … · (n−k+1) = n! / (n−k)!

The factor (n−k)! in the denominator "cancels" the tail of n! that is not needed: n! = n·(n−1)·…·(n−k+1) · (n−k)!.

If instead the same element can be repeated (the digits of a PIN), at each place you always have n choices: nᵏ.

Combinations: choosing without order

The analogy: the podium and the committee

The podium has a first, a second and a third; the committee of three people does not: Ada-Bea-Carlo and Carlo-Ada-Bea are the same committee. To pass from arrangements to combinations it is enough to ask: how many times did I count the same committee? Each group of k people appears in all its k! orderings (the queues of its members). Dividing gives the binomial coefficient:

C(n, k) = n! / (k! · (n−k)!)

(read "n choose k" and also written (n k)).

Properties, with proof

1. Symmetry: C(n, k) = C(n, n−k). Choosing the k that go in is equivalent to choosing the n−k that stay out. (From the formula: the denominator k!(n−k)! is symmetric.)
2. Pascal's identity: C(n, k) = C(n−1, k−1) + C(n−1, k). Fix a "special" element: the groups of k that contain it are completed by choosing the other k−1 among the n−1 remaining (C(n−1, k−1) ways); those that do not contain it are chosen entirely among the n−1 remaining (C(n−1, k) ways). The two classes are disjoint and cover all the cases: they add up.
3. Sum of a row: C(n,0) + C(n,1) + … + C(n,n) = 2ⁿ. The left-hand side counts the subsets by size, the right-hand one counts them all with the "in or out" method of the previous lesson. They are two counts of the same set.

The three properties are the same fact seen from different angles: counting groups is equivalent to counting their complement (symmetry), a group either contains a given element or does not (Pascal), and the sum of the possible sizes exhausts all the subsets. The most useful in practice is Pascal's, because it lets you compute a coefficient with sums of integers alone, without ever forming a huge factorial.

Pascal's triangle

Pascal's identity allows building the coefficients without factorials: each number is the sum of the two above it.

def triangolo(righe):
    """Builds the first `righe` rows of Pascal's triangle using addition only."""
    t = [[1]]
    for n in range(1, righe):
        precedente = t[-1]
        nuova = [1] + [precedente[k - 1] + precedente[k] for k in range(1, n)] + [1]
        t.append(nuova)
    return t

t = triangolo(8)
for r in t:
    print(r)

print("row 6 matches math.comb:", t[6] == [math.comb(6, k) for k in range(7)])
print("sum of row 6 = 2^6:", sum(t[6]) == 2 ** 6)

Output obtained:

[1]
[1, 1]
[1, 2, 1]
[1, 3, 3, 1]
[1, 4, 6, 4, 1]
[1, 5, 10, 10, 5, 1]
[1, 6, 15, 20, 15, 6, 1]
[1, 7, 21, 35, 35, 21, 7, 1]
row 6 matches math.comb: True
sum of row 6 = 2^6: True

What the code does. Each new row begins and ends with 1 and, in the middle, adds in pairs the elements of the previous row (it is Pascal's identity translated into a comprehension). The last two lines verify two facts: the triangle coincides with math.comb (the coefficients computed with the formula) and the sum of a row is 2ⁿ.

The binomial theorem

Expanding (a + b)ⁿ means multiplying n factors (a + b). From each factor you choose a or b; the term aᵏ · bⁿ⁻ᵏ appears as many times as there are ways to choose which k factors supply the a, that is C(n, k):

(a + b)ⁿ = Σₖ C(n, k) · aᵏ · bⁿ⁻ᵏ (sum over k from 0 to n).

For a = b = 1 one finds 2ⁿ again; for a = p, b = 1−p one finds that the probabilities of the binomial distribution sum to 1, as you will see in the lesson on random variables.

With repetitions: anagrams and stars and bars

Anagrams

How many words are obtained by permuting the letters of MAMMA? If the letters were all distinct there would be 5! = 120. But the 3 Ms are indistinguishable: each word has been counted 3! times (the swaps among the Ms), and each swap of the 2 As another 2! times. So

5! / (3! · 2!) = 10.

In general with n objects of which n₁ equal of one type, n₂ of another, etc., the distinct permutations are the multinomial coefficient n! / (n₁! · n₂! · …).

Stars and bars: choosing with repetition

In how many ways can k identical balls be distributed into n boxes (some may remain empty)? Lay out the k balls (the "stars") in a row and insert n − 1 separators (the "bars") to divide the boxes. A sequence is determined by which of the k + n − 1 positions are bars: C(n+k−1, k).

from itertools import permutations

n_scatole, k_palline = 3, 5
formula = math.comb(n_scatole + k_palline - 1, k_palline)

# direct count: the distributions are tuples (a, b, c) of integers >= 0 with a + b + c = 5
diretto = sum(1 for a in range(k_palline + 1) for b in range(k_palline + 1 - a))
print("stars and bars:", formula, "| direct count:", diretto)

anagrammi = len(set(permutations("MAMMA")))
print("anagrams of MAMMA:", anagrammi, "| formula 5!/(3!2!):", math.factorial(5) // (math.factorial(3) * math.factorial(2)))

Output obtained:

stars and bars: 21 | direct count: 21
anagrams of MAMMA: 10 | formula 5!/(3!2!): 10

What the code does. The first comparison counts the triples (a, b, c) of non-negative integers with sum 5: for each a and b the value of c is determined (5 − a − b), so it is enough to count the valid pairs (a, b). The second really generates the 120 permutations of "MAMMA" and turns them into a set to eliminate the duplicates (it is the sets lesson working for us). Both formulas find 21 and 10.

Application to machine learning. The bootstrap sample (we will meet it in the statistics lesson) draws n observations with repetition from n: the distinct possible samples, ignoring the order, are C(2n−1, n). For n = 10 they are already 92,378.

The birthday paradox

In a group of n people, what is the probability that at least two share a birthday? (Simplifying assumptions: 365 days, all equally likely.)

Strategy: count the complement. "At least two" is hard to count (two pairs? three people together?); "no one in common" is easy. All the possible birthdays are 365ⁿ (multiplication principle: each person has 365 choices). The cases without coincidences are the arrangements: the first person has 365 days, the second 364, ... the n-th 365−n+1. Therefore

P(at least one coincidence) = 1 − (365 · 364 · … · (365−n+1)) / 365ⁿ.

import numpy as np

def p_coincidenza(n, giorni=365):
    return 1 - math.perm(giorni, n) / giorni ** n

for n in (10, 20, 23, 30, 50, 70):
    print(f"n = {n:>2}: P(at least two equal birthdays) = {p_coincidenza(n):.4f}")

# check by simulation: 200000 groups of 23 people
rng = np.random.default_rng(0)
gruppi = rng.integers(0, 365, size=(200_000, 23))
ordinati = np.sort(gruppi, axis=1)
con_doppio = (ordinati[:, 1:] == ordinati[:, :-1]).any(axis=1)
print("simulation n = 23:", round(float(con_doppio.mean()), 4), "| formula:", round(p_coincidenza(23), 4))

Output obtained:

n = 10: P(at least two equal birthdays) = 0.1169
n = 20: P(at least two equal birthdays) = 0.4114
n = 23: P(at least two equal birthdays) = 0.5073
n = 30: P(at least two equal birthdays) = 0.7063
n = 50: P(at least two equal birthdays) = 0.9704
n = 70: P(at least two equal birthdays) = 0.9992
simulation n = 23: 0.5077 | formula: 0.5073

What the code does. math.perm(giorni, n) computes exactly the arrangements 365!/(365−n)! without building them. The simulation generates 200,000 groups of 23 random dates (one row per group), sorts each row and checks whether two adjacent dates are equal: after sorting, a duplicate is always adjacent. The fraction of rows with a duplicate estimates the probability, and coincides with the formula within the noise of the simulation.

The result is surprising: already with 23 people the probability exceeds 50%. Intuition is wrong because it compares "me against the others" (22 pairs) instead of all the possible pairs, which are C(23, 2) = 253. The number of pairs grows as the square of the group, and the probability of coincidence with it. It is the same phenomenon that makes the values of a hash function collide much sooner than expected (the collisions, hash tables lesson).

Huge numbers and numerical computing

Factorials soon exceed the capacity of a float. Python handles exact integers, but NumPy and the scientific libraries work in floating point (lesson 4 of module 01), where the maximum number is about 1.8 × 10³⁰⁸.

grande = math.factorial(200)
print("200! has", len(str(grande)), "digits")
try:
    float(grande)
except OverflowError as e:
    print("conversion to float:", e)

log10_fatt = math.lgamma(201) / math.log(10)      # log10(200!) without computing 200!
print("log10(200!) =", round(log10_fatt, 4), "| digits =", int(log10_fatt) + 1)

# C(1000, 500) exact versus via logarithms
esatto = math.comb(1000, 500)
via_log = math.exp(math.lgamma(1001) - 2 * math.lgamma(501))
print("C(1000,500) has", len(str(esatto)), "digits; via logarithms ≈ 10^", round(math.log10(via_log), 2))

Output obtained:

200! has 375 digits
conversion to float: int too large to convert to float
log10(200!) = 374.8969 | digits = 375
C(1000,500) has 300 digits; via logarithms ≈ 10^ 299.43

What the code does. 200! exists as an integer (375 digits) but not as a float: the conversion raises OverflowError. The solution is to work with logarithms: math.lgamma(n+1) returns ln(n!) (the gamma function extends the factorial: Γ(n+1) = n!), and a product of huge terms becomes a sum of small logarithms. It is a trick used everywhere in probability and machine learning (the log-likelihood). For C(1000, 500) the last computation shows that the result via logarithms coincides in order of magnitude with the exact integer.

Applications in machine learning

• Number of training/test splits: with n examples and a test of m, there are C(n, m) different splits. For n = 100, m = 25 the number has 24 digits: two scientists with the same dataset and a different random_state evaluate different models, and this is the source of the variability of results.
• Subsets of features: 2ᵈ (previous lesson).
• Hyperparameter grid: product of the numbers of values, n₁·n₂·…, multiplied by the k folds of cross-validation.
• Decision trees and partitions: the number of ways to divide n examples into two groups is 2ⁿ⁻¹ − 1: no algorithm can examine them all, and this is why trees use greedy choices (step by step).

The four items share the same question: how many cases exist before you start searching. The first explains why two results obtained with different seeds cannot be compared to the last decimal digit. The second and the fourth say that exploring everything is impossible and a strategy is needed. The third, the grid, is the only one you can compute exactly before launching the job, and it is the one in the listing that follows.

n, m = 100, 25
print("possible train/test splits (n=100, test=25):", f"{math.comb(n, m):,}")

valori = [4, 3, 5, 2]                       # 4 hyperparameters with 4, 3, 5, 2 values
griglia = math.prod(valori)
print("configurations:", griglia, "| trainings with 5 folds:", griglia * 5)

Output obtained:

possible train/test splits (n=100, test=25): 242,519,269,720,337,121,015,504
configurations: 120 | trainings with 5 folds: 600

What the code does. math.comb(100, 25) gives the number of distinct test sets, math.prod multiplies the values per hyperparameter (multiplication principle) and the factor 5 counts the validation folds. A grid that seems small (120 configurations) already requires 600 trainings.

Typical mistakes (symptom, cause, remedy)

• Order yes or no? Symptom: result too large by a factor k!. Remedy: ask yourself "does swapping two elements change the result?".
• Repetitions yes or no? Symptom: confusing n!/(n−k)! with nᵏ. Remedy: ask whether the same element can be chosen twice.
• Counting twice (overlapping cases). Remedy: disjoint cases, or inclusion-exclusion.
• "At least one" counted directly. Remedy: 1 − none (as for the birthdays). Example: the 4-digit PINs with at least one repetition are 10⁴ − 10·9·8·7 = 4960.
• Floats instead of integers. Symptom: inf or OverflowError with factorials beyond 170. Remedy: exact integers (math.comb) or logarithms (lgamma).
• Swapping C(n, k) with C(n, k) · k!. Remedy: know which of the two you need.
• Off-by-one errors (the ends): range(1, n) does not include n. Remedy: verify with a small case.

Almost all these mistakes come from a single omission: not having declared, before writing the formula, whether order matters and whether the elements can be repeated. The four possible combinations of answers correspond to the four formulas in the lesson, and choosing the formula becomes a mechanical operation.

How to work in practice

Before using a formula on a large case, run it on a tiny one that you can list with itertools: if the two numbers do not match, the error is in the formula and not in the data. Write in one line what you are counting ("ways to choose 3 people out of 10, without order, without repetition") and draw the tree when you do not know whether a product is legitimate. Check the order of magnitude before launching any exhaustive search, and prefer math.comb, math.perm and math.prod, which are exact and readable.

The vocabulary is short. The factorial n! is the product n·(n−1)·…·1, with 0! = 1. A permutation orders all the elements, an arrangement orders k chosen among n, a combination chooses k without order and its count is the binomial coefficient C(n, k). The complement is everything outside a set, and brute force consists in trying every case.

Mini-project

1. Write a function numero_configurazioni(griglia: dict) that, given a dictionary {name: list of values}, returns the number of configurations, and verify that it coincides with len(list(product(...))) on a small grid.
2. Estimate in years the time to try a grid of 8 hyperparameters with 10 values each, if each training takes 30 seconds (declared assumption), with 5 validation folds. Compare it with a budget of 100 random configurations.
3. Compute the probability of the birthday paradox for a year of 365 days and of 12 months (groups of n people, birth months), and explain the difference.

The first point is a one-line function with math.prod. In the second, the product of the values gives the configurations, the factor 5 the validation folds and the 30 seconds the total time: the order of magnitude you get is centuries, and this is why the budget of 100 random configurations is not a fallback but the only practicable choice. The third point changes only the number of "days" (12 instead of 365) and shows that the 50% threshold drops to very few people.

A worked challenge: counting the poker hands with one pair

Let us test the formulas on a classic problem. A hand is a set of 5 cards drawn from a deck of 52 (13 values, 4 suits); the order does not count. How many hands contain exactly one pair (two cards of the same value, and the other three of values all different from each other and from the pair)?

Construction, step by step (multiplication principle).

1. Choose the value of the pair: 13 ways.
2. Choose which two suits (among 4) have the cards of the pair: C(4, 2) = 6 ways.
3. Choose the three different values for the other cards, among the 12 remaining: C(12, 3) = 220 ways (it is a choice without order).
4. For each of those three values choose the suit (4 possibilities): 4 · 4 · 4 = 64 ways.

Total: 13 · 6 · 220 · 64 = 1,098,240 hands. The possible hands in all are C(52, 5) = 2,598,960, so the classical probability is favourable cases / possible cases ≈ 0.4226.

import math
import numpy as np

favorevoli = 13 * math.comb(4, 2) * math.comb(12, 3) * 4 ** 3
possibili = math.comb(52, 5)
print("hands with exactly one pair :", f"{favorevoli:,}")
print("possible hands              :", f"{possibili:,}")
print("exact probability           :", round(favorevoli / possibili, 5))

# check by simulation: 300,000 random hands
rng = np.random.default_rng(1)
prove = 300_000
mazzo = np.arange(52)
valori = mazzo % 13                                    # 13 values repeated across 4 suits
una_coppia = 0
for _ in range(prove):
    mano = rng.choice(mazzo, size=5, replace=False)
    conteggi = np.bincount(valori[mano], minlength=13)
    if sorted(conteggi[conteggi > 0]) == [1, 1, 1, 2]:  # one value twice, three values once
        una_coppia += 1
print("simulated frequency         :", round(una_coppia / prove, 5))

Output obtained:

hands with exactly one pair : 1,098,240
possible hands              : 2,598,960
exact probability           : 0.42257
simulated frequency         : 0.42254

How to read the code. The first three lines apply the formula just built (math.comb for the binomial coefficients, 4 ** 3 for the suits). The simulation draws 300,000 hands: rng.choice(mazzo, size=5, replace=False) draws five cards without putting back (without repetitions), valori = mazzo % 13 associates to each card its value (the cards 0, 13, 26, 39 have the same value), np.bincount counts how many cards of the hand have each value, and the condition sorted(...) == [1, 1, 1, 2] recognises exactly the configuration "one value twice and three values only once" (it excludes three of a kind, two pairs, full house, four of a kind). The simulated frequency approaches the theoretical value within the small noise of a simulation.

Note the structure of the count: steps 1 and 3 are choices without order (combinations), step 4 has three independent choices with 4 options each (a power), and everything is multiplied because each choice can be made whatever the previous one was. A frequent error is counting the same hand twice, for example by first choosing "two values" and then deciding which of the two is the pair: the construction of the example avoids the problem because every hand is built in a single way. Checking this uniqueness is the test that every combinatorial count should pass. The method holds as a general rule: build the object step by step, count the choices at each step, multiply; then, when you can, verify with a simulation.

Challenge

• Easy. How many licence plates exist with 2 letters (26), 3 digits and 2 letters? Verify the multiplication principle with a reduced subset (2 letters, 1 digit).
• Medium. Prove with Pascal's identity and induction that Σₖ C(n,k) = 2ⁿ, and verify for n from 0 to 15.
• Hard. How many 5-card hands contain a full house (three of a kind and a pair of different values)? Count with the multiplication principle (value of the three, suits of the three, value of the pair, suits of the pair) and verify with a simulation like the one worked in the lesson.
• Expert. Prove that the probability of a birthday coincidence is approximated by 1 − exp(−n(n−1)/(2·365)) (hint: ln(1−x) ≈ −x for small x) and compare numerically with the exact formula.
• Enterprise. For a real project, compute how many combinations the exhaustive search of its hyperparameters would contain and declare the computing budget; choose and justify a reduced search.

The five challenges grow in abstraction, not in amount of code: the first two consolidate the multiplication principle and the sum of a row of the triangle, the third and the fourth force you to count and to prove, the last asks you to decide how much budget to spend. If you pick only one, pick Hard: the full house is built with the same four steps as the hands with one pair.

To practise on the simulated chess club tournament you will find the skeleton with the parts to complete and, at the bottom, the commented solutions in esercizi.py: it compares the round-robin counts with the itertools enumeration, builds the binomial coefficient with Pascal's recurrence and checks stars and bars by enumeration. It requires only Python 3.12 or later and is launched with python3 esercizi.py; the file was not executed for this article.

What you can do now

You can choose among the four formulas by asking yourself whether order matters and whether the elements repeat, estimate in advance whether an exhaustive hyperparameter search is practicable, count "at least one" by passing through the complement, and work with huge factorials through exact integers or logarithms. The next step is probability, which uses these counts as numerator and denominator, and right after it the random variables for the cases where the outcomes are not equally likely. As a cross-cutting review, statistics will use the bootstrap you met here as a count with repetition.

The same series also includes: 15 - scikit-learn: Estimators, Pipelines and Validation, 21 - Calculus 1: Limits and Derivatives (which refer back to this lesson).