15 - scikit-learn: Estimators, Pipelines and Validation

A classification model that guesses 92% of the cases looks like a success. If the data on which you measured it are pure noise, where by construction no model can exceed 50%, that 92% is instead a methodological error, and you will see it produced by code that runs without a single error message. This lesson gives you the tools to avoid the trap: scikit-learn's uniform interface (fit, predict, transform), the separation between training and test data, cross-validation and the Pipeline, which makes data leakage structurally impossible. I do not merely describe the leak: I measure it with an experiment in which the true answer is known in advance.

scikit-learn is the reference library for "classical" machine learning on tabular data: linear models, trees, nearest neighbours, clustering, dimensionality reduction, and everything needed to prepare the data and evaluate the results. Its strength is not a single algorithm but a uniform interface, which you learn once and which holds for hundreds of objects. You need Python 3, NumPy, pandas and scikit-learn (pip install scikit-learn pandas in a virtual environment); the listings were run with scikit-learn 1.8.0, while the documentation pages cited below are those of the "stable" version at the time of consultation (1.9.1). The listings of the lesson form a single program, to be run in the order in which they appear: each one reuses the imports and variables of the previous one, except the last (the solution of the mini-project), which is self-contained. If you come from the lessons on NumPy and Pandas you already have everything you need, and if you want to review how to look at the data before modelling there is the lesson on Matplotlib.

The API: three verbs

An estimator is an apprentice with an empty notebook. The verb fit(X, y) makes it study the examples: at the end the notebook contains what it has learned (coefficients, thresholds, means). The verb predict(X) makes it answer on new examples, consulting the notebook without modifying it. The verb transform(X), specific to transformers such as the scaler, applies what has been learned to modify the data instead of predicting a label. Four conventions help to read other people's code. The hyperparameters, the knobs you choose yourself such as the regularisation strength C, are passed to the constructor: LogisticRegression(C=1.0). Everything that is learned from the data has a name ending with an underscore (coef_, mean_, classes_), and the scikit-learn developer guide states it as a rule: if an attribute with the underscore exists, the object has already been trained. X has shape (n_examples, n_features) and y has shape (n_examples,): rows are examples, columns are features, the same convention as NumPy. Finally fit returns the object itself, so calls can be chained: modello.fit(X, y).predict(X_nuovo).

This uniformity is a deliberate design choice, described by the authors themselves in the paper on the API design of scikit-learn (Buitinck et al., 2013). The practical advantage is that you can replace one algorithm with another by changing a single line, and that the surrounding tools (Pipeline, cross-validation, hyperparameter search) work with any object that respects the three conventions.

Training and test: the exam with questions never seen

We work on the breast-cancer diagnostic dataset included in the library: 569 cases, 30 numeric measurements per case, 212 malignant and 357 benign. It is a teaching dataset, used here only to show the method. The idea of the separation is that of the exam: if I examine you with the same exercises you have already seen I measure memory; to measure the ability to generalise you need new questions. So we split the data into two sets, one to train and one kept aside for the final judgement only. In the listing stratify=y keeps the same proportion between the classes in the two sets, and random_state=0 makes the split repeatable (if you omit it, every run gives a different result). Before the real model I build the baseline, a DummyClassifier with the most_frequent strategy that always answers with the most frequent class (the documentation lists prior as the default; here I choose the strategy explicitly).

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix

X, y = load_breast_cancer(return_X_y=True)
print("shape of X:", X.shape, "| classes (0=malignant, 1=benign):", np.bincount(y))

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
print("training:", X_tr.shape[0], "examples | test:", X_te.shape[0], "examples")

base = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)
print("baseline (always the most frequent class):", round(base.score(X_te, y_te), 3))

modello = LogisticRegression(max_iter=5000).fit(X_tr, y_tr)
print("logistic regression, accuracy on the test:", round(modello.score(X_te, y_te), 3))

Output obtained (Python 3.14.4, NumPy 2.3.5, pandas 3.0.3, scikit-learn 1.8.0; run twice with identical output):

shape of X: (569, 30) | classes (0=malignant, 1=benign): [212 357]
training: 426 examples | test: 143 examples
baseline (always the most frequent class): 0.629
logistic regression, accuracy on the test: 0.937

The baseline is worth 0.629: always answering "benign" is enough on its own for 63% of the cases, and this is the floor. Logistic regression reaches 0.937, and only read against that 63% is the result a real victory; a 94% on a problem where 63% is free is less impressive. The random split is not always the right one. If the data have a time order (forecasts on historical series), splitting at random lets the model see the future: the documentation describes TimeSeriesSplit for this, in which successive training sets contain the previous ones and the test always comes afterwards. If several rows belong to the same person or the same machine, an example in the training set and its "sibling" in the test set distort the judgement, and the cross-validation guide indicates GroupKFold, which guarantees that the same group never appears in both sets.

Beyond accuracy

Accuracy is a single figure. The confusion matrix shows where the mistakes are: the rows are the true class, the columns the predicted class, and the diagonal contains the right answers.

pred = modello.predict(X_te)
print(confusion_matrix(y_te, pred))

Output obtained (Python 3.14.4, NumPy 2.3.5, pandas 3.0.3, scikit-learn 1.8.0; run twice with identical output):

[[49  4]
 [ 5 85]]

On 143 test cases row 0 is the "malignant" class: 49 recognised and 4 declared benign, which is the costliest error. Row 1 is the "benign" class: 5 declared malignant and 85 recognised. The errors have different costs: declaring a malignant tumour benign does not cost the same as the opposite, and the decision threshold is chosen on the basis of those costs, not of the average accuracy. With the matrix you can compute precision, recall and F1, which the next exercise has you derive by hand.

Why we scale (and when it is not needed)

Many models are sensitive to the scale of the features. Nearest neighbours compute distances: if one feature is worth thousands and another tenths, the first decides alone. The listing compares the k-nearest-neighbours model without and with standardisation (subtract the mean and divide by the standard deviation) on the wine dataset, also included in the library: 178 samples, 13 chemical measurements, three varieties. The evaluation uses a 5-fold validation, which I explain later; for now it is enough to know that it gives the mean of five accuracies. The scaler sits inside a make_pipeline, and the reason is the theme of the next section.

from sklearn.datasets import load_wine
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

Xw, yw = load_wine(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

grezzo = cross_val_score(KNeighborsClassifier(), Xw, yw, cv=cv)
scalato = cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier()), Xw, yw, cv=cv)
print("value ranges (min, max) per column:",
      [(round(float(Xw[:, j].min()), 1), round(float(Xw[:, j].max()), 1)) for j in (0, 1, 12)])
print("k neighbours without scaling:", round(grezzo.mean(), 3))
print("k neighbours with scaler:", round(scalato.mean(), 3))

Output obtained (Python 3.14.4, NumPy 2.3.5, pandas 3.0.3, scikit-learn 1.8.0; run twice with identical output):

value ranges (min, max) per column: [(11.0, 14.8), (0.7, 5.8), (278.0, 1680.0)]
k neighbours without scaling: 0.663
k neighbours with scaler: 0.961

Column 12 (proline) goes from 278 to 1680; column 1 from 0.7 to 5.8, column 0 from 11.0 to 14.8. Without scaling, the distance between two wines is decided almost entirely by proline and the other twelve measurements count very little: the accuracy is 0.663. With the scaler it rises to 0.961, without changing either the data or the model. Decision trees and forests use neither distances nor linear combinations: for them the scale is irrelevant, and scaling is not needed (it does no harm, but it does not help). Regularised linear models, nearest neighbours, support vector machines and neural networks instead want data on the same scale. There is no "always scale" rule: it depends on the algorithm, and the comparison in the mini-project at the end shows three.

Data leakage

The student of the exam has, hidden, a slip with the answers: he gets top marks, and the grade measures nothing. In machine learning the slip is any information about the test that ends up, directly or indirectly, in the training. The "Common pitfalls" page of the documentation defines it as the use, in building the model, of information that would not be available at prediction time, and lists its effect: overly optimistic estimates, even in cross-validation, and worse performance in production. The general rule of the same page is simple: never call fit on the test data. It is the most frequent and most insidious source of error in practice, because it produces beautiful results in the lab and disasters after release.

An experiment with a known answer

Let us build a problem in which the only possible accuracy is 50%: 50 examples, 1000 features of pure Gaussian noise, binary labels drawn at random and independent of everything. No algorithm can do better than chance, because there is nothing to learn. (The documentation page cited above builds an analogous experiment with 200 examples and 10,000 features; here the parameters are mine.) I compare two procedures. The wrong but very widespread one selects the 20 features "most correlated with the label" using all the data and then does the cross-validation on the selected subset. The right one puts the selection inside a Pipeline, so that it is redone inside each fold using only the training data of that fold. The experiment is repeated on 20 independent noise samples (the for seme in range(20) loop), so as not to trust a single lucky case.

from sklearn.feature_selection import SelectKBest, f_classif

n, p = 50, 1000
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
sbagliato, giusto = [], []

for seme in range(20):                                   # 20 independent experiments
    r = np.random.default_rng(seme)
    Xn = r.normal(size=(n, p))                           # pure noise
    yn = r.integers(0, 2, n)                             # random labels

    # WRONG: the selection sees ALL the data, then we validate
    Xsel = SelectKBest(f_classif, k=20).fit_transform(Xn, yn)
    sbagliato.append(cross_val_score(LogisticRegression(), Xsel, yn, cv=cv).mean())

    # RIGHT: the selection is a step of the pipeline, redone inside each fold
    pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
    giusto.append(cross_val_score(pipe, Xn, yn, cv=cv).mean())

print("accuracy expected from chance     : 0.500")
print("wrong procedure (mean over 20)   :", round(float(np.mean(sbagliato)), 3))
print("right procedure (mean over 20)   :", round(float(np.mean(giusto)), 3))

Output obtained (Python 3.14.4, NumPy 2.3.5, pandas 3.0.3, scikit-learn 1.8.0; run twice with identical output):

accuracy expected from chance     : 0.500
wrong procedure (mean over 20)   : 0.927
right procedure (mean over 20)   : 0.507

The wrong procedure "discovers" a mean accuracy of 0.927 in data that are pure noise; the right one goes back to 0.507, that is to the truth. The reason is this: with 1000 features independent of the label and only 50 examples, by pure chance some features seem correlated with the label in that sample. If you choose the 20 most correlated by looking at all the data, you have chosen precisely those in which chance worked in your favour even on the fold you will use to validate, and the validation is no longer independent. In the right procedure the selection sees only the training part of each fold; on the validation fold the features chosen by chance have no power and the accuracy goes back to 50%. The figure summarises the two orders.

Hastie, Tibshirani and Friedman call this situation, in The Elements of Statistical Learning, "the wrong way and the right way to do cross-validation" (section 7.10; I did not reopen the book for this lesson). The golden rule that follows: every step that learns something from the data (scaler, feature selection, imputation of missing values, dimensionality reduction, encoding) must be trained only on the training set and only applied to the test set. In scikit-learn this is obtained by putting all the steps inside a Pipeline. The cost is nil and the risk avoided is high; a case in which the rule is not enough is that of "twin" rows, treated in the exercise.

Pipeline and ColumnTransformer

A Pipeline is a chain of transformers followed by a final model, which behaves like a single estimator: fit trains each step in sequence, each transformer learning from the result of the previous one, and predict applies everything in order. With cross-validation, each fold retrains the whole chain from scratch, and this is why leakage is structurally impossible. Real data, however, are mixed: numbers, categories, holes. The ColumnTransformer applies a different pipeline to each group of columns. The listing builds a synthetic table of 1200 customers of a subscription service (data generated by the code, not real) with the goal of predicting who cancels. The three numeric columns go through imputation with the median and through standardisation; the two categorical ones through one-hot encoding. For the ColumnTransformer the columns not named are discarded (the remainder parameter is 'drop' if you do not write it, as the documentation states): here I list them all, so I lose none.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder

rng = np.random.default_rng(11)
m = 1200
clienti = pd.DataFrame({
    "eta": rng.integers(18, 80, m).astype(float),
    "canone": np.round(rng.normal(30, 10, m).clip(5, None), 2),
    "mesi_attivi": rng.integers(1, 60, m),
    "piano": rng.choice(["base", "plus", "pro"], m, p=[0.5, 0.3, 0.2]),
    "canale": rng.choice(["web", "negozio", "telefono"], m),
})
clienti.loc[rng.random(m) < 0.08, "eta"] = np.nan                # 8 % of ages missing

# churn depends on a high fee, few active months, base plan (a relationship we invented)
punteggio = 0.15 * (clienti["canone"] - 30) - 0.10 * (clienti["mesi_attivi"] - 30) \
            + 1.5 * (clienti["piano"] == "base") - 1.0
prob = 1 / (1 + np.exp(-punteggio.to_numpy()))     # to_numpy: from here on we work with arrays without an index
disdetta = (rng.random(m) < prob).astype(int)
print("cancellation rate:", round(float(disdetta.mean()), 3))

numeriche = ["eta", "canone", "mesi_attivi"]
categoriche = ["piano", "canale"]

prepara = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeriche),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categoriche),
])
pipeline = make_pipeline(prepara, LogisticRegression(max_iter=1000))

X_tr, X_te, y_tr, y_te = train_test_split(clienti, disdetta, test_size=0.25, stratify=disdetta, random_state=0)
pipeline.fit(X_tr, y_tr)
print("accuracy on the test:", round(pipeline.score(X_te, y_te), 3))
print("baseline (most frequent class):", round(max(y_te.mean(), 1 - y_te.mean()), 3))
print("pipeline steps:", [nome for nome, _ in pipeline.steps])
print("columns after the transformation:", pipeline[:-1].transform(X_te).shape[1])

Output obtained (Python 3.14.4, NumPy 2.3.5, pandas 3.0.3, scikit-learn 1.8.0; run twice with identical output):

cancellation rate: 0.452
accuracy on the test: 0.817
baseline (most frequent class): 0.55
pipeline steps: ['columntransformer', 'logisticregression']
columns after the transformation: 9

The model clearly beats the baseline (0.817 against 0.55): it is rediscovering from the data the relationship that I invented in the code (cancellation more likely with a high fee, few active months, base plan). The number of columns after the transformation, 9, confirms the encoding: 3 numeric, 3 for the plan and 3 for the channel. Why one-hot instead of writing base equal to 0, plus equal to 1, pro equal to 2? Because the numbers would impose an order and a distance ("pro is double plus") that do not exist; with separate columns the model assigns each category its own weight.

handle_unknown="ignore" avoids an error in production if a category never seen in training appears. The imputation computes the median on the training set, the scaler computes mean and standard deviation on the training set, and on the test those numbers are applied without recomputing them: by hand it is very easy to get it wrong (fit_transform on the test instead of transform), with the pipeline the mistake cannot be made. Note also the use of pandas for the customer table, already seen in the lesson on Pandas, and of .to_numpy() to move to an array without an index.

Cross-validation and the choice of hyperparameters

A single test set has a defect: the result depends on which examples happened to land there, and with little data the estimate fluctuates a lot. k-fold cross-validation divides the data into k groups of equal size. For k times it uses one as validation and the remaining k−1 as training. At the end there are k scores, and the estimate is their mean, with the standard deviation as an indication of how stable it is. With n examples each fold has n/k; each example appears exactly once in validation and k−1 times in training; each model is trained on n·(k−1)/k examples. For k = 5 80% of the data trains and 20% evaluates, in rotation.

The listing computes the five scores with cross_val_score on the training set only and then reproduces them by hand, with a loop that calls fit and score fold by fold, to show that there is no magic. The np.allclose comparison prints True when the two ways coincide. StratifiedKFold with shuffle=True and a fixed random_state is the most common form: it keeps the class proportions in every fold and makes the result repeatable.

punteggi = cross_val_score(pipeline, X_tr, y_tr, cv=StratifiedKFold(5, shuffle=True, random_state=0))
print("scores of the 5 folds:", np.round(punteggi, 3))
print("mean:", round(float(punteggi.mean()), 3), "| standard deviation:", round(float(punteggi.std()), 3))

# the same thing by hand: shows that there is no magic
manuale = []
for idx_tr, idx_val in StratifiedKFold(5, shuffle=True, random_state=0).split(X_tr, y_tr):
    pipeline.fit(X_tr.iloc[idx_tr], y_tr[idx_tr])
    manuale.append(pipeline.score(X_tr.iloc[idx_val], y_tr[idx_val]))
print("by hand, identical:", np.allclose(manuale, punteggi))

Output obtained (Python 3.14.4, NumPy 2.3.5, pandas 3.0.3, scikit-learn 1.8.0; run twice with identical output):

scores of the 5 folds: [0.817 0.833 0.872 0.772 0.861]
mean: 0.831 | standard deviation: 0.035
by hand, identical: True

The five scores range from 0.772 to 0.872: the mean is 0.831 and the standard deviation 0.035. The latter is the reason why it is worth reporting mean and variability, not just one number: two models that differ by one percentage point, with a standard deviation of three points, cannot be told apart. To know how far to trust a score you need the tools of the statistics lesson, which shows how to build a confidence interval.

Choosing a hyperparameter without touching the test

We want to choose the regularisation strength C of the logistic regression: the smaller it is, the more the model is "braked". If we chose it by looking at the test, the test would cease to be one (we would be back to the student with the slip). The choice therefore happens inside the training set, with cross-validation, and the test is looked at only once, at the end. GridSearchCV tries every combination of a grid; the grid is a dictionary that associates the name of a parameter with a list of values, the structure described in the lesson on lists, tuples, dictionaries and sets. The name logisticregression__C uses the double underscore to say "the parameter C of the step logisticregression": in a pipeline the parameters are addressed by the name of the step.

from sklearn.model_selection import GridSearchCV

griglia = {"logisticregression__C": [0.001, 0.01, 0.1, 1, 10, 100]}
ricerca = GridSearchCV(pipeline, griglia, cv=StratifiedKFold(5, shuffle=True, random_state=0))
ricerca.fit(X_tr, y_tr)

for C, media in zip(griglia["logisticregression__C"], ricerca.cv_results_["mean_test_score"]):
    print(f"C = {C:>7}: validation accuracy {media:.3f}")
print("chosen:", ricerca.best_params_)
print("test, looked at only once:", round(ricerca.score(X_te, y_te), 3))

Output obtained (Python 3.14.4, NumPy 2.3.5, pandas 3.0.3, scikit-learn 1.8.0; run twice with identical output):

C =   0.001: validation accuracy 0.719
C =    0.01: validation accuracy 0.814
C =     0.1: validation accuracy 0.823
C =       1: validation accuracy 0.831
C =      10: validation accuracy 0.830
C =     100: validation accuracy 0.829
chosen: {'logisticregression__C': 1}
test, looked at only once: 0.817

Read the curve: with C = 0.001 the model is braked to the point of not managing to learn even the true structure (underfitting, 0.719); from C = 1 upwards the validation flattens around 0.83, a sign that further degrees of freedom do not help. The value on the test (0.817) is close to the validation estimate (0.831): it is what you expect when the process is clean. The final model is retrained on the whole training set (refit=True is the default of GridSearchCV), and only then evaluated on the test.

More values in the grid and more hyperparameters mean more combinations: with 5 hyperparameters and 10 values each there are 10⁵ configurations, each multiplied by k trainings, a count you find explained in the lesson on combinatorics. There is also a subtle risk: if you compare hundreds of configurations, the best one will be partly lucky, and its validation estimate will be optimistic. For an honest estimate you need a test never used or a nested validation (the scikit-learn documentation points to an example that compares nested validation with the simple one). Finally, if you need to save the model to disk with joblib or pickle, remember the warning of the model persistence page: loading a file of this kind can execute arbitrary code, so load only files you produced yourself or that you trust; the lesson on functions, files, exceptions and objects explains why.

Typical errors

Most of these rows have a single root, data leakage in one of its forms: the test (or a part of it, or a twin of it) has influenced a choice made in training. The remedy is always structural, not a matter of attention: build the process so that the mistake cannot be made, with a pipeline and with a split made by statistical unit.

Four rules to take away

• Baseline before the model: a DummyClassifier (or DummyRegressor) as a comparison.
• Everything inside the Pipeline: imputation, scaling, encoding, selection, model.
• The test set is opened once: if you look at it twice, you have a new training set.
• Report mean and variability of the validation, not just one number.

They are independent and you can apply them in any order, but the first is the most underrated: without a floor you do not know whether a 94% is a result or a coincidence, as happened with the baseline of 0.629 on the tumour dataset. Also check the shapes (X.shape, y.shape) and the input types, because most beginners' errors are there.

Mini-project: the wines, from start to finish

With the wine dataset, work in four steps.

1. Split into training (75%) and test (25%) with stratification and random_state=0.
2. Compare, with 5-fold cross-validation on the training set, three models inside a pipeline with StandardScaler: logistic regression, KNeighborsClassifier, DecisionTreeClassifier(max_depth=3).
3. Choose the best and evaluate it once on the test, reporting the confusion matrix.
4. Repeat step 2 without the scaler and write two lines on which model changes and which does not, and why.

The second step is the one in which people most often go wrong, because it is easy to compute the validation on the whole dataset instead of on the training set alone: remember that the test is out of the game until the third step. Try to do it on your own before reading the commented solution below.

The solution is a single self-contained program: separation, comparison between models with cross-validation inside the training set, choice and a single final evaluation on the test. For each model cross_val_score computes the mean accuracy over 5 folds of the training set alone, twice: inside a pipeline with the scaler and without. The dictionary modelli contains three estimators not yet trained, which cross_val_score clones at every fold.

import warnings
import numpy as np
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

warnings.filterwarnings("ignore")
X, y = load_wine(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

modelli = {
    "logistic regression": LogisticRegression(max_iter=5000),
    "k neighbours": KNeighborsClassifier(),
    "tree (depth 3)": DecisionTreeClassifier(max_depth=3, random_state=0),
}
print(f"{'model':<24} {'with scaler':>14} {'without scaler':>16}")
risultati = {}
for nome, modello in modelli.items():
    con = cross_val_score(make_pipeline(StandardScaler(), modello), X_tr, y_tr, cv=cv).mean()
    senza = cross_val_score(modello, X_tr, y_tr, cv=cv).mean()
    risultati[nome] = (con, senza)
    print(f"{nome:<24} {con:>14.3f} {senza:>16.3f}")

migliore = max(risultati, key=lambda k: risultati[k][0])
finale = make_pipeline(StandardScaler(), modelli[migliore]).fit(X_tr, y_tr)
print("\nmodel chosen on the validation:", migliore)
print("accuracy on the test (opened once):", round(finale.score(X_te, y_te), 3))
print("confusion matrix (rows = true class):\n", confusion_matrix(y_te, finale.predict(X_te)))

Output obtained (Python 3.14.4, NumPy 2.3.5, pandas 3.0.3, scikit-learn 1.8.0; run twice with identical output):

model                       with scaler   without scaler
logistic regression               0.963            0.962
k neighbours                      0.955            0.713
tree (depth 3)                    0.910            0.917

model chosen on the validation: logistic regression
accuracy on the test (opened once): 1.0
confusion matrix (rows = true class):
 [[15  0  0]
 [ 0 18  0]
 [ 0  0 12]]

The comparison shows the effect of the scale on the three algorithms. The k nearest neighbours gain an enormous advantage (from 0.713 to 0.955), because they use distances. Logistic regression and the tree stay practically the same (0.963 against 0.962 and 0.910 against 0.917). For the tree this is expected: the thresholds on the values do not depend on the scale. For logistic regression I have not verified the cause; a plausible explanation is that, with many iterations allowed, the model compensates for the scale with coefficients of different magnitude, but the regularisation penalises them, so the difference is not guaranteed on other data. The model with the best validation with the scaler (logistic regression) is then chosen, retrained on the whole training set, and the test is looked at only once. The result is 1.0: all 45 test examples classified correctly, and the confusion matrix is diagonal.

Be careful how you read that 100%: with only 45 examples it means these 45 cases went well, not that the model is perfect, and the uncertainty on an accuracy estimated from 45 examples is wide. The validation estimate (about 0.96) is more informative than a single test this small. If, looking at the test, you were tempted to change model, remember the rule: that test has already been used, and the number you would get after the second attempt would no longer be an honest estimate.

Challenges

• Easy. Compute the baseline DummyClassifier(strategy="stratified") and explain why its expected value, with classes 63%/37%, is about 0.63² + 0.37² ≈ 0.53 and not 0.63.
• Medium. In the leakage experiment, vary k (number of features chosen) in {5, 20, 100} and n (examples) in {50, 200, 1000}. With which combination does the effect of the leak disappear? Why?
• Hard. Implement by hand a scaler compatible with the API (fit, transform, attributes with _) inheriting from BaseEstimator and TransformerMixin, and check that inside a Pipeline it gives the same results as StandardScaler.
• Expert. Build a nested validation (a GridSearchCV inside a cross_val_score) and compare the score with that of the best inner score. How optimistic is the latter?
• Enterprise. Describe for a real case of your organisation what the statistical unit is (customer, order, machine), which examples might "contaminate" each other between training and test, and how you would build the split to avoid it.

The first three challenges are solved with what was seen in the lesson; the fourth requires reading the documentation of cross_val_score and GridSearchCV. In the Medium challenge it is best to keep the generator seed fixed and change one parameter at a time, otherwise you do not know what to attribute the difference to.

In the next lesson, Sets and logic, the mathematics module begins: the vocabulary with which probability is written.

Sources

Bibliographic references cited but not opened for this revision: T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009, sec. 7.10 (the "wrong and right way" to do cross-validation); S. Kaufman et al., Leakage in Data Mining: Formulation, Detection, and Avoidance, ACM TKDD 6(4), 2012; M. Stone, Cross-Validatory Choice and Assessment of Statistical Predictions, JRSS B 36(2), 1974; F. Pedregosa et al., Scikit-learn: Machine Learning in Python, JMLR 12, 2011.