01 - Introduction to Machine Learning: Taxonomy and Workflow

A traditional program applies rules that somebody wrote; a machine learning model derives the rule from examples. The difference shows up at once with a spam filter: writing by hand the rules that separate useful mail from advertising is endless work, whereas giving the system ten thousand messages that are already labelled and letting it find on its own the signals that tell them apart works, provided you know how to check that it learned something true. This article opens the ML Fundamentals series and serves two decisions: which family the problem in front of you belongs to, and which steps separate a notebook that "works" from a result you can trust. It assumes only a little Python and the idea of mean and proportion; the rest we define as we go.

All the examples run on data bundled with scikit-learn (scikit-learn 1.9.0, numpy 2.4.6, Python 3.12), with a fixed seed, downloading nothing. Every output shown was obtained twice with the same result. For the implementation of a model from scratch, go on to the next chapter, 02 - Linear regression from scratch.

What "learning from data" means

A model is a function with adjustable parameters: it receives features as input (the measured quantities, for example floor area, storey and neighbourhood of a flat) and returns a prediction (the price). Training means choosing the parameters that make the errors on the available examples small, as measured by a loss function. So far statistics describes the same thing; what machine learning adds is the insistence on a single question: how does the model do on data it has never seen? A model that memorises the training examples has zero error and is useless: that is the subject of chapter 04 - Bias-Variance Tradeoff.

The taxonomy: which kind of problem do I have

The distinction that organises everything else is whether you hold the right answer for the examples. The textbook by James, Witten, Hastie and Tibshirani (An Introduction to Statistical Learning, second edition) covers it in section 2.1.4, "Supervised Versus Unsupervised Learning", and takes it up again in chapter 12 for unsupervised learning.

Supervised learning: regression and classification

In supervised learning every example comes with its answer, called the label or target. If the target is a number (the price of a flat, tomorrow's temperature) the problem is one of regression; if it is a category (benign or malignant tumour, spam message or not) it is one of classification. Chapters 02 and 03 build an example of each.

Unsupervised learning

Without labels you can only look for structure: group similar examples (clustering) or compress the features while keeping the information (dimensionality reduction). Here the "right answer" does not exist, so the judgement on the result is less clear-cut, as the following comparison on the Iris dataset shows: 150 flowers of three species measured with four lengths. The same dataset is used once with the labels and once ignoring them.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, adjusted_rand_score
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
clf = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
print("supervised, test accuracy:", round(accuracy_score(yte, clf.predict(Xte)), 3))
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(X)
print("unsupervised, adjusted Rand index:", round(adjusted_rand_score(y, km.labels_), 3))
print("cluster sizes:", sorted(np.bincount(km.labels_).tolist()))

# Output:
# supervised, test accuracy: 1.0
# unsupervised, adjusted Rand index: 0.73
# cluster sizes: [38, 50, 62]

The classifier sees the labels during training and on the 45 flowers set aside (test_size=0.3, so 30% of 150) it makes no mistake. stratify=y keeps the three species in equal proportion in the two blocks, otherwise one of them might get too few. An accuracy of 1.0 on 45 examples is not a guarantee: with a different split seed you might see an error, and that is one reason why chapter 05 repeats the evaluation on several splits. K-means, which has never seen the species, finds three groups of 38, 50 and 62 flowers against the real 50, 50, 50: the adjusted Rand index (adjusted_rand_score) is 1 for two identical partitions and about 0 for random partitions, and 0.73 says that the grouping largely but not entirely coincides with the species. The method almost recovered the structure without knowing anything about botany, and that is what you can ask of an unsupervised algorithm.

The other families, one line each

In semi-supervised learning only a small part of the examples has a label. In self-supervised learning the label is derived from the data themselves, for example by hiding a word of a sentence and asking the model to guess it. In reinforcement learning an agent chooses actions and receives a reward, without anybody telling it the right move. This series concentrates on the first two families, which cover most of the work on tabular data.

To find your bearings, the question is which data you really have available:

• Examples with a numeric target: regression.
• Examples with a category to predict: classification.
• Examples with no target at all: clustering or dimensionality reduction.
• Few labelled examples and many unlabelled ones: semi-supervised.
• A sequence of decisions with a final reward: reinforcement.

The items are alternatives, not a sequence, but the choice is not always free: often your problem admits two formulations, and it is better to prefer the one for which the label can actually be obtained. Predicting whether a customer will leave the service is a classification; predicting in how many days they will do so is a regression, which however requires waiting for the customer to leave, so the cost of collecting labels changes.

The workflow, from the question to monitoring

The steps have an order that cannot be inverted without damage, in particular the third with respect to the others:

1. Formulate the question and the metric with which you would know you answered well (for example "less than 5% of useful messages ending up in spam").
2. Look at the data: how many rows, how many features, how many missing values, what proportion the classes have.
3. Set aside a test set before making any decision by looking at the data, and do not touch it again until the end.
4. Build a trivial baseline (for example always answering with the most frequent class) to know what "doing nothing" is worth.
5. Train the real model and compare it with the baseline on validation data.
6. Put it into use and check that the input data do not change over time (chapter 12).

The list is a true sequence: if step 3 comes after step 5, the test set has already influenced your choices and no longer measures anything independent. The step that is easiest to skip is 4, and it is the one that gives meaning to every later number, as the next experiment shows on the Wisconsin Breast Cancer dataset, 569 tumours described by 30 measurements and classified as benign (class 1) or malignant (class 0).

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
print("examples, features:", X.shape, "- class 1 (benign):", round(y.mean(), 3))
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
base = DummyClassifier(strategy="most_frequent").fit(Xtr, ytr)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, ytr)
print("baseline (always the most frequent class):", round(base.score(Xte, yte), 3))
print("logistic regression with standardization:", round(model.score(Xte, yte), 3))
print("training accuracy:", round(model.score(Xtr, ytr), 3))

# Output:
# examples, features: (569, 30) - class 1 (benign): 0.627
# baseline (always the most frequent class): 0.629
# logistic regression with standardization: 0.986
# training accuracy: 0.988

The baseline DummyClassifier(strategy="most_frequent") always answers "benign" and scores 0.629 on the test set (90 correct cases out of 143): it is the threshold to beat, and it almost coincides with the proportion of benign tumours in the dataset, 0.627. The real model reaches 0.986 (141 out of 143). make_pipeline joins StandardScaler and logistic regression in a single object: the scaler computes mean and standard deviation from the training data alone and applies them to the test set, which is exactly the separation that step 3 asks for. The accuracy on the training set (0.988) is just above that on the test set (0.986): the difference is small, so there is no visible overfitting here. Without the baseline, "0.986" would look like a result and nothing more; with the baseline it reads as a jump from 0.63 to 0.99.

Data leakage: when the metric lies

Data leakage is the use, during training, of information that you would not have in production; the scikit-learn documentation ("Common pitfalls and recommended practices") treats it with examples of preprocessing performed before the split. The most insidious version is feature selection done by looking at all the labels. To see it with numbers we build a case where the correct answer is known: 50 examples, 5000 features of pure noise, labels chosen at random. No model can do better than 50%.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
X = rng.standard_normal((50, 5000))      # 50 examples, 5000 features of pure noise
y = np.repeat([0, 1], 25)                # labels that do not depend on X
cv = StratifiedKFold(5, shuffle=True, random_state=0)

# WRONG way: pick the 20 best features looking at ALL the labels, then validate
X_sel = SelectKBest(f_classif, k=20).fit_transform(X, y)
wrong = cross_val_score(LogisticRegression(), X_sel, y, cv=cv).mean()

# RIGHT way: the selection sits inside the pipeline, redone in every fold
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
right = cross_val_score(pipe, X, y, cv=cv).mean()
print("selection outside the fold:", round(wrong, 3))
print("selection inside the fold:", round(right, 3))

# Output:
# selection outside the fold: 0.96
# selection inside the fold: 0.52

With the selection outside the cross-validation the estimated accuracy is 0.96, a result that would look exceptional on data that are pure noise. The reason: by choosing the 20 features out of 5000 that are most correlated with the labels of all the examples, some features coincide by chance with the labels even in the examples later used for the test, and the model "discovers" a regularity that does not exist. With SelectKBest inside the pipeline the selection is redone in every fold using only the training examples of that fold, and the estimate drops to 0.52, that is, to chance. The defect is not in the model but in the point where every decision that uses the labels is born: that is where to look. Here cross_val_score is used as a ready-made tool; how it works and when it distorts the estimate is explained in chapter 05.

Worked exercise: the misleading accuracy

A hospital ward wants a model that flags patients with a certain disease: out of 1000 patients, 20 are sick (2%). Somebody proposes a model that always answers "healthy" and declares "98% accuracy". Let us compute what that really means, with synthetic data built for the occasion.

import numpy as np
from sklearn.metrics import accuracy_score, recall_score

y_true = np.array([1] * 20 + [0] * 980)      # 20 sick patients out of 1000 (synthetic data)
y_pred = np.zeros(1000, dtype=int)           # the model always says "healthy"
print("accuracy:", accuracy_score(y_true, y_pred))
print("recall on the sick:", recall_score(y_true, y_pred))

# Output:
# accuracy: 0.98
# recall on the sick: 0.0

Accuracy 0.98 and recall 0.0: recall is the share of sick patients actually found (true positives divided by all the real positives), and here none of the 20 is flagged. A useless model got a high mark because the class of interest is rare. The practical lesson is twofold: always compare the model with the baseline, and choose the metric according to the cost of the errors and not out of habit. Chapter 11 - Metrics and evaluation develops them all.

When not to use machine learning

A model is a statistical bet; when it can be avoided, avoid it. Three situations in which it is not worth it:

• The rule exists and is exact: computing a tax or a deadline is written as code, and a model would only add errors.
• You do not have enough examples for the phenomenon you want to describe, in particular for rare classes.
• An error of the system has a cost that nobody is willing to pay and nobody checks the output.

The common criterion is that machine learning serves where the rule is too complex to write but examples abound and the occasional error is tolerable; in the other cases a table or a conditional statement is cheaper, easier to explain and easier to check.

What you can do now

You can tell a supervised problem from an unsupervised one by looking at the data you have, you can set up an honest comparison with a baseline, and you can recognise the point at which feature selection makes the labels leak into the validation. The next step is to build a model from scratch to see its parameters: 02 - Linear regression from scratch in Python.