04 - Implementing AI in a Company

Implementing AI in a Company: From Problem to Decision, With the Costs in Hand

Machine Learning Engineer course · Module 00 · Data-driven mindset · Lesson 4

Goal. Teach how to evaluate a machine learning initiative before writing code, and how to turn the output of a model (a probability) into a decision with a cost. The reader learns the five-field card, the CRISP-DM cycle, the optimal threshold derived by hand and checked in code, and a criterion for choosing between building, buying or not doing.

Myth. "The ML project consists in building the model." The model is the smallest piece; the rest is defining the decision, procuring the data, inserting the result into a process, and maintaining it.

Introduction

Lesson 03 gave you the metrics. This one gives you the step the metrics alone do not take: saying "so what do we do?". A model returns probabilities; a company makes decisions (call or do not call a customer, block or let a payment through). The bridge between the two is the threshold, and the right threshold depends on costs, not on the mathematics of the model.

Roadmap. (1) The five-field card. (2) The CRISP-DM life cycle. (3) From score to decision: formula and derivation. (4) Code: searching for the threshold. (5) Build, buy, or do not do. (6) The cost you do not see: the technical debt of ML systems.

What it is, why it exists, when NOT to do it

The five-field card

Before every project, five lines. If one remains empty, the project is not ready.

Field 3 hides the most expensive trap of machine learning, leakage of information from the future (data leakage): using as an input of the model a datum that in reality is not yet available. In the lab it works wonderfully; in production it does not exist.

When NOT to do it

• If the decision does not change according to the prediction (fields 1 and 2): even a perfect model produces no value.
• If you cannot obtain the data at the right time (field 3).
• If the cost of the problem is lower than the cost of building and maintaining the model (ROI section).
• If a simple rule already captures almost all the value: try the domain baseline (for example "customers who have not bought for 90 days") before a model.

The four cases are in order of cost of discovery: you see the first two by filling in the card, in an afternoon and without writing code; the third requires an estimate of the costs, which we will make below; the fourth requires actually trying the simple rule. The advantage of discovering them first is that stopping a project at this point costs nothing.

The life cycle: CRISP-DM

The CRISP-DM process (2000) describes six phases in a cycle, not in a line:

1. Business understanding: objectives, success criteria (the five-field card).
2. Data understanding: what is there, quality, gaps.
3. Data preparation: cleaning, building the variables.
4. Modelling: choice of algorithms, tuning.
5. Evaluation: does the model meet the business objective, not just the technical metric?
6. Deployment: insertion into the process, monitoring.

The arrows go back: evaluation often sends you back to business understanding, because you discover the question was wrong. The course follows the same sequence: modules 01-02 are the tools for phases 2-3, 03-08 for phase 4, 11 for phase 6.

First example: from score to decision

Analogy: the fire-alarm threshold

A smoke detector has an adjustable sensitivity. Too sensitive: it rings every time you toast bread (false alarms, and sooner or later you disable it). Not sensitive enough: it stays silent during a real fire. Where do you set the sensitivity? It depends on what a false alarm costs (annoyance) and what an undetected fire costs (catastrophe). It is not a technical choice: it is a choice of costs. In a hospital and in a kitchen the answer is different.

Formula: the optimal threshold, derived

The model gives you p = P(y = 1 | x): the probability that the case is positive. We define the costs:

• C_FN: cost of a false negative (you say "no" but it was "yes").
• C_FP: cost of a false positive (you say "yes" but it was "no").
• Correct outcomes cost 0 (a simplification; it generalises easily).

They are only two numbers, but they are the ones lesson 03 did not give: the confusion matrix counted the errors, here we weigh them. If you cannot estimate even the order of magnitude of the ratio between the two costs, the threshold remains a hypothesis and the sensitivity analysis further down serves exactly this purpose. Given a case with probability p, compare the expected cost of the two actions:

• If you say "yes" (you act): you are wrong when the case is negative, with probability 1 − p. Expected cost: (1 − p) · C_FP.
• If you say "no" (you do not act): you are wrong when the case is positive, with probability p. Expected cost: p · C_FN.

The two expressions are symmetric: each is the cost of the error multiplied by the probability of making it. There is no action without risk; you choose the one whose expected error weighs less, and the point where the two weigh the same is the threshold. It pays to say "yes" when its expected cost is lower or equal:

(1 − p) · C_FP  ≤  p · C_FN
C_FP − p·C_FP   ≤  p·C_FN
C_FP            ≤  p · (C_FN + C_FP)
p               ≥  C_FP / (C_FP + C_FN)

Optimal threshold = C_FP / (C_FP + C_FN) (a result known in the literature, for example in Elkan 2001, under the assumption that the probabilities are calibrated).

Read it: if a false negative costs 20 times a false positive (C_FN = 100, C_FP = 5), the threshold is 5/105 ≈ 0.048. That is, you act even when the case has only a 5% probability of being positive. The "automatic" threshold 0.5 is optimal only if the two errors cost the same, which is almost never true.

Check with the code

We simulate a problem (customer churn, invented data), train a logistic regression, and look for the threshold that minimises the total cost by brute force. Then we compare with the formula.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(11)
n = 20_000
X = rng.normal(size=(n, 3))
logit = -2.2 + 1.4 * X[:, 0] - 0.9 * X[:, 1] + 0.5 * X[:, 2]
p_vera = 1 / (1 + np.exp(-logit))
y = (rng.random(n) < p_vera).astype(int)
print("base rate", y.mean().round(3))

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, random_state=0, stratify=y)
modello = LogisticRegression().fit(X_tr, y_tr)
p = modello.predict_proba(X_te)[:, 1]

C_FN, C_FP = 100.0, 5.0          # INVENTED PARAMETERS: replace them with yours

def costo_totale(soglia):
    pred = p >= soglia
    fn = np.sum((~pred) & (y_te == 1))
    fp = np.sum(pred & (y_te == 0))
    return C_FN * fn + C_FP * fp

soglie = np.linspace(0.01, 0.99, 99)
costi = np.array([costo_totale(s) for s in soglie])
s_best = soglie[costi.argmin()]
s_teorica = C_FP / (C_FP + C_FN)

print("empirical optimal threshold:", round(float(s_best), 2), "| cost", costi.min())
print("theoretical threshold      :", round(s_teorica, 3), "| cost", costo_totale(s_teorica))
print("cost at threshold 0.5      :", costo_totale(0.5))
print("cost 'do nothing'          :", C_FN * y_te.sum())
print("cost 'act on everyone'     :", C_FP * (y_te == 0).sum())

How to read the code. The first lines build a synthetic problem: three random features X, a linear combination logit saying how much each pushes towards churn, and the true probability p_vera = 1/(1 + e^{-logit}) (the sigmoid function of the calculus lesson). The label y is drawn with that probability: so we know how the world is made, which in reality never happens. The data are split into half training and half test (stratify=y keeps the share of positives), the logistic regression is trained and the probabilities on the test set are obtained with predict_proba(...)[:, 1]. costo_totale(soglia) counts false negatives and false positives for a given threshold and multiplies them by the respective costs; the loop tries 99 thresholds and takes the best. The last lines compute the costs of the two business baselines: "do nothing" (all positives are false negatives) and "act on everyone" (all negatives are false positives).

Output obtained (seed 11, NumPy 2.3.5, scikit-learn 1.8):

base rate 0.178
empirical optimal threshold: 0.06 | cost 31435.0
theoretical threshold      : 0.048 | cost 31435.0
cost at threshold 0.5      : 115790.0
cost 'do nothing'          : 178400.0
cost 'act on everyone'     : 41080.0

What to read:

• The theoretical threshold (0.048) and the empirical one (0.06) give the same total cost, 31435: around the minimum the cost curve is flat, so the 0.01 grid does not tell the two apart. The formula works.
• With the default threshold 0.5 the cost is 115790, more than three times 31435. With asymmetric costs, 0.5 is an expensive choice.
• The two business baselines are 178400 ("do nothing") and 41080 ("act on everyone"). The model at the optimal threshold (31435) beats both, but the distance from the second is the real measure of its value: 41080 − 31435 = 9645 on the test sample. If the model costs more than this to operate over the same period, it does not pay off.
• Note also what threshold 0.5 shows: with that value the model costs less than doing nothing (115790 versus 178400), but more than acting on everyone (41080). A badly tuned model can lose against the most banal baseline.

The number to bring to the meeting is the third point, not the first: the value of the model is the distance from the best baseline, because "act on everyone" is the alternative management already has in hand. The fourth point is the warning: the default threshold is not neutral, and in this example it turns a useful model into one worse than a trivial rule. The costs 100 and 5 are hypotheses, not market data.

A second, contrasting example: when the formula is wrong

The trade-off below says that the formula holds for calibrated probabilities. Let us see what happens when they are not. A simple way to obtain uncalibrated probabilities is to ask the logistic regression to give more weight to positives (class_weight="balanced"): it is often used to "help" the model with rare classes, but it shifts the probabilities upwards.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(11)
n = 20_000
X = rng.normal(size=(n, 3))
logit = -2.2 + 1.4 * X[:, 0] - 0.9 * X[:, 1] + 0.5 * X[:, 2]
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, random_state=0, stratify=y)

C_FN, C_FP = 100.0, 5.0
def costo(p, soglia):
    pred = p >= soglia
    return C_FN * np.sum((~pred) & (y_te == 1)) + C_FP * np.sum(pred & (y_te == 0))

soglia_teorica = C_FP / (C_FP + C_FN)
griglia = np.linspace(0.01, 0.99, 99)
for nome, modello in (("standard", LogisticRegression()), ("balanced weights", LogisticRegression(class_weight="balanced"))):
    p = modello.fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
    costi = [costo(p, s) for s in griglia]
    print(f"{nome:<16}: mean probability {p.mean():.3f} (true rate {y_te.mean():.3f}) | "
          f"best threshold {griglia[int(np.argmin(costi))]:.2f} cost {min(costi):.0f} | "
          f"with the theoretical threshold {soglia_teorica:.3f} cost {costo(p, soglia_teorica):.0f}")

Output obtained:

standard        : mean probability 0.180 (true rate 0.178) | best threshold 0.06 cost 31435 | with the theoretical threshold 0.048 cost 31435
balanced weights: mean probability 0.387 (true rate 0.178) | best threshold 0.19 cost 31355 | with the theoretical threshold 0.048 cost 36840

How to read the code. The same problem is repeated with two models. A probability is calibrated if, among all cases where the model says "0.3", about 30% are really positive; a quick way to notice bad calibration is to compare the predicted mean probability with the real rate. The standard model is calibrated (mean 0.180 against a true rate of 0.178), and the formula works: the cost at the theoretical threshold coincides with the minimum. With balanced weights the mean probability rises to 0.387, more than double the real rate: the model "exaggerates" and the theoretical threshold 0.048, applied to inflated probabilities, is too low and triggers the action on too many cases (cost 36840 against a minimum of 31355, obtained with a threshold around 0.19). Lesson: the formula has a condition. If you break it (weights, resampling, some algorithms), either you recalibrate the probabilities (scikit-learn offers CalibratedClassifierCV), or you look for the threshold experimentally on validation data, as we did with the grid.

Sensitivity analysis: how much does everything depend on the invented costs?

The costs 100 and 5 are hypotheses. Before trusting the result you must know how much the conclusion depends on them. The method is elementary: repeat the computation with halved and doubled costs, and see whether the answer ("the model pays off") changes.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(11)
n = 20_000
X = rng.normal(size=(n, 3))
logit = -2.2 + 1.4 * X[:, 0] - 0.9 * X[:, 1] + 0.5 * X[:, 2]
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, random_state=0, stratify=y)
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
positivi, negativi = int(y_te.sum()), int((y_te == 0).sum())

print("C_FN   C_FP   threshold  model cost  best baseline  saving")
for c_fn in (50.0, 100.0, 200.0):
    for c_fp in (2.5, 5.0, 10.0):
        soglia = c_fp / (c_fp + c_fn)
        pred = p >= soglia
        costo_modello = c_fn * np.sum((~pred) & (y_te == 1)) + c_fp * np.sum(pred & (y_te == 0))
        baseline = min(c_fn * positivi, c_fp * negativi)          # the better of "do nothing" and "act on everyone"
        print(f"{c_fn:5.0f} {c_fp:6.1f} {soglia:7.3f} {costo_modello:14.0f} {baseline:18.0f} {baseline - costo_modello:10.0f}")

Output obtained:

C_FN   C_FP   threshold  model cost  best baseline  saving
   50    2.5   0.048          15718              20540       4822
   50    5.0   0.091          26230              41080      14850
   50   10.0   0.167          39840              82160      42320
  100    2.5   0.024          17722              20540       2818
  100    5.0   0.048          31435              41080       9645
  100   10.0   0.091          52460              82160      29700
  200    2.5   0.012          18925              20540       1615
  200    5.0   0.024          35445              41080       5635
  200   10.0   0.048          62870              82160      19290

How to read the code. For each pair of costs it computes the theoretical threshold, the cost of the model at that threshold, and the cost of the best business baseline (the smaller of never acting and always acting). The "saving" column is the difference: the gross value of the model on 10 000 test customers. In all nine combinations the saving stays positive: the conclusion "the model pays off" is robust against halved or doubled costs; what changes is the order of magnitude of the gain. If instead, in a plausible combination, the saving were negative or smaller than the running cost, the project would have to be rethought. The analysis is simple but disciplines the discussion: it moves the debate from "your costs are wrong" to "for which costs does the decision change?".

A data check: leakage from the future, seen in numbers

Field 3 of the card ("data available at the moment of the decision") seems a formality. We show that one wrong variable can make the model look perfect. We add to the data a feature "has opened a cancellation request", which in reality is generated after the decision to leave.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(11)
n = 20_000
X = rng.normal(size=(n, 3))
logit = -2.2 + 1.4 * X[:, 0] - 0.9 * X[:, 1] + 0.5 * X[:, 2]
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

richiesta_disdetta = np.where(y == 1, rng.random(n) < 0.95, rng.random(n) < 0.01).astype(float)   # arises AFTER leaving
X_leak = np.column_stack([X, richiesta_disdetta])

for nome, M in (("without the future variable", X), ("with the future variable", X_leak)):
    X_tr, X_te, y_tr, y_te = train_test_split(M, y, test_size=0.5, random_state=0, stratify=y)
    p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
    print(f"{nome:<32}: AUC on the test set = {roc_auc_score(y_te, p):.3f}")

Output obtained:

without the future variable     : AUC on the test set = 0.850
with the future variable        : AUC on the test set = 0.990

How to read the code. The variable richiesta_disdetta is 1 for 95% of those who leave and for 1% of those who stay: it arises as a consequence of leaving, so in production, when deciding whom to contact, it does not yet exist. The AUC (area under the ROC curve) is a measure of how well the model orders the cases, between 0.5 (chance) and 1 (perfect): with the future variable it goes from a realistic value to a value close to 1. This is exactly the symptom of leakage: results that are too good in the lab. The remedy is in the card: for each feature, ask "when do I know it?". If the answer is "after the outcome", it must be excluded.

Trade-off. The formula holds if the probabilities of the model are calibrated (a p = 0.3 means that 30% of those cases are really positive). Logistic regression is typically well calibrated; trees and forests often are not, and require a calibration (CalibratedClassifierCV). If they are not, look for the threshold by brute force on validation data, as in the code.

The value of a model and the build / buy / skip choice

Formula of the value

With costo_baseline the expected cost of the best behaviour without a model, costo_modello that with the model at the optimal threshold, and E the running cost (infrastructure, monitoring, people, retraining):

valore_netto = (costo_baseline − costo_modello) − E

Over the same period of time. If valore_netto ≤ 0, do not do it, however elegant the model. If the number depends heavily on your invented parameters, do the sensitivity analysis: repeat the computation with halved and doubled costs.

Build, buy or do not do

The hidden cost

Sculley et al. (2015) argue that ML systems accumulate a particular kind of technical debt: data dependencies harder to see than those on code, feedback loops (the model influences future data), intricate configurations. The practical consequence for whoever plans: the running cost E is not a detail, and it grows with time.

The five-field card in a second domain

To see that the card is general, fill it in for a very different problem: the predictive maintenance of a production machine. Decision: when to stop the machine for an inspection. Action: the maintenance department schedules an intervention within the next 48 hours. Data: sensor readings up to one hour before the decision (not the log of repairs done after the failure). Metric: hours of unplanned downtime avoided per month. Cost of errors: a false negative is a sudden failure with a line stoppage; a false positive is a useless inspection. Note that the two costs are in different units (production hours, labour hours): the first job of the project is to convert them into a common unit, typically euros, and to state the assumptions of the conversion.

Typical mistakes

Note on point 3 (split by time). When the data have a temporal dimension, shuffling rows at random lets the training see the future of the test. You sort by date, train on the past and evaluate on the following period.

From pilot to production: what to look at

A well-made pilot answers a single question: "if we put the model into the process, do costs really go down?" For this reason it is worth fixing three things in writing before starting. First, the period and size of the pilot (how many customers, for how many weeks), so as to have enough cases to tell a real improvement from noise (lesson 6 shows how to compute how many are needed). Second, the control group: a part of the flagged customers who are not treated, because without a comparison you do not know whether the retained customers would have stayed anyway. Third, the decision criterion at the end: "if the net saving per customer exceeds X, we extend; otherwise we close". Writing the criterion in advance protects against the very common temptation to read the results in a way favourable to the project one has invested in.

After the launch the work does not end: the data change (seasonality, new products, price changes) and the optimal threshold with them. You need at least monitoring of the base rate and of the distribution of probabilities, a sample of cases verified by hand every month, and a date for reviewing the costs.

Best practices

• Fill in the five-field card in writing, shared with whoever will have to act.
• Start from a domain baseline and two business baselines.
• Run a small pilot with before/after measurement, ideally with a control group (lesson 06).
• Include monitoring and retraining in the budget, not afterwards.
• Document the cost parameters, who set them and when they must be reviewed.

The first two rules are needed before writing code, the third before investing, the last two after the launch. If you must pick only one, pick the baselines: without the comparison with "do nothing" and "act on everyone" you cannot know whether the model is worth anything.

A minimal vocabulary

• Decision threshold: the probability value above which you act.
• Expected cost: the cost of an action weighted by the probability that it is wrong.
• Calibration: correspondence between predicted probabilities and real frequencies.
• Business baseline: the cost of the best behaviour without a model (do not act, act on everyone, a simple rule).
• Net value: saving with respect to the baseline minus the running cost.
• Leakage: use of a datum not available at the moment of the decision.
• AUC: area under the ROC curve; a measure of ordering between 0.5 and 1.
• Sensitivity analysis: repeating the computation varying the assumptions to see whether the conclusion changes.

The eight entries follow the flow of the lesson: the decision (threshold, expected cost), the comparison (baseline, net value), the traps (calibration, leakage), the verification tools (AUC, sensitivity). If in a meeting someone says "the model is good", ask which baseline and which threshold they refer to.

Mini-project

Take the code of the initial example and:

1. Change C_FN and C_FP to five different pairs (for example 10/10, 50/5, 100/5, 500/5, 100/50). For each print the theoretical threshold, the empirical threshold and the saving with respect to threshold 0.5.
2. Add a running cost E and compute the valore_netto with respect to the better of the two baselines.
3. Find the value of E beyond which the project stops paying off (the break-even point).

The mini-project retraces the article in miniature: the first item shows you how the threshold moves with the costs, the second adds the running cost, the third turns it into a decision number, the break-even point. That last one is the result you would bring to whoever decides the budget.

Challenge

• Easy. With C_FN = C_FP, prove that the threshold becomes 0.5.
• Medium. Extend the derivation to the case where the "correct" outcomes also have a cost: C_TP for a true positive (for example the cost of the recovery action) and C_TN. How does the threshold change?
• Hard. Show with an experiment that, if the probabilities are not calibrated (use DecisionTreeClassifier(max_depth=3)), the theoretical threshold does not coincide with the empirical one. Then calibrate and try again.
• Expert. Simulate a distribution shift (the base rate doubles) and show how the optimal threshold moves with the same rule. When is retraining needed and when is it enough to change the threshold?
• Enterprise. Write the five-field card for a real decision of your organisation and identify which field is the weakest.

Easy and Medium are algebraic proofs, Hard and Expert are experiments that break the assumptions of the formula (calibration and a stable distribution), Enterprise brings it into your work. Do at least Hard: it is the only one that shows, with your own numbers, why the formula has a condition.

Recap

Mind map. Decision → Action → Data (available then) → Metric (business) → Cost of errors. Optimal threshold C_FP/(C_FP+C_FN). Net value = saving − running cost. Build / buy / skip.

3 concepts: five-field card; threshold from cost; net value. 3 mistakes: disconnected metric; threshold 0.5 by default; temporal leakage. 3 best practices: business baseline; pilot with control; running cost in the budget. 1 exercise: the mini-project of this lesson.

Next lesson: data culture: turning vague questions into questions that data can answer.

The same series also includes: 03 - AI Opportunities, Challenges and Risks (the previous lesson), 05 - Data Culture (the next lesson).