03 - AI Opportunities, Challenges and Risks

AI Opportunities, Challenges and Risks: How They Are Measured, Not Feared

Machine Learning Engineer course · Module 00 · Data-driven mindset · Lesson 3

Goal. Replace opinions ("AI is dangerous", "AI solves everything") with measurements. The reader learns to do three things: discover that accuracy can lie, audit a model by group, and understand why certain fairness goals cannot be satisfied together (with a proof).

Myth. "If the model is accurate, it is also fair." In the code of this lesson the group with the highest accuracy is the one that is penalised.

Introduction

Every technology has opportunities and risks. With AI the debate gets lost in opinions straight away. An engineer has an advantage: they can measure. This lesson gives you the essential measuring tools and the European regulatory framework in which one works.

Roadmap. (1) The opportunities, concretely. (2) The challenges, in seven families. (3) First measurement: the confusion matrix and the accuracy that lies. (4) Second measurement: the audit by group. (5) Why you cannot have everything: the proof. (6) The regulatory and governance framework. (7) What to do on Monday morning.

What it is, why it matters, when NOT to trust

Opportunities: where AI really helps

They can be reduced to four verbs, each with its own form of task:

The common thread is that AI does well the tasks that are repetitive, with many examples, where an occasional error is tolerable, and does them at a scale human attention cannot sustain.

Challenges: seven families

1. Data quality: missing values, duplicates, wrong labels.
2. Bias: the model inherits the distortions of the data and of the labelling.
3. Privacy: personal data requires a legal basis and minimisation.
4. Explainability: "why did it decide that?" does not always have an answer.
5. Robustness and security: adversarial inputs, prompt injection, distribution shift.
6. Costs: computation, data, people, continuous maintenance.
7. Accountability: who answers for the error?

The seven items do not weigh the same in every project, but none can be skipped: the first two come from the data, the last five from the way the model is used. The section that follows turns each one into a check you can run, because a risk you cannot measure remains an opinion.

When NOT to trust a performance number

When (a) the baseline, (b) the separation between training and test, (c) the breakdown by subgroups, (d) the asymmetric cost of errors is missing. The next two sections cover (a) and (c).

The seven challenges, and how each is measured

A list of risks is not enough: for each one we need a concrete check that an engineer can run. Here is the operational translation.

1. Data quality: measured with simple counts. How many duplicate rows? What percentage of missing values per column? How many labels disagree when two people label the same example? A high disagreement rate between labellers says the problem is ill-defined before you even begin.
2. Bias: measured by comparing the metrics per group (this is the audit of the next section). There is no single test: there is a family of criteria to choose from, and the choice is a decision, not a calculation.
3. Privacy: measured with the inventory: which personal data enter the model, with what legal basis, for how long, and whether you really need all of them (minimisation principle). A practical check: remove a personal column and see how much performance drops; if the drop is negligible, the column should not have been collected.
4. Explainability: measured by asking whether, for a single decision, you can say which features determined it and whether a domain expert considers those reasons sensible.
5. Robustness: measured by perturbing the inputs (noise, extreme values, categories never seen) and observing how much the output changes; and by comparing the performance on today's data with that on data from a year ago (distribution shift).
6. Costs: measured explicitly, including the ones usually forgotten: labelling, periodic retraining, monitoring, human review.
7. Accountability: measured with a question written in the project document: "If this model fails seriously, who is notified, who decides to switch it off, who answers?" If there is no answer, the risk is not managed.

The cross-cutting rule: a risk without a measure is an opinion, and an opinion cannot be monitored over time. Note that six checks out of seven are counts or comparisons done in a few lines of code or with a spreadsheet: the hard part is not computing them, it is deciding in advance which value triggers an intervention, and writing it down before seeing the results.

Preparation

pip install numpy scikit-learn

No external data: all data are simulated with generators we declare. All the numbers in the text are produced by the code, with a fixed seed.

First example: the accuracy that lies

Analogy: the sleeping guard

Imagine a night guard in a building where a thief enters once every fifty nights. If the guard sleeps always and declares "no thief", he is right forty-nine nights out of fifty: 98%. His "success rate" is very high and his value is zero, because he never sees the only event that matters.

Before the definitions, a mental image. Every prediction of a binary classifier is a verdict on a case: "alarm" or "all clear". The verdict can be right or wrong in two different ways: an alarm on a harmless case (false alarm) or silence in front of a real case (missed detection). These two errors almost always have different costs (an anti-spam filter that sends an important email to spam versus one that lets an advertisement through), and the confusion matrix serves precisely to keep them separate instead of merging them into a single number.

Definitions: the confusion matrix

For a two-class problem (positive = the event we look for, for example fraud) every prediction falls into one of four boxes:

From here, the fundamental measures (I define them all, none is "obvious"):

• Accuracy = (TP + TN) / (TP + TN + FP + FN): fraction of right answers.
• Recall / TPR = TP / (TP + FN): among the true positives, how many I found.
• False positive rate / FPR = FP / (FP + TN): among the true negatives, how many I mistook for positives.
• Precision / PPV = TP / (TP + FP): among those I flagged, how many really were.

The first two lines look at all the answers and at the true positives, the last two look at the error per class. The sleeping guard has accuracy 98% and recall 0%: none of the four measures is enough on its own, and recall and precision are the ones that expose the guard. FPR and PPV look like relatives but have different denominators (the true negatives versus the flagged cases) and, as you will see, they come apart when the base rate changes.

import numpy as np
from sklearn.metrics import confusion_matrix

rng = np.random.default_rng(7)
n = 10_000
y = (rng.random(n) < 0.02).astype(int)          # about 2% of positive cases
pred_pigra = np.zeros(n, dtype=int)             # "it is never fraud"

tn, fp, fn, tp = confusion_matrix(y, pred_pigra).ravel()
print("lazy accuracy:", (tn + tp) / n)
print("recall:", tp / (tp + fn))

How to read the code. y is the truth: a vector of zeros and ones in which about 2% of the values are 1 (rng.random(n) < 0.02 is true with probability 0.02, and astype(int) turns it into 1). pred_pigra is a model that always says 0. confusion_matrix(y, pred_pigra) returns the 2×2 table and .ravel() flattens it into four numbers in the order tn, fp, fn, tp; since the model never says "positive", tp and fp are zero and the recall tp/(tp+fn) is zero. Accuracy is simply the fraction of zeros in the vector y.

Output obtained: accuracy 0.9784, recall 0.0. A model that finds nothing, with almost 98% accuracy.

A number to grasp what an error costs. If a missed detection costs 50 euros and a false alarm 2 euros, a model that flags more (high recall, low precision) can pay off even if it "errs more often": the expected cost counts, not the number of errors. This is why the decision threshold is not left at the default value 0.5: it is chosen with the costs, as we will see in lesson 4.

Trade-off. Precision and recall pull in opposite directions: if you flag everything, recall rises to 1 and precision collapses; if you flag only the very certain cases, precision rises and recall falls. Which to prefer depends on the cost of the two errors (lesson 04): a screening test wants high recall (do not miss the sick), a filter that blocks accounts wants high precision (do not block the innocent).

Theory: audit by group and the impossibility of having everything

Why the average hides

An average number is an average weighted over the groups. If a group is small, it can be doing badly without the average noticing. The audit consists in computing the same metrics separately for every relevant subgroup.

Three classical criteria (names from the literature):

• Demographic parity: the selection rate P(ŷ = 1) is equal across groups.
• Equal opportunity (Hardt, Price, Srebro 2016): the recall P(ŷ = 1 | y = 1) is equal across groups. Whoever deserves the positive outcome has the same probability of getting it.
• Predictive parity: the precision P(y = 1 | ŷ = 1) is equal across groups. "A signal means the same thing in all groups."

The three criteria look at different points of the confusion matrix: the first ignores the truth and counts only who is selected, the second looks at who deserves it, the third looks at who was flagged. They are all reasonable requests and each one protects a different person; further on you will see that they cannot be satisfied together when the base rates differ.

The experiment

Two groups, A and B. In B the phenomenon is rarer (base rate 15% versus 40%) and the measuring instrument is less informative (the separation between classes is 0.8 versus 1.5). The data are simulated: they serve to show a mechanism, not to describe a real case. We train a single model on everyone.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix

rng = np.random.default_rng(7)

def genera(n, gruppo, rng):
    p = 0.40 if gruppo == "A" else 0.15        # base rate
    sep = 1.5 if gruppo == "A" else 0.8        # how well the signal tells the classes apart
    y = (rng.random(n) < p).astype(int)
    x = rng.normal(0, 1, n) + sep * y
    return x.reshape(-1, 1), y

XA, yA = genera(6000, "A", rng)
XB, yB = genera(2000, "B", rng)
X = np.vstack([XA, XB])
y = np.concatenate([yA, yB])
g = np.array(["A"] * len(yA) + ["B"] * len(yB))

modello = LogisticRegression().fit(X, y)
pred = modello.predict(X)

def metriche(y, p):
    tn, fp, fn, tp = confusion_matrix(y, p, labels=[0, 1]).ravel()
    return {
        "accuracy": (tp + tn) / len(y),
        "selection_rate": (tp + fp) / len(y),
        "TPR": tp / (tp + fn),
        "FPR": fp / (fp + tn),
        "PPV": tp / (tp + fp),
    }

for nome in ("A", "B"):
    m = metriche(y[g == nome], pred[g == nome])
    print(nome, {k: round(float(v), 3) for k, v in m.items()})

How to read the code. genera builds a group: it chooses the base rate p (the share of positives), draws the label y and produces a single signal x that is on average sep units higher for positives than for negatives (the larger sep is, the better the signal separates the classes). Group A is larger (6000 examples), with the more frequent phenomenon and a better signal; group B is small and has a weak signal. One single model is trained on everyone (LogisticRegression, a model that estimates the probability of the positive class and turns it into a verdict with threshold 0.5) and the metrics are computed separately for each group with the function metriche, which reads the four boxes of the confusion matrix.

Output obtained (seed 7, scikit-learn 1.8):

Read it carefully, because it is counter-intuitive:

• Group B has the highest accuracy (0.809 versus 0.774).
• But in group B the model finds only 39% of the true positive cases, against 63% for A. A person in B who would deserve the positive outcome has 24 percentage points less probability of getting it (0.388 versus 0.626).
• The accuracy of B is high for the same reason as the guard: positives are rare (15%), so saying "no" almost always is almost always right.
• The FPR is practically equal (0.123 versus 0.118): if you looked only at that, you would conclude the model is fair.

The four observations matter in the reverse order to the one in which you read them: the equal FPR is the most reassuring and the most misleading, while the gap in recall (0.626 versus 0.388, about 24 percentage points) is what a person in B actually suffers. No single metric tells the story. This is why several metrics are audited, per group. For brevity the metrics here are computed on the same data used for training: with a two-parameter model and eight thousand examples the difference is small, but in a real audit they are computed on held-out data.

A second, contrasting example: bias coming in through the back door

A common mistake: "let us remove the group column from the model and the problem disappears". We build a counterexample. A minority group (about 30% of cases) has, in the historical data, a lower rate of positive outcomes (the labels embody an unfair decision of the past). The real competence is identical in the two groups. There is also a variable, the residence "zone", correlated with the group. We compare three models, with different sets of features.

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(11)
n = 20_000
gruppo = (rng.random(n) < 0.3).astype(int)              # 1 = minority group
zona = gruppo * 1.2 + rng.normal(0, 1, n)                # proxy: correlated with the group
competenza = rng.normal(0, 1, n)                         # same in the two groups
# historical label: depends on competence, but also (unfairly) on the group
y = ((competenza - 0.8 * gruppo + rng.normal(0, 0.7, n)) > 0).astype(int)

print("zone-group correlation:", round(float(np.corrcoef(zona, gruppo)[0, 1]), 3))
print("historical positive-outcome rate -> group 0:", round(float(y[gruppo == 0].mean()), 3), "| group 1:", round(float(y[gruppo == 1].mean()), 3))

insiemi = {
    "competence only": np.column_stack([competenza]),
    "competence + zone (no group)": np.column_stack([competenza, zona]),
    "competence + zone + group": np.column_stack([competenza, zona, gruppo]),
}
for nome, X in insiemi.items():
    previsione = LogisticRegression().fit(X, y).predict(X)
    tassi = [float(previsione[gruppo == g].mean()) for g in (0, 1)]
    print(f"{nome:<30}: selection rate group 0 = {tassi[0]:.3f}, group 1 = {tassi[1]:.3f}")

Output obtained:

zone-group correlation: 0.485
historical positive-outcome rate -> group 0: 0.494 | group 1: 0.26
competence only               : selection rate group 0 = 0.402, group 1 = 0.407
competence + zone (no group)  : selection rate group 0 = 0.424, group 1 = 0.359
competence + zone + group     : selection rate group 0 = 0.490, group 1 = 0.218

How to read the code. y is built so that the minority group is disadvantaged in the historical labels (the term -0.8 * gruppo). The first model sees only competence, which is distributed in the same way in the two groups: it selects practically equal shares (about 40% and 41%). The second does not receive the column gruppo, but receives zona, which reflects it: the model uses it to reproduce the pattern of the historical labels, and the share of group 1 falls (about 36% versus 42%). The third, with the group column as well, amplifies the gap (22% versus 49%). The lesson is precise: removing the sensitive column is not enough if proxies exist, and a model trained on distorted labels learns the distortion if it has the chance. The countermeasure is not a code trick, it is the process: measure the outcome by group, question the origin of the labels, and remove the variables that serve only to reconstruct the group.

Intervening on the threshold: what rebalancing costs

One family of remedies acts after training: use a different decision threshold per group, chosen so as to equalise recall (equal opportunity). We try it on the model of the previous section (same data, same seed), looking for the threshold that brings the recall of group B to the level of A.

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(7)
def genera(n, p, sep):
    y = (rng.random(n) < p).astype(int)
    return (rng.normal(0, 1, n) + sep * y).reshape(-1, 1), y

XA, yA = genera(6000, 0.40, 1.5)
XB, yB = genera(2000, 0.15, 0.8)
modello = LogisticRegression().fit(np.vstack([XA, XB]), np.concatenate([yA, yB]))
pA, pB = modello.predict_proba(XA)[:, 1], modello.predict_proba(XB)[:, 1]

def metriche(y, p, soglia):
    pred = p >= soglia
    tp = int((pred & (y == 1)).sum()); fn = int((~pred & (y == 1)).sum())
    fp = int((pred & (y == 0)).sum()); tn = int((~pred & (y == 0)).sum())
    return tp / (tp + fn), fp / (fp + tn), tp / max(tp + fp, 1)

tpr_A = metriche(yA, pA, 0.5)[0]
for soglia_B in (0.5, 0.35, 0.25, 0.2, 0.15):
    tpr, fpr, ppv = metriche(yB, pB, soglia_B)
    print(f"threshold B = {soglia_B:.2f}: recall {tpr:.3f} (target {tpr_A:.3f}) | FPR {fpr:.3f} | precision {ppv:.3f}")

Output obtained:

threshold B = 0.50: recall 0.388 (target 0.626) | FPR 0.118 | precision 0.362
threshold B = 0.35: recall 0.527 (target 0.626) | FPR 0.230 | precision 0.283
threshold B = 0.25: recall 0.673 (target 0.626) | FPR 0.346 | precision 0.251
threshold B = 0.20: recall 0.728 (target 0.626) | FPR 0.433 | precision 0.225
threshold B = 0.15: recall 0.816 (target 0.626) | FPR 0.539 | precision 0.207

How to read the code. predict_proba returns for every example the estimated probability of the positive class; deciding "positive" if it exceeds a threshold is the decision rule, and changing it does not require retraining anything. metriche computes recall, false positive rate and precision for a given threshold. As the threshold for group B decreases, recall rises (more cases are flagged) towards that of A, but the false positive rate rises and precision falls: you pay in false alarms. It is the identity of the previous section showing itself: with different base rates, equalising recall shifts the other metrics. With threshold 0.25 the recall of B (0.673) reaches and exceeds that of A (0.626), at the price of a false positive rate of 0.346 and a precision of 0.251. Whether this is acceptable depends on the context: in an early diagnosis, a few false alarms in order not to miss the true cases may be right; in a punitive measure, no.

Why you cannot fix everything together: proof

We would like a model with equal TPR, equal FPR and equal PPV across groups. Is it possible? Full derivation (Chouldechova 2017).

Let p = P(y = 1) be the base rate of the group. Consider a population of N people in the group.

1. True positives: N·p. True negatives: N·(1 − p).
2. True positives detected: TP = TPR · N·p.
3. False positives: FP = FPR · N·(1 − p).
4. Precision is PPV = TP / (TP + FP).
5. Substitute: PPV = TPR·p / ( TPR·p + FPR·(1 − p) ).
6. Multiply by the denominator: PPV·TPR·p + PPV·FPR·(1 − p) = TPR·p.
7. Isolate the term with FPR: PPV·FPR·(1 − p) = TPR·p − PPV·TPR·p = TPR·p·(1 − PPV).
8. Therefore:

The derivation uses only two definitions and a little algebra: steps 1-3 count the people, 4 and 5 express precision as a function of TPR, FPR and base rate, from 6 to 8 the FPR is isolated. The result below is an identity, not an approximation: it holds for any binary classifier, because it says how three measures and the base rate are tied together.

FPR = TPR · p · (1 − PPV) / ( PPV · (1 − p) )

The identity ties the three metrics to the base rate p. Now the point. If two groups have the same TPR and the same PPV, but different base rates p_A ≠ p_B, then the right-hand side is different, hence the FPR is different too (except in the degenerate cases PPV = 1 or TPR = 0). There is no non-trivial classifier that makes TPR, FPR and PPV equal when the base rates differ. Kleinberg, Mullainathan and Raghavan (2017) prove a related result for calibrated scores.

Numerical check on our data: the identity must hold.

for nome in ("A", "B"):
    yy, pp = y[g == nome], pred[g == nome]
    m = metriche(yy, pp)
    p = yy.mean()
    fpr_formula = m["TPR"] * p * (1 - m["PPV"]) / (m["PPV"] * (1 - p))
    print(nome, "measured FPR", round(float(m["FPR"]), 4), "FPR from formula", round(float(fpr_formula), 4))

Output: A measured FPR 0.1228 FPR from formula 0.1228 and B measured FPR 0.1178 FPR from formula 0.1178. They coincide: the formula is an arithmetic identity, not an approximation.

Practical consequence. Fairness is not a property you "switch on": it is a choice among incompatible criteria, and the choice depends on the context and on the harm. Deciding which criterion to protect is a product and governance decision, not a technical detail, and it must be documented.

The regulatory and governance framework (essential map)

It is not legal advice: it is the map to know where to look. The texts are official sources of the European Union.

Always check the current text and the application dates on the official site (EUR-Lex for the regulations, nist.gov for the framework): the application of the AI Act is gradual, and deadlines may be updated. In this lesson I do not report them for exactly this reason.

The bridge with what we saw above. "Measuring" (the Measure function of the NIST RMF) is exactly what we did: break the metrics down by group, and state which fairness criterion is protected.

A minimal vocabulary

• Confusion matrix: 2×2 table of predictions against truth.
• Accuracy, precision, recall, FPR: the four derived measures (definitions in the text).
• Base rate: the frequency of the phenomenon in the population (p).
• Audit by group: the same metrics computed separately for each subgroup.
• Proxy: a variable that, though not sensitive, reflects another one.
• Decision threshold: the probability value above which you say "positive".
• Model card: a sheet documenting the purpose, data and limits of a model.

The seven entries are read in two groups: the first three are measures (matrix, metrics, base rate) and the last four are practices or phenomena (audit, proxy, threshold, card). When you find yourself discussing fairness with someone who has not read the lesson, start from the confusion matrix: everything else can be rebuilt from there.

Typical mistakes

Best practices

• Model card: purpose, training data, metrics per group, known limits.
• Dataset sheet: provenance, purpose, period, known gaps.
• Threshold chosen with the cost of errors, not left at the default 0.5.
• Human review where the error is serious or irreversible.
• Monitoring over time of the metrics per group, not just of the aggregate.
• Decision log: which fairness criterion, why, who approved it.

The first two sheets are documents, the third is a cost rule, the last three are continuing processes. None requires special infrastructure: they need discipline and a named person in charge, which is exactly the answer to the accountability question posed at the beginning of the lesson.

Mini-project

Take the code of the section on the audit by group and:

1. Add a third group C (base rate 0.25, separation 1.1, 1500 examples) and repeat the audit.
2. Compute the maximum difference in TPR between groups (the violation of equal opportunity).
3. Try to reduce it by choosing a different threshold per group (modello.predict_proba). What happens to the FPR and to precision? Check with Chouldechova's identity that the result is consistent.
4. Write the mini model card (8-10 lines) with the choice of criterion and the motivation.

The four steps go from calculation to decision: the first two extend the audit to one more group and measure the violation of equal opportunity, the third shows its price, the fourth forces you to put the choice in writing. If you have time for only one step, do the third: it is the one that turns a formula into a compromise.

Challenge

• Easy. Compute by hand the confusion matrix for 10 invented predictions and check with confusion_matrix.
• Medium. Plot the precision-recall curve by varying the decision threshold on group B.
• Hard. Repeat the experiment with equal base rates (0.3 and 0.3) but different separations: does the recall gap remain? What does the identity imply?
• Expert. Prove that with TPR = 1 and FPR = 0 (perfect classifier) all the criteria coincide, whatever the base rate.
• Enterprise. Write a quarterly audit procedure: which groups, which metrics, which alarm threshold, who decides.

The five challenges start from hand calculation and arrive at the business process. Hard and Expert are the most instructive: the first isolates the effect of how informative the signal is from the base rate, the second shows why the perfect classifier is the only case in which the incompatibility disappears.

Recap

Mind map. Confusion matrix → four metrics (accuracy, TPR, FPR, PPV). Audit = the same metrics per subgroup. Identity FPR = TPR·p·(1−PPV)/(PPV·(1−p)) → with different base rates, not all metrics can be equalised. Governance: GDPR, AI Act, NIST RMF.

3 concepts: confusion matrix; audit by group; incompatibility of the criteria. 3 mistakes: accuracy alone; the average as truth; "removing the column" as a cure. 3 best practices: model card; threshold from the cost of errors; monitoring by group. 1 exercise: the mini-project of this lesson.

Next lesson: how to turn a business problem into a data problem, and how to choose the threshold with real costs.

The same series also includes: 02 - From Model to Agent (the previous lesson), 04 - Implementing AI in a Company (the next lesson).