05 - Data Culture

Data Culture: Turning a Vague Problem into a Question Data Can Answer

Machine Learning Engineer course · Module 00 · Data-driven mindset · Lesson 5

Goal. Show that "being data-driven" does not mean having many charts, but knowing how to do three things: formulate verifiable questions, choose metrics that do not deceive, and accept that the data may contradict one's own idea. The reader learns a translation procedure "problem → question → metric → decision", and meets with numbers two classic traps: Simpson's paradox and Goodhart's law.

False myth. "Data speak for themselves." They do not: they answer the question you asked, and if the question is badly posed they answer well a wrong question.

Introduction

Many organisations collect data and decide out of habit. Others decide by intuition and use data to justify afterwards. A real data culture is different: important decisions start from an explicit question, with a measure agreed beforehand, and a shared way to say "I was wrong".

Roadmap. (1) What being data-driven means and what it does not. (2) The translation procedure. (3) The three types of metric. (4) Trap 1: Simpson. (5) Trap 2: Goodhart. (6) How to build the culture in a team.

What it is, why it exists, when it is NOT enough

Three levels of maturity

A common classification of data analysis distinguishes four questions, in order of difficulty:

Machine learning lives mostly in the two upper levels, but you cannot skip the lower ones: a predictive model built on poorly defined descriptive metrics inherits the mess.

Data-driven does not mean data-obedient

A useful distinction is between data-driven (the decision is determined by the data) and data-informed (the data is one input among others, together with context, experience and values). In repetitive, measurable decisions (which message works best) the former wins. In one-off decisions, with few observations or where unmeasurable values matter, the data informs but does not decide. When data is NOT enough: few cases; an effect the metric cannot see; an ethical decision.

Analogy: the car dashboard

The dashboard does not drive: it signals. If you look only at the speed and ignore the oil warning light, you are "driving in a data-driven way" and you will seize the engine. A data culture means choosing which instruments sit on the dashboard and when to look at them, with the humility to remember that the road, outside the windscreen, remains the reality.

Preparation

No installation: just Python. The examples use lists and dictionaries, deliberately, because modules 01 will teach the tools for tables.

Another translation, step by step

A second case, in another sector, to show that the procedure is general. The vague sentence: "Customer support is slow, improve it."

Two choices deserve attention. The first is the median instead of the mean: a few extremely complicated tickets shift the mean but do not represent the typical experience. The second is the segmentation by time band, decided before looking at the data because there is a causal reason (the workload changes over the course of the day): segmenting by any available column, just to see what comes out, is the quickest way to find "discoveries" that do not exist.

First example: the translation procedure

A boss says: "Sales are going badly, do an analysis." It is a problem, not a question. The procedure turns it into six steps:

1. Decision. Which choice depends on the answer? (If none, stop.)
2. Question. Reformulate as a question with subject, measure, period, comparison.
3. Metric. Define the number with formula, unit, source, time window.
4. Comparison. "Compared to what?" (last month, the same month last year, a control group.)
5. Threshold. Which result would change the decision?
6. Action plan. What will you do in each of the two cases?

The six steps are in order because each depends on the previous one: the question makes sense only if there is a decision, the metric only if the question is precise, the threshold only if the metric is defined. The step skipped most often is the fifth, the threshold, and skipping it allows any result to be read as a confirmation. Applied to the example:

Notice the phrase "for the same type of product". That is where the first trap hides, the next section.

Tukey (1962) put it like this: it is far better to have an approximate answer to the right question than an exact answer to the wrong question.

Theory and code: two traps hidden by the numbers

Trap 1: Simpson's paradox

Analogy. Two restaurants, A and B. A has a lower average rating than B. But in every dish category (pizzas, desserts, fish) A has a higher rating than B. How is this possible? Because A sells mostly difficult dishes (fish), which receive lower ratings in all restaurants, while B sells mostly easy dishes (pizzas). The average confuses the quality of the restaurant with the composition of the orders.

Real data. The historical case: comparison of two treatments for kidney stones (Charig et al., BMJ 1986, re-examined by Julious and Mullee 1994). Successes among patients, divided by stone size. The four counts below match those reported for this example in popular literature (Wikipedia), but I could not open the text of Charig et al.: I take them as textbook data, not as a check of the original article.

dati = {
    "A": {"piccoli": (81, 87), "grandi": (192, 263)},
    "B": {"piccoli": (234, 270), "grandi": (55, 80)},
}

for trattamento, gruppi in dati.items():
    for gruppo, (successi, totale) in gruppi.items():
        print(f"{trattamento} {gruppo:8s}: {successi:3d}/{totale:3d} = {successi/totale:.1%}")
    S = sum(s for s, _ in gruppi.values())
    T = sum(t for _, t in gruppi.values())
    print(f"{trattamento} OVERALL: {S}/{T} = {S/T:.1%}\n")

# Output (Python 3.14.4, two identical runs):
# A piccoli :  81/ 87 = 93.1%
# A grandi  : 192/263 = 73.0%
# A OVERALL: 273/350 = 78.0%
#
# B piccoli : 234/270 = 86.7%
# B grandi  :  55/ 80 = 68.8%
# B OVERALL: 289/350 = 82.6%

How to read the code. The dictionary dati contains, for every treatment and every group, the pair (successes, total). The double loop prints the success percentage of each group; after the inner loop, S and T add up successes and totals of all groups of the treatment, and the overall percentage is S/T. The delicate point is that the overall percentage is not the simple mean of the group percentages: it is a mean weighted with the sizes of the groups, and that is where the paradox is born. The format :.1% prints a ratio as a percentage with one decimal digit.

Output (exact arithmetic): A wins in both groups (93.1% versus 86.7% on small ones; 73.0% versus 68.8% on large ones), but loses overall (78.0% versus 82.6%).

Explanation, with the formula. The overall percentage is a weighted mean of the group percentages:

P_overall = Σ_g w_g · P_g, where w_g = n_g / n is the share of group g.

A treats mostly large cases (263 out of 350, that is 75% of its patients), which are harder for everyone; B treats mostly small cases (270 out of 350, 77%). The weights w_g differ between treatments, and this can reverse the comparison. There is no calculation error: there is a variable (size) that influences both which treatment is chosen and the outcome. It is called a confounding variable.

Checking it with the weights

To see the mechanism in numbers, we compute the group weights explicitly and then recompute the overall rate with the same weights for both treatments (this is called direct standardisation: it answers the question "what result would each treatment have on a population with the same composition?").

dati = {
    "A": {"piccoli": (81, 87), "grandi": (192, 263)},
    "B": {"piccoli": (234, 270), "grandi": (55, 80)},
}

pesi_propri = {}
for t, gruppi in dati.items():
    N = sum(tot for _, tot in gruppi.values())
    pesi_propri[t] = {g: tot / N for g, (_, tot) in gruppi.items()}
    complessivo = sum(pesi_propri[t][g] * s / tot for g, (s, tot) in gruppi.items())
    print(f"{t}: own weights {({g: round(w, 3) for g, w in pesi_propri[t].items()})} -> overall rate {complessivo:.3f}")

# common reference population: all 700 patients of both treatments
totali = {g: dati["A"][g][1] + dati["B"][g][1] for g in ("piccoli", "grandi")}
N = sum(totali.values())
pesi_comuni = {g: totali[g] / N for g in totali}
print("common weights:", {g: round(w, 3) for g, w in pesi_comuni.items()})
for t, gruppi in dati.items():
    standardizzato = sum(pesi_comuni[g] * s / tot for g, (s, tot) in gruppi.items())
    print(f"{t}: rate with common weights = {standardizzato:.3f}")

Output obtained:

A: own weights {'piccoli': 0.249, 'grandi': 0.751} -> overall rate 0.780
B: own weights {'piccoli': 0.771, 'grandi': 0.229} -> overall rate 0.826
common weights: {'piccoli': 0.51, 'grandi': 0.49}
A: rate with common weights = 0.833
B: rate with common weights = 0.779

How to read the code. The first loop computes for each treatment the share of patients in each group (pesi_propri) and rebuilds the overall rate as the sum of weight × group rate: the numbers coincide with the previous ones (0.780 for A, 0.826 for B), confirming that the weighted mean is the same as the ratio S/T. You also see why A "loses": it has weight 0.75 on the hard group (large), B has 0.23. The second block applies to both the same weights, computed on the whole population of 700 patients: now the composition is the same and the comparison is honest. A turns out better (about 0.83 versus 0.78). This is the quantitative version of "comparing at equal size".

When segmenting is instead the mistake

Simpson's paradox does not authorise segmenting always. It depends on the causal mechanism, and the diagram below shows it for the stones case: stone size (confounder) influences both which treatment is chosen and the outcome; comparing at equal size blocks the path through the confounder and leaves only the effect of the treatment.

There is also the opposite case, in which the variable you would segment by is a consequence of the treatment (a mediator), not a cause. Example: a drug shortens the length of hospital stay (consequence) and the length of stay influences the outcome. If you compare patients "at equal length of stay" you eliminate exactly the effect you wanted to measure. The two situations have the same arithmetic and opposite conclusions: no calculation, alone, tells which is the right one; you need knowledge of the process that generated the data. This is why the translation procedure starts from a decision and from a model of how the phenomenon works, not from a table.

Which number is the right one? It depends on the causal mechanism, not on the mathematics. Here the stone size is a common cause of the choice of treatment and of the result, so the comparison at equal size (the per-group numbers) is the one that answers the question "which treats better?". In another problem, with a different mechanism, the aggregate might be the correct one. Causal reasoning (module 02, probability) is what decides, not an inspection of the numbers.

Trade-off. Segmenting gives fairer answers but with smaller samples and more noise, and you can segment endlessly until you find anything (see p-hacking, lesson 06). Practical rule: segment by variables you have a causal reason to consider relevant, written down before looking at the data.

Trap 2: Goodhart's law

Statement (in the form of Strathern 1997): when a measure becomes a target, it ceases to be a good measure.

Analogy. A teacher evaluated on the pass rate of their students. If the pass rate is a good measure of teaching quality, the observation makes sense. If it becomes the target (bonuses, career), the cheapest way to raise it is to lower the standard, not to teach better. The measure keeps rising; the quality does not.

A minimal model. A team has one unit of effort to split between two activities: real work (which improves real quality) and "metric optimisation" (which improves the number but not the quality). Suppose optimisation yields double per unit of effort, because it is the cheapest route. Let s be the share of effort on real work (from 0 to 1).

• Real quality: Q(s) = s.
• Observed metric: M(s) = s + 2·(1 − s).

The two formulas share the same term s, but the metric adds a second contribution that produces no value: it is the "game" played on the measure, which yields double for every unit of effort. The rest of the model is the arithmetic of this difference.

The metric is always decreasing in s (because M(s) = 2 − s): to maximise the metric the best choice is s = 0, that is zero real work.

def qualita(s):          # real work
    return s

def metrica(s):          # real work + metric optimisation (which yields double)
    return s + 2 * (1 - s)

griglia = [i / 10 for i in range(11)]
migliore_per_metrica = max(griglia, key=metrica)
print("s that maximises the metric:", migliore_per_metrica)
print("metric at that point:", metrica(migliore_per_metrica), "| real quality:", qualita(migliore_per_metrica))
print("at the choice s=1:         ", "metric:", metrica(1.0), "| real quality:", qualita(1.0))

Output: s = 0.0, metric 2.0, quality 0.0; at s = 1.0: metric 1.0, quality 1.0. Whoever is rewarded on the metric doubles the number and zeroes the value.

The model is deliberately crude: it serves to show the mechanism, not to estimate a real phenomenon. The mechanism is this: as long as nobody optimises the measure, measure and quality are correlated; the moment someone optimises the measure, that correlation breaks.

How to defend yourself.

• Guardrail metrics: a metric that must remain stable while the other grows (for example, support response time and customer satisfaction).
• More metrics, not just one: it makes the game harder.
• Qualitative review on a sample: really read 20 cases.
• Change the measure over time, when you notice that it is being circumvented.

How to read the code. qualita(s) is the real value produced by real work (proportional to how much effort s you dedicate to it, between 0 and 1). metrica(s) is what is measured: the real effort plus double the effort spent "optimising the metric". griglia contains 11 values of s; max(griglia, key=metrica) finds the value that maximises the metric. The result is the extreme s = 0: whoever follows the measure abandons the real work. Since M(s) = 2 − s is decreasing, no intermediate value is a rational compromise for someone rewarded on the number.

The guardrail metric in action

A model is useful only if it shows its effect. We add a guardrail to the measure: if the real quality falls below a threshold (measured on a sample by a reviewer), the metric is zeroed. What is now the best choice for someone rewarded on the metric?

def qualita(s):
    return s

def metrica_grezza(s):
    return s + 2 * (1 - s)

def metrica_con_guardia(s, soglia_qualita=0.6, probabilita_di_controllo=1.0):
    """If the reviewer checks and the quality is below the threshold, the metric is 0."""
    controllo_scopre = qualita(s) < soglia_qualita
    atteso_penalizzato = probabilita_di_controllo * controllo_scopre
    return (1 - atteso_penalizzato) * metrica_grezza(s)

griglia = [i / 10 for i in range(11)]
for p_controllo in (0.0, 0.3, 1.0):
    scelta = max(griglia, key=lambda s: metrica_con_guardia(s, probabilita_di_controllo=p_controllo))
    print(f"check probability {p_controllo:.1f}: best choice for someone rewarded on the metric -> s = {scelta:.1f} (real quality {qualita(scelta):.1f}, metric {metrica_con_guardia(scelta, probabilita_di_controllo=p_controllo):.2f})")

Output obtained:

check probability 0.0: best choice for someone rewarded on the metric -> s = 0.0 (real quality 0.0, metric 2.00)
check probability 0.3: best choice for someone rewarded on the metric -> s = 0.0 (real quality 0.0, metric 1.40)
check probability 1.0: best choice for someone rewarded on the metric -> s = 0.6 (real quality 0.6, metric 1.40)

How to read the code. metrica_con_guardia multiplies the raw metric by 1 − (probability that the check discovers the problem). If nobody checks (probability 0) we get the choice of before: s = 0, zero quality. If you check always, the rational choice moves to s = 0.6, that is to the minimum quality required by the reviewer: the metric has stopped rewarding the zeroing of real work, even if the quality obtained is only the threshold, not the maximum. If you check 30% of the time, we are exactly at the tie (expected metric 1.40 both with s = 0 and with s = 0.6, and the code picks the first): a slightly lower check probability is enough for ignoring quality to become the best choice again. This shows two things: the guardrail works if it is credible, and it does not produce the optimum but a behaviour that satisfies the constraint. It is a good reason to use several guardrails and to rotate the checking criteria.

The three types of metric

A mature data culture chooses at least one of each type for every important decision.

How a culture is built in a group

It is not built with a platform. It is built with small and repeated habits:

1. Write the question and the threshold before looking at the data. It prevents adapting the decision to the result.
2. A glossary of metrics with formula, source, window, owner. Two people who say "active customer" meaning different things are the most common source of sterile discussions.
3. Share failed experiments too. If only successes are shown, the group learns that making mistakes is dangerous and stops experimenting.
4. A place to return to: a decision log with the question, the data, the outcome.
5. Train everyone in basic literacy: reading a chart, telling mean from median, knowing that a small sample is noisy. Module 02 is this.

Trade-off. Every rule costs time. A heavy procedure for every micro-decision blocks the organisation. Calibrate: rigour proportional to the cost of the error and to the reversibility of the decision. The five habits are in order of increasing cost: the first can be adopted tomorrow, the last takes months of training, and it is best to start with those that need no tools.

A minimal vocabulary

• Data-driven / data-informed: decision determined by the data / data as one of the inputs.
• Metric: a number defined with formula, unit, source and time window.
• Confounder: a variable that influences both the presumed cause and the effect.
• Mediator: a variable through which the cause produces the effect.
• Simpson's paradox: the aggregate comparison reverses with respect to the per-group one.
• Direct standardisation: comparison with common group weights.
• Goodhart's law: a measure that becomes a target stops measuring well.
• Guardrail metric: a measure that signals collateral damage.

The eight entries fall into three groups: how we decide (data-driven, metric), how numbers deceive (confounder, mediator, Simpson, Goodhart) and how we defend ourselves (standardisation, guardrail). When a colleague brings a chart, ask which of these terms apply to their case.

Typical mistakes

Best practices

• One dashboard per decision, not per data source.
• Every number shows how it is computed (one click to see the formula).
• Always compare with something (previous period, control group).
• Show the uncertainty (intervals, sample size), not just the point value.
• Review the metrics on a fixed schedule and retire those that guide no decision.

The five rules share one thread: every number must be open to questioning (how it is computed, compared with what, with how much uncertainty) and must serve a decision. The last two are the least practised; if you have to start with one, start by retiring metrics, because every useless dashboard has a cost in attention.

Mini-project

1. Choose a real (or plausible) decision from your work or study environment.
2. Fill in the six steps of the translation procedure in a table.
3. Indicate an outcome metric, a leading one, a guardrail one, with formula.
4. List two possible confounding variables for the comparison you want to make and how you would control them.
5. Describe a way in which your main metric could be circumvented (Goodhart) and the countermeasure.

The five steps follow the lesson: the first three build the question and the metric, the fourth applies Simpson and the fifth Goodhart. The fifth is the most useful to do in pairs with a colleague, because it is easier to imagine how to circumvent someone else's metric.

Challenge

• Easy. In the kidney stone data, compute the weights w_g of each treatment and rebuild the overall rate as a weighted mean, checking that it coincides.
• Medium. Invent a numerical Simpson example (two groups, two treatments) with your own numbers and check it in Python.
• Hard. Prove that the paradox requires the shares w_g to differ between treatments: if the shares are equal, a treatment that is better in every group is better overall too. (Hint: use the weighted-mean formula with the same weights.)
• Expert. In the Goodhart model replace the yield 2 with a parameter r > 1 and add a penalty on the gaming activity (if it is discovered with probability q, the metric is zeroed). For which (r, q) does honest behaviour become rational?
• Enterprise. Prepare a glossary of 8 metrics of your organisation with formula, owner, review frequency and associated guardrail metric.

Easy and Medium consolidate the computation of the weights, Hard proves it, Expert explores Goodhart's law with parameters, Enterprise brings the glossary into your work. The Hard challenge is the most formative: it makes clear that the paradox arises only when the weights differ.

Recap

Mind map. Vague problem → decision → verifiable question → metric (outcome, leading, guardrail) → comparison → threshold → plan. Traps: Simpson (aggregate that deceives because of different weights), Goodhart (a measure that becomes a target).

3 concepts: translation procedure; confounding; guardrail metrics. 3 mistakes: undefined metric; mean without segments; optimised proxy. 3 best practices: question and threshold before the data; glossary of metrics; decision log. 1 exercise: the mini-project of this lesson.

Next lesson: "Test and learn": how the confounding problem is solved by running a controlled experiment.

The same series also includes: 04 - Implementing AI in a Company (the previous lesson), 06 - Test and Learn: Controlled Experiments and A/B Tests (the next lesson).