93 - Prompt Engineering for Developers: the Levers That Matter, the Techniques With a Paper, and Use on Data and Code

A prompt is a specification: it says what is needed, in what form, with what constraints and with what reference examples. It is not a magic formula and it is not a string of adjectives: formulas such as "act as a world-class expert" have, in this lesson, no measurement to justify them (if you want to know whether they move anything on your task, there is the measurement protocol further down). The levers I propose are four: context, format, examples, constraints. On top of these sit techniques with a literature behind them, each with a token cost worth knowing before adopting it. And underneath everything sits a measurement protocol, without which every change to a prompt is a bet and not an experiment.

What I ran, and what I did not. Code tested on 2026-11-12 with Python 3.14.4, pandas 3.0.3 and openai (client) in a temporary environment outside the repository. No model was called, not even Ollama: wherever an answer from the model is needed there is a fake stand-in, declared as such, rule-based or scripted. The outputs therefore prove the code around the model (prompt construction, parameters sent, measurement, checks, execution) and not the quality of a model: every accuracy figure you see below belongs to the stand-in, not to a real model. In production code the endpoint is Ollama's compatible one (http://localhost:11434/v1) and the model llama3.1:8b, but the same prompts apply on any endpoint; only the client line changes. Every listing that was run was run twice with identical output; the listings of a single section are pieces of the same file. The bibliographic references are to the original papers on arXiv: where I cite a result, I cite the work that measured it.

The analogy: the brief to an external contractor

Imagine handing a job to a brilliant, fast contractor who does not know your company and cannot ask you questions. You have one chance: the brief. If you write "make me a presentation on our results", you will get something generic and formally impeccable. If you write "ten slides, audience: non-technical investors, sober tone, one figure per slide taken from the attachment, no future projections, 16:9 format", you will get something usable.

The four things you added are not stylistic. You added the context (who the audience is), the format (ten slides, 16:9, one figure per slide), the constraints (no projections, data from the attachment) and, had you attached a previous presentation, the examples. A language model is in exactly this situation: it does not know your context and cannot ask. The prompt is the brief, and the same rule applies - the quality of what comes back depends a lot on the precision of what you ask.

The analogy holds on a less comforting point too: a perfect brief does not guarantee perfect work. You still have to check, and the final part of this lesson is devoted to how you check when the work concerns data and code.

The four levers, before and after

It is worth seeing the four levers applied to the same task, because the difference between before and after is the operational content of this lesson.

Before - a plausible request that produces an unusable result:

PROMPT_1 = "Analyse this review and tell me what you think of it: {text}"

What does not work: you do not know who will read the answer, so the model picks a register at random; you do not know what form it will come back in, so downstream you have to interpret prose; you do not know what to do if the review is ambiguous, so the model will decide; and "what you think of it" is not a task, it is an invitation to ramble.

After - the four levers, one per block:

PROMPT_1 = "Analyse this review and tell me what you think of it: {text}"

CONTEXT = """\
You are the review triage system of an online shop. Your outputs are read
by a program, not by a person, and route the review to the product team,
the logistics team or the archive."""

FORMAT = """\
Classify the review according to this JSON schema, and NOTHING ELSE:
{{"rationale": "<max 2 sentences, quote the words of the review>",
  "topic": "product" | "shipping" | "support" | "other",
  "sentiment": "positive" | "neutral" | "negative",
  "action": "product_team" | "logistics_team" | "archive" | "human_review"}}"""

EXAMPLES = """\
Examples:

Review: "Arrived in two days, perfect packaging. The fabric itches."
{{"rationale": "Praises the shipping ('two days', 'perfect packaging') but
criticises the fabric ('itches'): the problem is the product.",
  "topic": "product", "sentiment": "neutral", "action": "product_team"}}

Review: "It never arrived and nobody answers."
{{"rationale": "Failed delivery ('never arrived') plus no reply:
the primary problem is the shipping.",
  "topic": "shipping", "sentiment": "negative", "action": "logistics_team"}}

Review: "ok"
{{"rationale": "Text too short to determine the topic.",
  "topic": "other", "sentiment": "neutral", "action": "archive"}}"""

CONSTRAINTS = """\
Constraints:
- The rationale comes BEFORE the classification and quotes the words of the
  review.
- If the text has content but does not let you choose between two topics, use
  action="human_review": do not guess.
- If there are two topics and one prevails, choose the one the review insists on more.
- Do not invent details that are not in the text.
- Do not add text outside the JSON."""

PROMPT_2 = "\n\n".join([CONTEXT, FORMAT, EXAMPLES, CONSTRAINTS, 'Review: "{text}"'])

parts = {"context": CONTEXT, "format": FORMAT, "examples": EXAMPLES, "constraints": CONSTRAINTS}
total = len(PROMPT_2.format(text=""))
for name, text in parts.items():
    print(f"{name:11} {len(text.format()):4} characters")
print(f"PROMPT_1    {len(PROMPT_1.format(text='')):4} characters")
print(f"PROMPT_2    {total:4} characters, of which examples {len(EXAMPLES.format()) / total:.0%}")
print(PROMPT_2.format(text="Nice but the zip broke").splitlines()[-1])

# Output:
# context      181 characters
# format       319 characters
# examples     651 characters
# constraints  388 characters
# PROMPT_1      54 characters
# PROMPT_2    1557 characters, of which examples 42%
# Review: "Nice but the zip broke"

The prompt is now built in pieces, one per lever, and the four variables can be read one by one; the output says how much each one weighs: context, format and constraints fit in about 890 characters, the examples alone take up 651 (42% of the total of 1557). It is a measurement in characters, not in tokens, and it serves to show the order of magnitude of the cost of the examples, which is paid on every call. Four observations on what has changed, and why.

The context says what the output is for, not only who reads it. "Your outputs are read by a program and route the review" is a different piece of information from "answer in a professional manner": it defines the consequence of a wrong answer. I have not measured how much it changes the behaviour of a real model; it is the reason I write it this way, and the measurement protocol serves to verify it on your task.

The format uses an enum, not a description. Listing the allowed values between vertical bars closes the space of answers. It is the "prose" version of the Literal from the lesson on structured output, and when the endpoint supports an enforced schema it is better done with that, but the prompt must say it anyway, because it also describes the criterion of choice and not only the values.

The examples cover the ambiguous cases, not the easy ones. The first example is a mixed case (praises the shipping, criticises the product) and shows the precedence rule in action. The third is a degenerate case ("ok") and shows what to do when there is no information. An easy example teaches little; an edge example shows a rule that a sentence struggles to express (this is the reasoning I follow, not a measurement). In the constraints I removed a contradiction of the previous version: requiring a quote and at the same time providing an example that quotes nothing ("ok"); now the quote applies when there is something to quote, and "human_review" is reserved for texts that have content but are ambiguous.

The rationale comes before the classification. It is the same choice as in the lesson on structured output, and the reason is the same: models generate left to right, so a field placed before the decision influences it, and one placed after justifies it. Here there is one more effect: the obligation to quote the words of the review makes the answer verifiable - if the quote is not in the text, the classification rests on nothing, and you find out with a string search.

Decoding parameters: which ones to touch

The four levers concern what you ask. The decoding parameters concern how the model generates, and few of them matter.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

def classify(text: str) -> str:
    r = client.chat.completions.create(
        model="llama3.1:8b",
        messages=[{"role": "user", "content": PROMPT_2.format(text=text)}],
        temperature=0.0,        # classification task: no creativity
        max_tokens=300,         # hard ceiling: avoids rambling and surprise costs
        stop=["\nReview:"],     # cuts off if the model tries to invent another case
    )
    return r.choices[0].message.content

# Output:
# {"topic": "product"}
# {'model': 'llama3.1:8b', 'max_tokens': 300, 'stop': ['\nReview:'], 'temperature': 0.0}
# 1 message, 1579 characters

To test the listing without a server I gave the client a simulated transport that records the request; the output has three lines, written by the test harness around the listing: the stand-in's answer, the body sent (excluding the text of the prompt) and the length of the message, 1579 characters. The parameters go out as I wrote them: temperature 0.0, max_tokens 300 and the stop sequence. Ollama's compatibility page lists stop, temperature, top_p and max_tokens among the supported fields of the /v1/chat/completions endpoint; what a real model does with them I did not test. top_p does not appear in the listing, on purpose (see below).

temperature controls how far the generation moves away from the most probable token. At zero the model always takes the most probable one: it is what you want to classify, extract, decide. Values around 0.7-1.0 are a convention of use for writing text where variety is a value, not a measured threshold. The point that gets confused: a high temperature does not make the model "more creative" in the sense of smarter, it makes it less predictable, and on a deterministic task this is just noise.

top_p limits the choice to the most probable portion of the distribution. It is another way of controlling the same thing, and the practical rule is: adjust one, not both: it is a rule of prudence, because moving them together produces behaviours that are hard to attribute. With temperature=0 the choice is always the most probable token and top_p does not change the outcome, so leaving it at its default value is the cleanest choice.

max_tokens is a hard ceiling on the length of the output, and it should always be set. Not for quality: for cost and latency. A model that rambles costs money and makes people wait.

stop is the sequence that interrupts the generation. It is underrated and very useful with few-shot prompts: a model that has seen three "Review → JSON" pairs may carry on by inventing a fourth review. Putting "\nReview:" among the stop sequences closes the problem in one parameter.

The measurement protocol: how to know whether a change worked

This section is the reason the rest of the lesson makes sense. Without a measurement, "I improved the prompt" means "the answer I looked at pleased me more than before", which on a probabilistic system is not information.

Little is needed: twenty or thirty real cases with the correct outcome annotated by hand, and a loop that compares two versions.

import json, statistics
from dataclasses import dataclass

@dataclass(frozen=True)
class Case:
    text: str
    topic: str
    action: str

CASES: list[Case] = [
    Case("Arrived broken, packaging crushed", "shipping", "logistics_team"),
    Case("Lovely but the zip came open right away", "product", "product_team"),
    Case("ok", "other", "archive"),
    Case("It never arrived and nobody answers", "shipping", "logistics_team"),
    Case("The fabric is rough, the size is right", "product", "product_team"),
    Case("Right size, but the courier left the parcel out in the rain", "shipping", "logistics_team"),
    Case("Great purchase, recommended", "product", "archive"),
    Case("I wrote to customer service three times, never an answer", "support", "human_review"),
    # ... in a real case: twenty or thirty, taken from real cases and annotated by you
]

# --- stand-ins for the test: in your code they are the model call and the JSON extraction
calls = 0
def call_model(prompt: str) -> str:
    """FAKE model: keyword rules, with a format error every 11 calls."""
    global calls
    calls += 1
    if prompt.startswith("Analyse"):
        return "It seems an overall positive review, with some reservations."
    if calls % 11 == 0:
        return "Here is the classification requested: unfortunately I cannot decide."
    t = prompt.rsplit('Review: "', 1)[1].lower()
    if t.startswith("ok"):
        topic, action = "other", "archive"
    elif "never arrived" in t or "arrived broken" in t:
        topic, action = "shipping", "logistics_team"
    elif "customer service" in t:
        topic, action = "support", "human_review"
    elif "size" in t or "zip" in t or "fabric" in t:
        topic, action = "product", "product_team"       # gets the rain case wrong
    else:
        topic, action = "product", "product_team"
    return json.dumps({"rationale": "x", "topic": topic, "sentiment": "neutral", "action": action})

def extract_json(text: str) -> str:
    i, j = text.find("{"), text.rfind("}")
    return text[i:j + 1] if 0 <= i < j else text

# --- the measurement protocol
def evaluate(prompt: str, cases: list[Case], passes: int = 3) -> dict:
    """Runs every case 'passes' times. Returns mean, spread and conformity."""
    topic_scores, action_scores, conforming = [], [], []
    for _ in range(passes):
        t = a = c = 0
        for case in cases:
            raw = call_model(prompt.format(text=case.text))
            try:
                d = json.loads(extract_json(raw))
            except Exception:
                continue                      # non-conforming: does not count as correct
            c += 1
            t += int(d.get("topic") == case.topic)
            a += int(d.get("action") == case.action)
        n = len(cases)
        topic_scores.append(t / n); action_scores.append(a / n); conforming.append(c / n)
    return {
        "conformity": round(statistics.mean(conforming), 3),     # does the format hold?
        "topic": round(statistics.mean(topic_scores), 3),        # does it classify well?
        "action": round(statistics.mean(action_scores), 3),      # does it route well?
        "action_spread": round(max(action_scores) - min(action_scores), 3),
    }

print("v1:", evaluate(PROMPT_1, CASES))
print("v2:", evaluate(PROMPT_2, CASES))

# Output:
# v1: {'conformity': 0.0, 'topic': 0.0, 'action': 0.0, 'action_spread': 0.0}
# v2: {'conformity': 0.917, 'topic': 0.792, 'action': 0.667, 'action_spread': 0.125}

The output reads like this. Version 1 (none of the four levers) returns prose: conformity 0, hence topic and action at zero, because downstream no program would know how to read it. Version 2 has conformity 0.917 (the stand-in gets the format wrong once every eleven calls, by construction), accuracy on the topic 0.792 and on the action 0.667, and a spread of 0.125 across the three passes: the same prompt, on the same eight cases, gave different scores only because in different passes the format error fell on different cases. All of this belongs to the stand-in: it shows that the method separates conformity, correctness and variability, not that prompt 2 is good. Four rules that make this comparison honest, and that matter more than the code.

Change one lever at a time. If in the same version you add the examples and rewrite the constraints, you will know it improved and not why, and next time you will redo both things, even the useless one, paying for it on every call.

Repeat each case several times and look at the spread. Even with temperature=0 some variability remains, for the reasons seen in the lesson on traps. A difference of three points between two versions on a single pass is not a difference: it is noise.

Keep the cases separate from development. If you tweak the prompt while looking at the same twenty cases, after ten iterations the prompt is tailored to those twenty cases and the measurement no longer predicts anything. The minimum practice is to keep a second set of ten cases that you look at only at the end.

Measure conformity separately from correctness. A prompt that produced valid JSON in 70% of cases and classified that 70% perfectly (hypothetical example) is a broken prompt, and an aggregate score would hide it.

Advanced techniques: what the papers say, and what they cost

Beyond the four levers there are techniques with a literature behind them. It is worth knowing them with their mechanism and their price, because the difference between applying them on purpose and applying them at random is large.

Few-shot: examples in the prompt

Putting a few worked cases in the prompt is the technique documented by Brown and colleagues (Language Models are Few-Shot Learners, arXiv:2005.14165), the work that popularised the idea that a model can adapt to a task from examples in the context, without retraining. Mechanism: the examples define the task operationally, showing what a description cannot say. Cost: high and recurring: in this lesson's prompt the examples are 42% of the characters (measured above) and are paid on every call. When it pays off: when the format is particular, when style matters, and above all when there are ambiguous cases whose handling has to be shown. When not: when a sentence of constraint obtains the same effect. The check is that of the measurement protocol: remove the examples and measure.

Chain of thought

Asking the model to spell out the steps before the answer is the technique of Wei and colleagues (Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, arXiv:2201.11903). There is also the "zero-cost" variant of Kojima and colleagues (Large Language Models are Zero-Shot Reasoners, arXiv:2205.11916), which obtains a similar effect by merely adding "let's think step by step" without examples. Mechanism: the answer is generated after the steps, so the steps are part of the context it rests on. Cost: the tokens of the reasoning, in output, on every call. How much it weighs depends on the length of the reasoning and on the task: measure it on yours, do not trust a generic figure. When it pays off: arithmetic, multi-step problems, choices with intertwined conditions. When not: classification, extraction, tasks where the answer does not need steps - there the reasoning is a certain cost with an uncertain benefit: there is no reason to expect a gain, and on a real task you must measure whether the result gets worse.

An important practical note: with a structured output, chain of thought is implemented as a field before the decision, not as free text. It is the reason for the rationale field in the prompt at the beginning of this lesson.

Self-consistency

Generating several independent answers and taking the most frequent one is the technique of Wang and colleagues (Self-Consistency Improves Chain of Thought Reasoning in Language Models, arXiv:2203.11171). Mechanism: the intuition of the work is that a complex problem admits several reasoning paths that lead to the same correct answer, while errors tend to scatter; with a non-zero temperature the paths differ, and the majority vote exploits this asymmetry. Cost: k times everything, where k is the number of samples: it is the most expensive technique in the list. When it pays off: on a subset of high-value cases, where being wrong costs more than five calls.

from collections import Counter

def with_self_consistency(prompt: str, k: int = 5) -> dict:
    """Votes among k answers. Costs k times; use it only where it is needed."""
    answers = [extract_decision(call_model(prompt, temperature=0.8))
               for _ in range(k)]
    tally = Counter(r for r in answers if r is not None)
    if not tally:
        return {"decision": None, "agreement": 0.0}
    decision, how_many = tally.most_common(1)[0]
    return {"decision": decision,
            "agreement": how_many / k,             # 1.0 = unanimity
            "alternatives": dict(tally)}

def route(outcome: dict, threshold: float = 0.8) -> str:
    """With 5 samples: 4-1 passes (0.8), 3-2 does not (0.6)."""
    if outcome["decision"] is None or outcome["agreement"] < threshold:
        return "human_review"
    return outcome["decision"]

The listing has two functions: one votes, the other decides where to send the case. In the version I originally wrote the comment said "agreement < 0.6 → human review", but with five samples a 3-2 split is worth exactly 0.6 and would never have been sent back: the right threshold to catch it is 0.8. Tested with four scripts of answers (no model):

# Output:
# unanimous    {'decision': 'a', 'agreement': 1.0, 'alternatives': {'a': 5}} -> a
# 4-1          {'decision': 'a', 'agreement': 0.8, 'alternatives': {'a': 4, 'b': 1}} -> a
# 3-2          {'decision': 'a', 'agreement': 0.6, 'alternatives': {'a': 3, 'b': 2}} -> human_review
# unreadable   {'decision': None, 'agreement': 0.0} -> human_review

The biggest gain of this technique is not the most frequent answer: it is the degree of agreement, which is a measure of uncertainty that the model does not normally give you. A case on which five samples split 3-2 is a case you should not automate, and knowing it is worth more than the decision.

Decomposition (least-to-most)

Breaking a problem into sub-problems and solving them in order, using the previous answers, is the technique of Zhou and colleagues (Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, arXiv:2205.10625). Mechanism: each step is simpler than the whole problem, and the intermediate answers become verifiable context. Cost: one call per sub-problem. When it pays off: when you know the decomposition: and in that case you write it in the code, with a chain of calls, instead of asking the model for it. It is the connection with the first lesson of the module: if the sequence is known, it is a pipeline.

ReAct and reflection

ReAct (Yao et al., arXiv:2210.03629) alternates reasoning and action towards an external environment: it is the backbone of the agents of the whole of module 09, and I cite it here to close the circle - the prompting technique and the agentic architecture are the same idea at two levels of abstraction. Reflexion (Shinn et al., arXiv:2303.11366) adds a step of critique of one's own attempt, which becomes context for the next attempt.

On the latter a precise warning is needed, because it is the technique adopted with the most enthusiasm and most often without benefit. The work of Huang and colleagues (Large Language Models Cannot Self-Correct Reasoning Yet, arXiv:2310.01798) documents that, in the absence of an external signal of correctness, self-correction can worsen the answer: the model changes a right answer because it was asked to review it. The operational consequence is clear-cut: reflection works when there is something objective to restart from - the output of a compiler, a failing test, a written specification, a comparison with a source. It does not work as "think harder".

The cost table, to decide

def chars(prompt: str) -> int:
    return len(prompt.format(text=""))

without_examples = "\n\n".join([CONTEXT, FORMAT, CONSTRAINTS, 'Review: "{text}"'])
base, with_examples = chars(without_examples), chars(PROMPT_2)
K = 5

rows = [
    ("technique", "input (characters)", "output"),
    ("bare task (PROMPT_1)", chars(PROMPT_1), "as requested"),
    ("three levers: context, format, constraints", base, "as requested"),
    ("with three examples (PROMPT_2)", f"{with_examples}  (+{with_examples - base})", "as requested"),
    ("chain of thought", "unchanged", "longer: to be measured on your task"),
    (f"self-consistency, k={K}", f"{K} x {with_examples} = {K * with_examples}", f"{K} x"),
]
for r in rows:
    print(f"{r[0]:44} {str(r[1]):26} {r[2]}")

# Output:
# technique                                    input (characters)         output
# bare task (PROMPT_1)                         54                         as requested
# three levers: context, format, constraints   904                        as requested
# with three examples (PROMPT_2)               1557  (+653)               as requested
# chain of thought                             unchanged                  longer: to be measured on your task
# self-consistency, k=5                        5 x 1557 = 7785            5 x

The figures in the output are characters measured on this lesson's prompt, not tokens, and they apply to this prompt; the multipliers (5 x for self-consistency with k=5) come from the definition of the technique. Reflection does not appear: its cost depends on how many attempts you allow, and its benefit on the external criterion you have available. The reading is the only general recommendation I feel able to give on prompt engineering: the four levers cost little and are always worth it; everything else is a multiplier that has to be justified with a measurement.

GenAI on data: the code-as-answer pattern

Here we move from technique to application, and there is a rule that by itself is worth the whole section: a model must never calculate a number. It must write the code that calculates it.

The reason is that the two things have incomparable guarantees (it is a design argument, not a measurement on a specific model). "What is the mean of this column?" produces a plausible, unverifiable number, generated by a system that cannot do arithmetic but knows what numbers look like. "Write the code that calculates the mean of this column" produces code that you run: the number comes from pandas, and if the code is wrong you see it because it raises an exception or because the result does not pass a check.

import pandas as pd

ANALYSIS_PROMPT = """\
You are a data analysis assistant. Write Python code, not answers in prose.

The DataFrame is called `df` and already exists. Its schema is:
{schema}

Non-negotiable constraints:
- Do NOT invent column names: use only those in the schema above.
- Do NOT write numbers into the code taken "by eye" from the data.
- Assign the final result to the variable `result`.
- If the question cannot be answered with these columns, write
  `result = "CANNOT BE ANSWERED: " + reason` and nothing else.
- No imports besides pandas (already imported as pd) and numpy (as np).

Question: {question}

Reply ONLY with the code, without explanations and without markdown blocks.
"""

def describe_schema(df: pd.DataFrame, example_rows: int = 2) -> str:
    """The REAL schema, generated from the data: the model must not guess the columns."""
    lines = [f"- {c}: {t}" for c, t in df.dtypes.astype(str).items()]
    return ("columns:\n" + "\n".join(lines) +
            f"\n\ntotal rows: {len(df)}\n\nfirst {example_rows} rows:\n"
            + df.head(example_rows).to_string())

The schema generated from the data is the most important line of this block. It is the practice that in the literature on text-to-SQL systems is called schema linking, and it solves the most common flaw of assisted analyses: the model invents a plausible column name (order_date instead of date_ordered) and the code fails - or worse, it uses an existing but wrong column and produces a credible number.

Execution requires just as much discipline. I rewrote the listing after trying it, because of a flaw that the first version hid: passing exec two separate dictionaries (global names and local names) makes the generated code unable to use df inside a lambda or a comprehension, because there names are looked up in the globals. The version below uses a single dictionary.

import builtins
import re
import numpy as np
import pandas as pd

ALLOWED_BUILTINS = {n: getattr(builtins, n) for n in (
    "len", "sum", "min", "max", "round", "abs", "sorted", "range", "float", "int", "str",
    "list", "dict", "set", "tuple", "bool", "enumerate", "zip", "isinstance")}

FORBIDDEN = ("import os", "import sys", "import subprocess", "open(",
             "eval(", "exec(", "__import__", "to_csv", "to_sql")

def clean_code(code: str) -> str:
    """Removes any markdown fences around the code."""
    return re.sub(r"^```(?:python)?\s*|\s*```$", "", code.strip())

def run_analysis(df: pd.DataFrame, question: str) -> dict:
    code = clean_code(call_model(ANALYSIS_PROMPT.format(
        schema=describe_schema(df), question=question), temperature=0.0))

    # 1. static checks BEFORE running
    found = [v for v in FORBIDDEN if v in code]
    if found:
        return {"ok": False, "error": "code_not_allowed", "constructs": found}

    # 2. column names: every df['...'] must exist in the DataFrame
    cited = set(re.findall(r"""df\[['"]([^'"]+)['"]\]""", code))
    nonexistent = sorted(cited - set(df.columns))
    if nonexistent:
        return {"ok": False, "error": "nonexistent_columns", "columns": nonexistent}

    # 3. run on a COPY, with reduced builtins, in ONE SINGLE dictionary of names
    space = {"__builtins__": ALLOWED_BUILTINS, "df": df.copy(), "pd": pd, "np": np}
    try:
        exec(code, space)
    except Exception as e:
        return {"ok": False, "error": f"{type(e).__name__}: {e}", "code": code}

    return {"ok": True, "result": space.get("result"), "code": code, "space": space}

I tested the listing with a fake model that returns code written by me, on a DataFrame of five rows. First the schema generated from the data, then five scenarios, then the single-dictionary test, then the case of the empty sum:

import os, tempfile

df = pd.DataFrame({"category": ["books", "books", "games", "games", "games"],
                   "amount": [10, 20, 5, 15, 30]})
print(describe_schema(df))
print("---")
def trial(name, code):
    globals()["call_model"] = lambda prompt, temperature: code
    r = run_analysis(df, "question")
    r.pop("space", None); r.pop("code", None)
    print(f"{name:22}", r)
trial("correct", "```python\nresult = df.groupby('category')['amount'].sum().idxmax()\n```")
trial("open()", "result = open('/etc/hostname').read()")
trial("invented column", "result = df['price'].sum()")
trial("import with 2 spaces", "import  os\nresult = os.getcwd()")
tmp = tempfile.mkdtemp(); os.chdir(tmp)
open("secret.csv", "w").write("a,b\nTHIS_SHOULD_NOT_LEAK,2\n")
trial("pd.read_csv", "result = pd.read_csv('secret.csv').iloc[0, 0]")
print("---")
code = "result = int(df['amount'].apply(lambda v: v > df['amount'].mean()).sum())"
for name, ns in {"two dictionaries": ({"__builtins__": ALLOWED_BUILTINS}, {"df": df, "pd": pd}),
                 "just one": ({"__builtins__": ALLOWED_BUILTINS, "df": df, "pd": pd}, None)}.items():
    try:
        g, l = ns
        exec(code, g, l) if l is not None else exec(code, g)
        print(f"{name:17}", (l or g)["result"])
    except Exception as e:
        print(f"{name:17}", type(e).__name__ + ":", e)
print("---")
empty = df[df["category"] == "Books"]            # 'Books' with a capital: no rows
print("sum:", empty["amount"].sum(), "| mean:", empty["amount"].mean(), "| rows:", len(empty))

# Output:
# columns:
# - category: str
# - amount: int64
#
# total rows: 5
#
# first 2 rows:
#   category  amount
# 0    books      10
# 1    books      20
# ---
# correct                {'ok': True, 'result': 'games'}
# open()                 {'ok': False, 'error': 'code_not_allowed', 'constructs': ['open(']}
# invented column        {'ok': False, 'error': 'nonexistent_columns', 'columns': ['price']}
# import with 2 spaces   {'ok': False, 'error': 'ImportError: __import__ not found'}
# pd.read_csv            {'ok': True, 'result': 'THIS_SHOULD_NOT_LEAK'}
# ---
# two dictionaries  NameError: name 'df' is not defined
# just one          2
# ---
# sum: 0 | mean: nan | rows: 0

The output reads in three parts. First, the three levels of defence work on the cases they were meant for: the correct code returns "games", the one with open( is stopped by the static check, the invented column price by the check on names. Second, the limits: import  os with two spaces is not caught by the substring search, and is stopped only by execution ("__import__ not found"); and above all pd.read_csv('secret.csv') passes all the checks and reads a file, because pd is in the namespace. Third, the single-dictionary test: with two dictionaries the code with the lambda gives NameError, with a single one it returns 2 (two amounts above the mean); Python's documentation explains it, because with two objects the code is executed "as if it were embedded in a class definition". The last line shows the case of the sum over zero rows, which I use further down.

Three levels of defence, and each stops a different thing. The static checks stop code that touches the filesystem or the network: not because the model is malicious, but because it might have read a hostile instruction in a data field - it is the prompt injection of the lesson on traps, with the data as the vector. The check on column names catches the most frequent error before it produces a wrong result. Execution on a copy and in a restricted namespace limits the damage of whatever passes the first two.

It has to be said clearly, because it is a responsibility that cannot be delegated: exec with a dictionary of reduced builtins is not a sandbox, and the output above proves it with pd.read_csv. It is a reduction of the surface, useful and insufficient. The check on column names also has a limit: it looks only at df['name'], not df.name nor df[['a', 'b']]; whatever slips through is stopped, at best, by an exception in the third step. If the generated code runs on data or on a machine that matter, you need real isolation: a separate process with minimal permissions, a container, or a dedicated environment with memory and time limits.

The check that unmasks an invented analysis

Even with the code executed, one question remains: how do you know the code answers your question and not another one? The most effective check costs little and is this: ask the code to produce the control numbers as well, and verify them yourself.

# addition to the prompt
ADDITION = """\
Besides `result`, always assign:
- `rows_used`: how many rows of the DataFrame enter the final calculation
- `filters_applied`: list of strings describing each filter applied
"""

# check after the run
def coherent(df, space: dict) -> list[str]:
    alarms = []
    n = space.get("rows_used")
    if n is None:
        alarms.append("the code does not declare how many rows it used")
    elif n == 0:
        alarms.append("zero rows used: the result rests on no data")
    elif n == len(df) and space.get("filters_applied"):
        alarms.append("declares filters but uses all rows: filter not applied?")
    return alarms

Tested on four pieces of code written by hand (same DataFrame, same fake model):

import pandas as pd
df = pd.DataFrame({"category": ["books", "books", "games", "games", "games"], "amount": [10, 20, 5, 15, 30]})
def trial(name, code):
    globals()["call_model"] = lambda prompt, temperature: code
    r = run_analysis(df, "question")
    print(f"{name:28} result={r['result']!r:6} alarms={coherent(df, r['space'])}")
trial("right filter", "d = df[df['category'] == 'books']\nresult = int(d['amount'].sum())\nrows_used = len(d)\nfilters_applied = ['category == books']")
trial("wrong filter (Books)", "d = df[df['category'] == 'Books']\nresult = int(d['amount'].sum())\nrows_used = len(d)\nfilters_applied = ['category == Books']")
trial("filter declared, not used", "result = int(df['amount'].sum())\nrows_used = len(df)\nfilters_applied = ['category == books']")
trial("check numbers missing", "result = int(df['amount'].sum())")

# Output:
# right filter                 result=30     alarms=[]
# wrong filter (Books)         result=0      alarms=['zero rows used: the result rests on no data']
# filter declared, not used    result=80     alarms=['declares filters but uses all rows: filter not applied?']
# check numbers missing        result=80     alarms=['the code does not declare how many rows it used']

The n == 0 case is the one worth always looking for: a filter that is too narrow (a category name spelled differently, a date in another format) produces an empty DataFrame, and a mean over zero rows becomes NaN or raises - but a sum over zero rows becomes zero (measured in the last line of the previous output: sum 0, mean nan), and zero is a perfectly credible number to report in a meeting. In the test the second case is exactly this: the filter written with "Books" instead of "books" gives 0 rows and the check flags it. The third and fourth show the other two alarms: filters declared but all rows used, and control numbers missing.

GenAI on code: where it pays off and where it costs

The last part concerns everyday use, and it is the one on which I have more to say in terms of judgment and less in terms of technique, because the technique is the one above and the judgment is everything. What follows is my design assessment, not a measurement.

The model is strong where the code is predictable and weak where you need to understand a system. This sentence alone explains the map.

Where the value is high and the risk low - the code you would have written yourself in the same way. Format conversions, configuration boilerplate, parametric tests on a pure function, mechanical migrations, regular expressions with reference examples, documentation of a function that can be read in full. Here the check is immediate: the test passes or it does not, the regular expression captures the cases you listed or it does not.

Where the value is high and the risk medium - the code you have to read carefully. A new function with a signature you defined, a query you then verify on real data, a local and circumscribed refactoring. Here the rule is one: if you are not able to review that code, you are not using it, you are adopting it. And adopting code you cannot evaluate is technical debt taken on in a moment of optimism.

Where the risk is high - everything that requires knowing things that are not in the context. Changes that cross several modules, architectural choices, code on an implicit domain constraint ("here VAT is calculated this way for a reason that lives in an email from 2019"), and above all security: a model produces with the same confidence a correct validation and one that looks correct.

The verification loop is not negotiable, and its form is this:

1. You write the signature and the tests, before the code. The contract is defined by you.
2. Ask for the implementation, with the tests in the prompt. The model has a target.
3. Run the tests. If they are red, pass the real error to the model: it is the external criterion that arXiv:2310.01798 talks about.
4. With the tests green, read the code. If you do not understand it, rewrite it.
5. Add a test for the edge case the model ignored.

Step 1 is the one that gets skipped and that makes the difference. Asking for the implementation and the tests from the same model means having the task and the marking written by the same hand: the tests will pass, because they were written for that code. Tests written beforehand, by you, are the external signal of correctness that self-correction lacks - and it is exactly the condition that the work of Huang and colleagues indicates as necessary.

On hidden costs, three items that nobody measures (and that I name here without measuring them) and that are worth keeping in mind. The review time: reading and validating code you did not write can be slower than reading your own, and on a long block it can exceed the writing time. The debt from plausible code: an implementation that works on the cases tried and fails on an unverified edge case is worse than no implementation, because nobody will suspect it. And the erosion of knowledge: code that nobody has reasoned about is code that nobody will know how to change in six months : a cost that is paid all at once, at the worst moment.

Trade-offs

• Long prompt versus cost per call. Every token of the prompt is paid on every call, and in a conversation on every remaining turn. The line that eliminates a recurring error is worth it, not the paragraph that repeats a concept.
• Tight constraints versus cases you had not foreseen. A closed enum and a rigid format eliminate the wrong answers and also the correct ones you had not imagined. Mitigation: an "other" or "human review" value that gives the model a place to put the uncertainty.
• Expensive techniques versus value of the case. Self-consistency and reflection multiply the cost. They are applied to a subset - the cases where being wrong costs more than five calls - not to all the traffic.
• Speed versus understanding, on code. Generating a module in a few minutes and reviewing it in forty (hypothetical numbers) is not a gain if the module is simple; it is if the module is boring and long. The discriminating factor is how much it costs you to verify, not how much it costs you to write.

The four items read in pairs: the first two concern how much you spend on each call, the last two how much you spend to trust the result. None is solved with a better prompt; they are solved by choosing where to apply the technique and where not.

The traps people really fall into

1. Changing two things and measuring one. It is the most effective way to accumulate prompts full of useless instructions that nobody dares remove.
2. Having the model calculate a number. It produces plausible, unverifiable figures. The model writes the code, the code calculates, you verify.
3. Not generating the schema from the data. Without the real schema in the prompt, the model invents column names, and in the worst case guesses one that exists but is wrong.
4. Treating exec with reduced builtins as a sandbox. It is not. You need process isolation if the data or the machine matter.
5. Asking the same model for code and tests in the same turn. The tests will always pass. You write the contract, beforehand.
6. Chain of thought on tasks that have no steps. Pure cost, and sometimes it makes things worse: the model builds an elaborate justification for a wrong choice.
7. Reflection without an external criterion. Without a test, a compiler or a specification, self-correction can worsen the answer (arXiv:2310.01798).
8. Not providing the "I don't know" state. An enum with no way out forces the model to choose among wrong options, and choosing is what it will do.
9. Putting volatile data in the prompt. Today's date, a price list, a balance: they go stale, and the fault they produce is intermittent. They must be passed as data on every call.

The nine traps fall into three families: of method (1, 8, 9), of caution on data and code (2, 3, 4, 5) and of choice of technique (6, 7). The first are corrected with the discipline of the measurement protocol, the second with a check written in the code, the third with a question before adopting the technique: what does it give me, what does it cost, how do I measure it.

What you can do now

A prompt is a specification, and the levers that move the output in a measurable way are four: context, format, examples, constraints. The techniques with a paper behind them are useful when the mechanism matches the problem - chain of thought where there are steps, self-consistency where a measure of uncertainty is needed, reflection only with an external criterion - and they are cost multipliers to be justified with a measurement, never to be adopted out of habit. On data and code a single rule applies: the model writes code, the code produces the result, you verify the result. It is the point at which a probabilistic system becomes usable for things that have to be right.

You can write a prompt in pieces, one per lever, and tell how much each one weighs; you can set up a comparison between versions that separates format, correctness and variability; you can recognise when a technique with a paper behind it is worth its cost; you can have a model write code and surround it with checks, knowing the limits of those checks. What stays out is what I could not test here: how a real model behaves on these prompts. In the next article we go down a level: how text becomes tokens.

The same series also includes Model Context Protocol: the Specification, and a Server Wired to an Agent (previous article).