Fine-tuning an LLM for Italian: LoRA, Datasets, and Evaluation

The dataset decides the result, and its defects are boring

The most common format is JSONL: one JSON object per line, holding an instruction-answer pair. The real work lies in the ordinary defects of a hand-collected file: lines that are not valid JSON, empty answers, instructions too short to say anything, duplicates that differ by one space, and (the most insidious) the same question with two answers, which ends up half in training and half in validation.

The program below works on an in-memory file of twenty lines with all these defects. The text of the examples is Italian (it is the data of the experiment: a question such as "Spiega che cos'è una caparra confirmatoria" means "Explain what a confirmatory deposit is"), while comments, labels and printed messages are in English. Two choices deserve an explanation. The first is the SHA-256 fingerprint in place of Python's hash() function: for strings the latter changes at every run (the seed is random), so a split built with hash() is not reproducible. The second is Unicode normalization with casefold: two strings that look the same can have the "è" written in two ways, and without normalizing the duplicate slips through.

import hashlib, json, unicodedata

# An in-memory JSONL "file": 8 rows with the typical defects of a hand-collected dataset.
GREZZO = """\
{"instruction": "Spiega che cos'è una caparra confirmatoria", "output": "È una somma versata alla stipula del preliminare che..."}
{"instruction": "Spiega che cos'è una caparra confirmatoria ", "output": "È una somma versata alla stipula del preliminare che..."}
{"instruction": "Cos'è l'IVA?", "output": "L'imposta sul valore aggiunto è..."}
{"instruction": "Riassumi il contratto in tre punti", "output": ""}
{"instruction": "Elenca i tre gradi di giudizio", "output": "Primo grado, appello, Cassazione."}
{"instruction": "Elenca i tre gradi di giudizio", "output": "Primo grado, appello e Cassazione."}
{"instruction": "Cos'è la prescrizione?" "output": "riga con una virgola mancante"}
{"instruction": "Che cosa significa recesso?", "output": "È la facoltà di sciogliersi da un contratto senza..."}
"""

# Six more questions, each with two different answers: in a real dataset this happens often
# (the same question collected from two people). The answers are placeholders.
DOMANDE = ["Come si calcola la scadenza di un termine", "Quando decade un diritto di garanzia",
           "Che cos'è un decreto ingiuntivo", "Chi risponde dei vizi della cosa venduta",
           "Come si impugna una sanzione amministrativa", "Che differenza c'è fra dolo e colpa"]
for d in DOMANDE:
    for variante in ("prima risposta", "seconda risposta"):
        GREZZO += json.dumps({"instruction": d, "output": f"{variante} a: {d.lower()}"}, ensure_ascii=False) + "\n"

def normalizza(s):
    # NFC: "è" written as e + combining accent becomes the same character; spaces and case do not count
    return " ".join(unicodedata.normalize("NFC", s).split()).casefold()

def impronta(record):
    chiave = normalizza(record["instruction"]) + "\x1f" + normalizza(record["output"])
    return hashlib.sha256(chiave.encode("utf-8")).hexdigest()

def pulisci(righe, parole_minime=3):
    visti, tenuti, scartati = set(), [], {"json": 0, "empty": 0, "short": 0, "duplicate": 0}
    for riga in righe:
        try:
            r = json.loads(riga)
            istruzione, output = r["instruction"].strip(), r["output"].strip()
        except (json.JSONDecodeError, KeyError):
            scartati["json"] += 1; continue
        if not output:
            scartati["empty"] += 1; continue
        if len(istruzione.split()) < parole_minime:
            scartati["short"] += 1; continue
        h = impronta(r)
        if h in visti:
            scartati["duplicate"] += 1; continue
        visti.add(h); tenuti.append((h, r))
    return tenuti, scartati

def quota(testo):
    h = hashlib.sha256(testo.encode("utf-8")).hexdigest()
    return int(h[:8], 16) / 0xFFFFFFFF          # a number in [0, 1], always the same for the same text

def dividi(tenuti, chiave, quota_validazione=0.5):
    """Deterministic assignment: it depends on the chosen key, not on the order nor on chance."""
    train, valid = [], []
    for h, r in tenuti:
        (valid if quota(chiave(h, r)) < quota_validazione else train).append(r)
    return train, valid

def istruzioni_in_comune(train, valid):
    return {normalizza(r["instruction"]) for r in train} & {normalizza(r["instruction"]) for r in valid}

tenuti, scartati = pulisci(GREZZO.splitlines())
print("input rows:", len(GREZZO.splitlines()))
print("discarded:", scartati)
print("kept:", len(tenuti))
for nome, chiave in [("by record", lambda h, r: h), ("by instruction", lambda h, r: normalizza(r["instruction"]))]:
    train, valid = dividi(tenuti, chiave)
    print(f"split {nome:15} train={len(train)} validation={len(valid)} shared instructions: {sorted(istruzioni_in_comune(train, valid))}")

# Output:
# input rows: 20
# discarded: {'json': 1, 'empty': 1, 'short': 1, 'duplicate': 1}
# kept: 16
# split by record       train=7 validation=9 shared instructions: ["che cos'è un decreto ingiuntivo", "che differenza c'è fra dolo e colpa", 'chi risponde dei vizi della cosa venduta', 'come si calcola la scadenza di un termine', 'come si impugna una sanzione amministrativa', 'quando decade un diritto di garanzia']
# split by instruction  train=14 validation=2 shared instructions: []

The output shows the four discards, one per type, and above all the difference between the two splits. Splitting by record, all six double-answer questions appear in both training and validation: the validation loss comes out optimistic because the model has already seen the question. Splitting by instruction, the two lists have no questions in common and validation really measures generalization. With a dataset of thousands of rows the same error is invisible to the eye, which is why you should assign every group of identical questions to one side only. The length filter (three words) is an arbitrary threshold to tune on your own data.

LoRA: training two thin matrices instead of one enormous matrix

The idea of the original paper is this: freeze the model's weights and, for each chosen matrix, add a correction built as the product of two thin matrices, A and B, with a small "rank" r. Instead of updating a matrix of 4096 by 4096 numbers, you train only 4096 times r plus r times 4096. The correction is multiplied by a scale factor, lora_alpha / r in the standard implementation of the PEFT library, and B starts at zero so that at the beginning the adapted model is exactly the starting model.

The following program does three things with NumPy. It computes how many parameters you train on Mistral-7B-v0.1 using the values of its config.json (4096 dimensions, 32 layers, 8 heads for keys and values); it checks that with B equal to zero the output does not change; and it trains by hand a rank-4 correction to see what happens as r varies.

import numpy as np

# --- 1. How many parameters you train: counts on the Mistral-7B-v0.1 configuration ---------
HIDDEN, KV_HEADS, HEADS, LAYERS = 4096, 8, 32, 32          # config.json of the model
KV_DIM = HIDDEN // HEADS * KV_HEADS                        # 128 x 8 = 1024: v_proj has 1024 outputs
MODULI = {"q_proj": (HIDDEN, HIDDEN), "v_proj": (HIDDEN, KV_DIM)}   # (input, output)

def parametri_lora(r, moduli=MODULI, strati=LAYERS):
    # for each matrix W (input x output) LoRA trains A (input x r) and B (r x output)
    return strati * sum(r * (i + o) for i, o in moduli.values())

for r in (8, 16, 64):
    n = parametri_lora(r)
    print(f"r={r:<3} trainable parameters = {n:>11,}  = {n / 7_241_732_096:.3%} of the model's 7.24 billion")

# --- 2. Why B starts at zero -------------------------------------------------------------------
rng = np.random.default_rng(0)
d_in, d_out, r, alpha = 64, 48, 4, 8
W = rng.normal(size=(d_in, d_out))
A = rng.normal(scale=0.1, size=(d_in, r))
B = np.zeros((r, d_out))
x = rng.normal(size=(5, d_in))
y0, y1 = x @ W, x @ W + (alpha / r) * (x @ A) @ B
print("at the start the adapted model is identical to the base:", np.allclose(y0, y1))

# --- 3. A small training run: how much a low rank is enough ------------------------------------
# The "change to learn" has true rank 4 plus a little noise.
delta = rng.normal(size=(d_in, 4)) @ rng.normal(size=(4, d_out)) + 0.05 * rng.normal(size=(d_in, d_out))
X = rng.normal(size=(512, d_in))
Y = X @ delta                                           # target: reproduce the effect of delta

def allena(rango, passi=3000, lr=0.003):
    A = rng.normal(scale=0.1, size=(d_in, rango)); B = np.zeros((rango, d_out))
    s = 1.0                         # alpha = rank: I keep the scale fixed, so the comparison is only about r
    for _ in range(passi):
        errore = X @ A @ B * s - Y
        gB = s * (X @ A).T @ errore / len(X)
        gA = s * X.T @ (errore @ B.T) / len(X)
        A -= lr * gA; B -= lr * gB
    residuo = np.linalg.norm(A @ B * s - delta) / np.linalg.norm(delta)
    return residuo, A.size + B.size

print(f"full matrix: {delta.size} parameters")
for rango in (1, 2, 4, 16):
    res, n = allena(rango)
    print(f"r={rango:<3} parameters={n:>5}  relative error on delta = {res:.3f}")

# Output:
# r=8   trainable parameters =   3,407,872  = 0.047% of the model's 7.24 billion
# r=16  trainable parameters =   6,815,744  = 0.094% of the model's 7.24 billion
# r=64  trainable parameters =  27,262,976  = 0.376% of the model's 7.24 billion
# at the start the adapted model is identical to the base: True
# full matrix: 3072 parameters
# r=1   parameters=  112  relative error on delta = 0.795
# r=2   parameters=  224  relative error on delta = 0.604
# r=4   parameters=  448  relative error on delta = 0.025
# r=16  parameters= 1792  relative error on delta = 0.017

The first three numbers correct a widespread idea: with q_proj and v_proj and r=16 you train 6,815,744 parameters, that is 0.094% of the model and not the "1-2%". For r=64 that is 27 million, about 0.38%. The MODULI dictionary in the program also shows why v_proj costs less than q_proj: the model uses 8 heads for keys and values against 32 for queries, so the matrix has 1024 outputs and not 4096. Then the experiment on the small problem: the error on the true correction stays high for r=1 and r=2 (0.795 and 0.604) and collapses to 0.025 when r reaches 4, which is the rank with which I built the target. Going up to 16 improves little (0.017). It is a toy model and proves nothing about languages, but it explains the practical rule: if r is below the required rank the task is not learned, if it is above you pay memory for no gain.

QLoRA: the base model in 4 bit, the adapters in high precision

QLoRA adds one step: the frozen base model is kept in 4 bit, and the gradients flow through those quantized weights to update the LoRA adapters, which stay in higher precision. The paper (Dettmers et al.) declares three ingredients: the 4-bit NormalFloat (NF4) data type, double quantization (which also quantizes the quantization constants) and "paged" optimizers to absorb memory spikes. The result declared by the authors is the fine-tuning of a 65-billion-parameter model on a single 48 GB GPU, without losing quality compared with 16 bit.

For a 7-billion model the 4-bit weights alone take about 3.6 GB (7.24 billion parameters at half a byte): the rest of the memory goes to activations, optimizer states and cache, and depends on sequence length and batch size. I do not give you a figure for a specific card because I did not measure it: try it on yours with a minimal batch and increase it.

The training script, line by line

The following script is a schema to adapt, not executed: on this machine there is neither PyTorch nor a GPU. It corrects three common errors in tutorials. The first: tokenizer.pad_token = tokenizer.eos_token looks harmless, but the Transformers data collator sets to -100 (ignored by the loss) every label equal to the padding token's identifier, so the model never sees the end of the answer and learns not to stop; here I use the unk token, which Mistral has. The second: tokenization does not pad up to the maximum length, the collator does it per batch. The third: validation is a separate file, the one produced by the split by instruction. The prompt template is written in English here; with an Italian dataset you would keep the headings that match your data.

import torch
from datasets import load_dataset
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import (AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig,
                          DataCollatorForLanguageModeling, Trainer, TrainingArguments)

BASE = "mistralai/Mistral-7B-v0.1"
dati = load_dataset("json", data_files={"train": "train.jsonl", "validation": "valid.jsonl"})

tok = AutoTokenizer.from_pretrained(BASE)
tok.pad_token = tok.unk_token                     # NOT eos: the collator zeroes out the labels of the padding

def prepara(esempio):
    testo = f"### Instruction:\n{esempio['instruction']}\n\n### Answer:\n{esempio['output']}{tok.eos_token}"
    return tok(testo, truncation=True, max_length=1024)

dati = dati.map(prepara, remove_columns=["instruction", "output"])

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
modello = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto")
modello = prepare_model_for_kbit_training(modello)
modello = get_peft_model(modello, LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                         target_modules=["q_proj", "v_proj"], bias="none", task_type="CAUSAL_LM"))
modello.print_trainable_parameters()              # expected: 6,815,744 with r=16 on q_proj and v_proj

args = TrainingArguments(output_dir="lora-it", num_train_epochs=3, learning_rate=2e-4,
                         per_device_train_batch_size=1, gradient_accumulation_steps=8,
                         gradient_checkpointing=True, optim="paged_adamw_8bit", bf16=True,
                         eval_strategy="epoch", save_strategy="epoch", logging_steps=10,
                         warmup_steps=20, report_to="none")
Trainer(model=modello, args=args, train_dataset=dati["train"], eval_dataset=dati["validation"],
        data_collator=DataCollatorForLanguageModeling(tok, mlm=False)).train()
modello.save_pretrained("lora-it-finale")         # saves only the adapters, not the base model

The lines that matter: the prompt string ends with tok.eos_token, so the end of the answer enters the labels; prepare_model_for_kbit_training prepares the quantized model to receive gradients; print_trainable_parameters is the consistency check against the previous program's calculation (the comment gives the number you should see, after checking your own model's architecture). Gradient accumulation simulates a batch of 8 with only one row at a time in memory. I did not keep max_steps, because together with num_train_epochs they contradict each other: if you set the steps, the number of epochs is ignored. bf16=True requires a GPU that supports bfloat16 (for NVIDIA, from the Ampere generation); on others use fp16=True.

The hyperparameters you will have to change

Each item in the list gives the parameter, the script's value and what happens if you change it. The starting values are starting choices, not results: the defaults of the PEFT library, read in its source code on 2026-11-12, are r=8, lora_alpha=8 and lora_dropout=0.0, so the script above deliberately replaces them.

• r=16: capacity of the adapter. Too low and the task is not learned (see the experiment on rank); too high wastes memory and raises the risk of fitting the noise.
• lora_alpha=32: with the standard scale lora_alpha / r equals 2. If you double r without touching alpha, the scale halves and the correction weighs less.
• target_modules=["q_proj", "v_proj"]: the matrices you attach LoRA to. Adding others (the key and output projections and the feed-forward block) increases the parameters and usually the capacity.
• learning_rate=2e-4: a typical value for adapters, much higher than that of full fine-tuning. If the validation loss rises while the training loss falls, it is too high or the epochs are too many.
• gradient_accumulation_steps=8: effective batch = per-device batch × accumulation. Change it if you run out of memory.

The list has no order of importance, but there is a useful sequence when something goes wrong: first you check the dataset, then the learning rate, then the rank. The most common symptoms (repetitive output, answers that do not end, a loss that does not fall) more often originate in the data or in the end token than in the rank, which is why they are checked first.

After training: merging the adapters and converting to GGUF

The saved adapters weigh little but are not a model on their own: to use them with a local inference engine based on llama.cpp you merge them into the base model in full precision and convert the result to the GGUF format. The commands below are schemas to adapt, not executed here. The first block merges the adapters, the second converts and quantizes; the name of the script and that of the executable are the current ones in the llama.cpp repository, read on 2026-11-12 (older instructions mention convert-hf-to-gguf.py and ./quantize, which are no longer the right names).

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", torch_dtype=torch.bfloat16)
unito = PeftModel.from_pretrained(base, "lora-it-finale").merge_and_unload()
unito.save_pretrained("mistral-it-unito", safe_serialization=True)
AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1").save_pretrained("mistral-it-unito")

python convert_hf_to_gguf.py mistral-it-unito --outfile mistral-it-f16.gguf --outtype f16
./build/bin/llama-quantize mistral-it-f16.gguf mistral-it-Q4_K_M.gguf Q4_K_M

Two things to know. Do the merge on the model in full precision (here bfloat16) and not on the 4-bit one, otherwise you add the correction to weights that are already rounded. And quantizing after the merge removes precision a second time: compare the GGUF's answers with those of the merged model on the same test examples, not only through the validation loss.

Evaluating without fooling yourself

The validation loss tells you whether the model is learning; it does not tell you whether you like the answers. The program below shows two calculations that are easy to get wrong. The first is perplexity: it is the exponential of the mean loss per token, so it is weighted on the number of tokens, not on documents. With three documents of 12, 15 and 400 tokens the mean over tokens gives 5.34, while the mean of the individual perplexities would give 12.67 because the two short documents, which are harder, weigh as much as the long one. Besides, two perplexities can be compared only if they come from the same tokenizer.

import math

# --- 1. Perplexity: the average is taken over TOKENS, not over documents ----------------------
# For each document: (number of tokens, mean loss per token in nats). Example numbers, invented.
DOCUMENTI = [(12, 2.9), (15, 2.7), (400, 1.6)]

perdita_totale = sum(n * loss for n, loss in DOCUMENTI)
token_totali = sum(n for n, _ in DOCUMENTI)
corretta = math.exp(perdita_totale / token_totali)
errata = sum(math.exp(loss) for _, loss in DOCUMENTI) / len(DOCUMENTI)
print(f"perplexity over tokens   : {corretta:.2f}")
print(f"mean of the perplexities : {errata:.2f}   (wrong: the 12-token document weighs as much as the 400-token one)")

# --- 2. Pairwise comparison between base and adapted: how much is 30 against 12? -------------
# 50 cases judged by a person: 30 better with the adapted model, 12 worse, 8 ties. Invented example.
vittorie, sconfitte, pari = 30, 12, 8

def p_bilaterale(k, n):
    """Exact sign test: probability of a gap at least this large if the two models were equal."""
    estremo = max(k, n - k)
    coda = sum(math.comb(n, i) for i in range(estremo, n + 1)) / 2 ** n
    return min(1.0, 2 * coda)

n = vittorie + sconfitte                                  # ties do not count
print(f"{vittorie} vs {sconfitte} out of {n} non-tied cases: p = {p_bilaterale(vittorie, n):.4f}")
print(f"18 vs 12 out of 30 non-tied cases: p = {p_bilaterale(18, 30):.4f}")

# Output:
# perplexity over tokens   : 5.34
# mean of the perplexities : 12.67   (wrong: the 12-token document weighs as much as the 400-token one)
# 30 vs 12 out of 42 non-tied cases: p = 0.0079
# 18 vs 12 out of 30 non-tied cases: p = 0.3616

The second calculation is for the pairwise judgment: a person looks at fifty answers from the base and the adapted model and says which one they prefer. The numbers are an invented example to show the method. With 30 preferences for the adapted model against 12 for the base (eight ties, which do not count) the exact sign test gives p = 0.0079: the difference is unlikely if the two models were equivalent. With 18 against 12 over thirty cases p rises to 0.36, and you cannot conclude anything. Before saying "the adapted model is better" you need more cases or a clearer advantage. The test is the same one you will use with your real answers; replace the three variables at the top.

Things that go wrong

• Out of memory: reduce the maximum length, lower the batch and raise the accumulation, keep gradient checkpointing on.
• Answers that do not stop: check that the end token is in the training text and that the padding does not coincide with it.
• Repetitive output: data with many nearly identical answers, or too many epochs on few examples.
• Validation loss that rises: overfitting; reduce epochs or learning rate, or add varied data.

The four items are not equivalent: the second and the third are defects of the data, the other two of the parameters. If you change one parameter at a time and always compare on the same test examples, you understand which of the causes you are facing; if you change three together, you do not.

Frequently asked questions

Does fine-tuning teach the model my documents?

It is not the right tool: a model adapted on a collection of texts imitates their style and may repeat some facts, but it remembers them unreliably and does not update when the documents change. For facts, use document retrieval at question time; for the style and format of answers, fine-tuning.

How many examples are needed?

There is no threshold valid for everyone. Start from a few hundred curated examples, measure with the pairwise comparison of the program above and add data where the model gets it wrong. A few consistent, curated examples are worth more than many collected without criteria.

Can I keep several adapters for different tasks?

Yes: adapters are separate, light files, so you can train one per task and load the one you need onto the base model. For the GGUF engine, instead, you must merge and convert each one, and you get a complete model per task.

What is the right starting model for Italian?

It depends on license, size and context: the choice is discussed, with the official model cards, in The Italian LLM Landscape in 2025: Models, Ecosystem, and Practical Choices. Before training, also check that the license allows you to redistribute a derivative.

What you can do now

You can prepare a JSONL dataset without the most common defects and split it without leaking questions, compute how many parameters you train for a given rank, write a QLoRA script with the end token in the right place, merge the adapters and convert to GGUF with the current commands, and compare two models with a test that tells you when the difference is not enough. The first thing to do is to collect fifty examples of the answers you would like to get: they serve both for training and for judging. If you need the base under the model, tokenization and lemmas, go to Italian NLP with spaCy and Transformers: A Practical Pipeline.