10 - Python for Data 3: Functions, Files, Exceptions, Objects

A text file does not contain text: it contains bytes, and text is born only when you choose an encoding to read them. This is why è becomes è when you open a CSV the wrong way. This lesson of the Machine Learning Engineer course ("Python for data" module) takes you from scattered lines to organised code: small, testable functions, files read and written without corrupting the accents, errors handled instead of ignored, and classes used only where they are needed. You need Python 3 and the standard library; nothing to install.

Functions

What it is, why it exists

A function is a procedure with a name, inputs and an output. Analogy: a recipe with a title: you do not rewrite it every time, you call it by name with the day's ingredients.

Why it exists: to remove repetition, to give a name to an idea, to be able to test an isolated piece. A function that does one thing only and has an honest name is the first form of documentation.

The method: for every function

problem → signature → input → output → implementation → explanation → edge case → test.

def media(valori, *, ignora_none=False):
    """Arithmetic mean of a sequence of numbers.

    If ignora_none is True, None values are dropped. Raises ValueError if nothing is left.
    """
    if ignora_none:
        valori = [v for v in valori if v is not None]
    if not valori:
        raise ValueError("at least one value is required")
    return sum(valori) / len(valori)

print(media([1, 2, 3]), media([1, None, 3], ignora_none=True))
try:
    media([])
except ValueError as e:
    print("ValueError:", e)

Output obtained (two identical runs): 2.0 2.0 and ValueError: at least one value is required.

• Signature. The * before ignora_none makes the argument keyword-only: you write media(x, ignora_none=True), never media(x, True). A boolean passed by position is unreadable.
• Edge case. Empty list: the mean is not defined (0/0), so an explicit error. Returning 0 would make a meaningless result look sensible.
• Docstring (PEP 257): it describes what the function does, not how.
• Test: the code itself shows two calls and an expected error. In a real project they become assert statements or automated tests.

The four entries are the method applied to media: the signature decides how the function is called, the edge case decides what it does when the data is wrong, the docstring declares it, the test proves it. The one that weighs most is the edge case: a mean that returns 0 on an empty list raises no error and produces plausible, false numbers, which is the worst defect in a data analysis.

Functions that produce functions: closures and decorators

In Python a function is an object like any other: it can be passed to another function, returned by a function, assigned to a name. This seemingly abstract idea has an everyday practical use, the decorator: a function that receives a function and returns an enriched version of it (one that counts the calls, measures their time, logs their arguments) without touching its code. The analogy is the frame you put around a painting: the painting (the original function) stays intact, the frame adds something around it.

import functools

def conta_chiamate(f):
    @functools.wraps(f)                       # copies name and documentation of the original function
    def interno(*args, **kwargs):
        interno.chiamate += 1                 # the function "remembers" the counter: it is a closure
        return f(*args, **kwargs)             # delegates to the original function, unchanged
    interno.chiamate = 0
    return interno

@conta_chiamate
def quadrato(x):
    """Return x squared."""
    return x * x

print([quadrato(n) for n in range(5)])
print("calls:", quadrato.chiamate, "| name:", quadrato.__name__, "| doc:", quadrato.__doc__)

Output obtained:

[0, 1, 4, 9, 16]
calls: 5 | name: quadrato | doc: Return x squared.

How to read the code. conta_chiamate receives f and defines a function interno that, on every call, increments a counter and then really calls f with the same arguments (*args and **kwargs forward any combination of arguments). interno remembers f even after conta_chiamate has finished: a function that carries along the names of the environment where it was defined is called a closure. The line @conta_chiamate above def quadrato is a shortcut for quadrato = conta_chiamate(quadrato). functools.wraps matters: without it quadrato.__name__ would become "interno" and the documentation would be lost, making debugging confusing. What you need to remember is the model: function inside function = a function built at runtime with a memory of its own. You will find it again in libraries (for example for caching, functools.lru_cache) and in the code of scikit-learn and Pandas.

Variable arguments, and scope

def somma_tutti(*args, **kwargs):
    return sum(args), kwargs

print(somma_tutti(1, 2, 3, a=1))      # (6, {'a': 1})

conta = 0

def incrementa():
    conta += 1        # conta is treated as local because you assign it here

try:
    incrementa()
except UnboundLocalError as e:
    print("UnboundLocalError:", e)
# Output: UnboundLocalError: cannot access local variable 'conta' where it is not associated with a value

*args collects the extra positional arguments in a tuple, **kwargs the keyword ones in a dictionary. The exact message is the one from Python 3.14.4 and changes between versions. Scope follows the LEGB rule (Local, Enclosing, Global, Built-in). If you assign to a name inside a function, Python treats it as local throughout the function, so reading it before the assignment gives UnboundLocalError. The remedy is not global: it is to pass the value as an argument and return the new one.

Why it matters for data: pure functions (same inputs → same output, no side effects) are easy to test and to run in parallel; they are the mental model of NumPy and Pandas.

Type hints (PEP 484). def media(valori: list[float]) -> float: does not change execution but documents and lets analysis tools find errors. The course uses them where they clarify.

Text, bytes and encoding

The analogy: the coded letter

A file is a strip of numbers from 0 to 255 (bytes). "Text" is what you get when you apply an encoding: a table that says which character corresponds to which bytes. If you read with the wrong table you get meaningless characters, like deciphering a message with the wrong code.

UTF-8

Unicode assigns a number (a code point) to every character. UTF-8 (RFC 3629) is a way of writing those numbers as bytes, with variable length: ASCII characters use 1 byte, è uses 2, the € symbol uses 3. It is the default encoding of the web and of modern systems.

s = "è€"
b = s.encode("utf-8")
print(len(s), len(b), b)          # 2 characters, 5 bytes
print(b.decode("utf-8"))          # è€
print(b.decode("latin-1"))        # mojibake: the bytes are read with the wrong table

Output: 2 5 b'\xc3\xa8\xe2\x82\xac', then è€, then a meaningless sequence that begins with èâ (this is the mojibake: the byte pair 0xC3 0xA8 of è read as two Latin-1 characters, Ã and ¨). Decoding those same bytes as ascii raises UnicodeDecodeError, because ASCII only knows values up to 127.

Golden rule: always specify encoding="utf-8" when you open text files. The default depends on the operating system and on the configuration, so the same code can work on your computer and break on another.

Data trap: a CSV exported from an Italian spreadsheet can be in cp1252 (Windows) or in UTF-8 with a BOM. If you see è in your columns, the problem is the encoding used to read, not the data.

An experimental check: three files, three readings

The theory is clear; the discomfort comes when you really hold files of different origin. Let us try every combination of three writing encodings and three reading ones: UTF-8, cp1252 (the historical default encoding of Windows for Western languages) and utf-8-sig (UTF-8 with a three-byte signature at the start, the BOM, which some spreadsheet programs add).

import tempfile
from pathlib import Path

cartella = Path(tempfile.mkdtemp())
testo = "café,quantity\nespresso,3\n"
codifiche = ("utf-8", "cp1252", "utf-8-sig")

for scrittura in codifiche:
    percorso = cartella / f"{scrittura}.csv"
    percorso.write_text(testo, encoding=scrittura)
    print(f"file written as {scrittura:<9}: first bytes = {percorso.read_bytes()[:6]!r}")
    for lettura in codifiche:
        try:
            prima_riga = percorso.read_text(encoding=lettura).splitlines()[0]
            esito = repr(prima_riga)
        except UnicodeDecodeError as errore:
            esito = f"DECODING ERROR ({errore.reason})"
        print(f"    read as {lettura:<9}: {esito}")

Output obtained:

file written as utf-8    : first bytes = b'caf\xc3\xa9,'
    read as utf-8    : 'café,quantity'
    read as cp1252   : 'café,quantity'
    read as utf-8-sig: 'café,quantity'
file written as cp1252   : first bytes = b'caf\xe9,q'
    read as utf-8    : DECODING ERROR (invalid continuation byte)
    read as cp1252   : 'café,quantity'
    read as utf-8-sig: DECODING ERROR (invalid continuation byte)
file written as utf-8-sig: first bytes = b'\xef\xbb\xbfcaf'
    read as utf-8    : '\ufeffcafé,quantity'
    read as cp1252   : 'café,quantity'
    read as utf-8-sig: 'café,quantity'

How to read the code. For each of the three encodings the program writes the same text (with the accented letter é in the first column name), shows the first bytes of the file, then rereads it with each of the three encodings and prints the first line with repr (which makes even invisible characters visible). Three things show up. One: UTF-8 read as cp1252 produces the mojibake: Ã followed by another symbol in place of the accented letter, without any error, because cp1252 accepts any sequence of bytes. Two: cp1252 read as UTF-8 instead fails with a decoding error, because the byte of é in cp1252 (0xE9) is not the start of a valid UTF-8 sequence (in UTF-8 that value would announce two continuation bytes, which are not there here).

Three: a file with the BOM signature, read as plain utf-8, keeps an invisible character () at the start of the first column name: "café" is not equal to "café", and the infamous KeyError on a column that "is there" has this origin. You cure it by reading with utf-8-sig, which discards the signature if present. The operational lesson: when characters are wrong or a column "disappears", look at the bytes (read_bytes()[:20]) before blaming the data.

Files: pathlib, json, csv

from pathlib import Path
import csv
import json
import tempfile

cartella = Path(tempfile.mkdtemp())           # temporary folder for the example

# JSON: nested structures (dictionaries, lists, numbers, strings)
f = cartella / "dati.json"
f.write_text(json.dumps({"nome": "Zoë", "voti": [30, 28]}, ensure_ascii=False), encoding="utf-8")
print(json.loads(f.read_text(encoding="utf-8")))

# CSV: tables. Careful: every value read is a STRING.
c = cartella / "t.csv"
with c.open("w", newline="", encoding="utf-8") as fh:
    w = csv.writer(fh)
    w.writerow(["id", "importo"])
    w.writerow([1, "10.5"])
with c.open(newline="", encoding="utf-8") as fh:
    righe = list(csv.DictReader(fh))
print(righe, type(righe[0]["importo"]))

Output: {'nome': 'Zoë', 'voti': [30, 28]} and [{'id': '1', 'importo': '10.5'}] <class 'str'>.

• pathlib.Path replaces strings for paths: cartella / "file" composes with the right separator for the operating system.
• with opens the file and always closes it, even if there is an error in the block. Forgetting to close a file opened for writing can leave unsaved data.
• ensure_ascii=False in json.dumps writes ë as a character, not as the escape sequence ë; more readable and more compact.
• newline="" for the csv module is required by the documentation to handle line breaks correctly.
• csv returns strings. "10.5" is not a number until you convert it. In the article on Pandas read_csv infers the types for you, but knowing the underlying behaviour explains the errors.

The five points are five causes of silent bugs: paths that break between operating systems, files left open, accents written in an unreadable way, doubled blank lines, numbers that are strings. The third and the fifth are the most insidious, because the program does not stop and produces a result that looks right.

When JSON, when CSV. JSON: nested data, configurations, service responses. CSV: flat tables, interoperable with spreadsheets. For large tables used many times there are columnar formats such as Parquet, which we do not cover in this lesson.

The danger of pickle

pickle serialises arbitrary Python objects into bytes. It is convenient (you save a trained model with one line). The official documentation warns that the module is not secure: loading pickle data from an untrusted source can execute arbitrary code, because deserialisation can invoke functions chosen by whoever created the file. Rule: never run pickle.load on files you did not produce yourself or do not fully trust. For models there are safer and more portable alternatives, which depend on the library used (the pickle documentation lists the alternatives for data).

Exceptions

The analogy: plan B

A cook knows an egg may be rotten. They do not bake the cake anyway: they have a plan B for that specific case. Exceptions are the mechanism for declaring "if this specific problem happens, do this other thing".

def leggi_importi(testi):
    """Convert strings to float; return (valid, discarded)."""
    ok, scartati = [], []
    for t in testi:
        try:
            ok.append(float(t))
        except ValueError:                 # catch ONLY the expected error
            scartati.append(t)
    return ok, scartati

print(leggi_importi(["1.5", "abc", "2", ""]))

Output: ([1.5, 2.0], ['abc', '']). The discarded ones are kept, not thrown away: you can report how many rows were malformed.

Complete structure:

try:
    int("x")
except ValueError:
    print("a")        # runs if there is the error
else:
    print("else")     # runs ONLY if there was no error
finally:
    print("finally")  # runs ALWAYS (cleanup)

Output: a and finally.

Rules.

1. Catch specific exceptions (ValueError), never a generic except: or except Exception: unless you know why: it hides bugs (even a typo in the code).
2. Never silence: except: pass is how problems become mysteries. Log or re-raise.
3. Fail fast: validate the inputs at the start and raise ValueError with a message that includes the offending value.
4. Exceptions for exceptional cases, not for normal flow control when an if is clearer.

The four rules say the same thing from different angles: an error must be seen by whoever can fix it. The first two concern what you catch and what you do with it; the last two say where to put the check, at the entry of the data and not at the end of the chain. If you apply only one, the second: an except: pass turns a fault into a mystery.

For data: one error in 3 million rows must not stop the processing, but it must not vanish either. Counting and reporting the discarded ones is a form of data quality.

Errors that say where and why: custom exceptions

A generic exception such as ValueError is fine for a local check. When a processing run crosses several layers (reading the file, validation, computation) a domain-specific error is better, one that carries its context: which row, which field, which value. Python also lets you link the original cause with raise ... from ..., so that the traceback shows both.

class ErroreDati(Exception):
    """Error in the input data, with the row number."""

def leggi_eta(riga, numero):
    try:
        eta = int(riga["eta"])
    except (ValueError, KeyError) as causa:
        raise ErroreDati(f"row {numero}: invalid 'eta' field ({riga.get('eta')!r})") from causa
    if not 0 <= eta <= 120:
        raise ErroreDati(f"row {numero}: age out of range ({eta})")
    return eta

righe = [{"eta": "34"}, {"eta": "abc"}, {"eta": "230"}, {}]
for n, r in enumerate(righe, start=1):
    try:
        print(f"row {n}: age = {leggi_eta(r, n)}")
    except ErroreDati as errore:
        causa = type(errore.__cause__).__name__ if errore.__cause__ else "none"
        print(f"{errore} | cause: {causa}")

Output obtained:

row 1: age = 34
row 2: invalid 'eta' field ('abc') | cause: ValueError
row 3: age out of range (230) | cause: none
row 4: invalid 'eta' field (None) | cause: KeyError

How to read the code. ErroreDati is a class that inherits from Exception: that is enough to create a new type of error that can be caught on its own. leggi_eta converts the field and, if the conversion fails (non-numeric text or missing key), raises ErroreDati from the original ValueError or KeyError (from causa), with a message containing the row number and the value. The second check, on the range, raises the error directly. The calling loop catches only ErroreDati and prints the message and the type of the cause. The advantage is that the error tells a complete story ("row 2: invalid 'eta' field 'abc'"), and the caller can decide to discard the row, log it or stop, without having to understand internal details.

Objects: when a class helps

Analogy: a class is the cookie cutter; the objects are the cookies. The cutter defines the shape (attributes) and what you can do with it (methods).

When to use a class. When you have data and behaviour that belong together and you want to guarantee consistency (a Punto with x, y and its distanza). When NOT to: for a function that has a single input and an output, to hold stateless functions (a module is enough), to "organise" in advance. A good rule: start from functions; introduce a class when you notice you are passing the same three arguments to five functions.

dataclass

from dataclasses import dataclass

@dataclass(frozen=True)
class Punto:
    x: float
    y: float

    def distanza(self, altro: "Punto") -> float:
        return ((self.x - altro.x) ** 2 + (self.y - altro.y) ** 2) ** 0.5

p, q = Punto(0, 0), Punto(3, 4)
print(p, p.distanza(q), p == Punto(0, 0))
try:
    p.x = 1
except Exception as e:
    print(type(e).__name__)

Output: Punto(x=0, y=0) 5.0 True and FrozenInstanceError.

@dataclass generates __init__, __repr__ and __eq__ from the annotated attributes. With frozen=True the object is immutable (and therefore hashable): you cannot change p.x. It is your default choice for data. The distance is the Pythagorean theorem: √(Δx² + Δy²), here √(9 + 16) = 5.

Special methods (dunder)

Methods with double underscores let your objects behave like the built-in types: __repr__ (how it prints), __add__ (the + operator), __len__ (the len function), __eq__ and __hash__ (equality and use as a key).

class Vettore:
    def __init__(self, *componenti):
        self.c = tuple(componenti)

    def __repr__(self):
        return f"Vettore{self.c}"

    def __add__(self, altro):
        return Vettore(*(a + b for a, b in zip(self.c, altro.c, strict=True)))

    def __len__(self):
        return len(self.c)

    def __eq__(self, altro):
        return self.c == altro.c

    def __hash__(self):
        return hash(self.c)

print(Vettore(1, 2) + Vettore(3, 4), len(Vettore(1, 2, 3)), Vettore(1) == Vettore(1))

Output: Vettore(4, 6) 3 True. Notes: __add__ returns a new object (immutability: it does not modify the operands); strict=True makes the sum of vectors of different lengths fail instead of silently truncating; if you define __eq__ you must also define a consistent __hash__, otherwise the object cannot be used as a key. NumPy and Pandas are built on this mechanism: this is why a + b works on arrays.

"Private" attributes. Python has no true private attributes. A name with a single underscore (_x) is a warning by convention; with two (__x) it undergoes name mangling: internally it becomes _Classe__x, to avoid collisions in subclasses, not to hide. It is not security.

The annotated solution of the mini-project: a robust loader

Let us put together immutable dataclasses, csv, exceptions and error counting. The program first writes a small test CSV (with accents and with deliberately wrong rows), then reads it.

import csv
import tempfile
from collections import defaultdict
from dataclasses import dataclass
from datetime import date
from pathlib import Path

@dataclass(frozen=True)
class Vendita:
    data: date
    prodotto: str
    quantita: int
    prezzo: float

def carica_vendite(percorso):
    vendite, errori = [], []
    with Path(percorso).open(newline="", encoding="utf-8") as fh:
        for numero, riga in enumerate(csv.DictReader(fh), start=2):       # row 1 is the header
            try:
                quantita = int(riga["quantita"])
                prezzo = float(riga["prezzo"])
                giorno = date.fromisoformat(riga["data"])
                if prezzo < 0:
                    raise ValueError("negative price")
                vendite.append(Vendita(giorno, riga["prodotto"], quantita, prezzo))
            except (ValueError, KeyError) as e:
                errori.append((numero, f"{type(e).__name__}: {e}"))
    return vendite, errori

contenuto = """data,prodotto,quantita,prezzo
2026-03-01,café,3,1.20
2026-03-02,thé,2,2.50
2026-03-02,café,2.5,1.20
2026-03-03,biscuits,4,-3.00
2026-13-01,café,1,1.20
2026-03-05,café,5,1.20
"""
percorso = Path(tempfile.mkdtemp()) / "vendite.csv"
percorso.write_text(contenuto, encoding="utf-8")

vendite, errori = carica_vendite(percorso)
print("valid sales:", len(vendite), "| discarded:", len(errori))
for numero, motivo in errori:
    print(f"  row {numero}: {motivo}")

per_prodotto = defaultdict(float)
for v in vendite:
    per_prodotto[v.prodotto] += v.quantita * v.prezzo
print({p: round(totale, 2) for p, totale in per_prodotto.items()}, "| total:", round(sum(per_prodotto.values()), 2))

Output obtained:

valid sales: 3 | discarded: 3
  row 4: ValueError: invalid literal for int() with base 10: '2.5'
  row 5: ValueError: negative price
  row 6: ValueError: month must be in 1..12, not 13
{'café': 9.6, 'thé': 5.0} | total: 14.6

How to read the solution. The dataclass with frozen=True represents a valid row and cannot be modified after creation. carica_vendite opens the file with an explicit encoding and newline="", and walks the rows with csv.DictReader (each row is a dictionary that uses the headers as keys); enumerate(..., start=2) numbers the rows the way a person opening the file sees them (row 1 is the header). Inside the try, every conversion can fail: int("2.5") fails (an integer has no decimal part), date.fromisoformat("2026-13-01") fails (month 13 does not exist), and the negative price is explicitly rejected. The errors do not stop the loading: each one is logged with row number and reason, and the three valid rows are processed. The total is obtained with a defaultdict(float) that starts from zero for each new product. The end user sees how many rows were discarded and why: it is the behaviour a real data pipeline needs.

Typical mistakes

From the long function to small functions: how to reason

When a function goes beyond thirty lines or so, or when you catch yourself writing comments to separate "the parts" inside it, it is usually doing more than one thing. The criterion for splitting it is simple: every part you can name with a verb (read, validate, convert, aggregate, write) becomes a function. The sales loader above, for example, mixes three responsibilities: opening and walking the file, converting a row into an object, accumulating the errors. A tidier version would have a function converti_riga(riga) -> Vendita that raises a specific error if the row is not valid, and a function carica_vendite that only walks the file and decides what to do with each error.

The advantages are three and show at once. Testability: converti_riga is tried with a hand-written dictionary, without creating files. Reuse: the same function serves to read rows from a file, from a database or from a web request. Readability: the body of carica_vendite becomes almost a sentence ("for each row: convert it, and if it fails, log the error"). The cost is having to choose names and signatures well; it is a cost that pays off at the first change. The practical rule many teams adopt is: if you have to scroll the screen to understand what a function does, it is too long; if a name like elabora_dati says nothing, the function does too many things or the name is lazy.

One last note on style with data: prefer functions that return results instead of printing them. A function that prints is usable only by a human in front of a terminal; one that returns a value can be tested, combined with others, written to a file, shown in a chart. Printing goes at the outermost point of the program, not inside the logic.

Best practices

• Small functions (under fifty lines or so) with a single task.
• Docstrings on public functions; type hints where they clarify.
• Explicit about encoding, always.
• Validate at the entry points of the system (files, user input), trust inside.
• Do not mutate arguments; return new objects.
• Prefer dataclass(frozen=True) for data records.
• Log errors with context (which file, which row, which value).

The seven rules boil down to two ideas: make explicit what the language leaves implicit (encoding, types, errors) and do not change what is not yours (arguments, globals). If you must choose where to begin in an existing codebase, begin with the encoding: it costs one line per file and prevents the class of bugs that is most expensive to diagnose.

Mini-project

Write carica_vendite(percorso) that reads a CSV with columns data, prodotto, quantita, prezzo:

1. It returns a list of Vendita (immutable dataclass) and a list of errors with the row number and the reason.
2. It discards rows with a non-integer quantity, a negative price or a malformed date (use datetime.date.fromisoformat).
3. It computes the total revenue and the revenue per product with a defaultdict.
4. Try it with a file that has accents in the product names, written in UTF-8 and then read with the wrong encoding to see the error.

The mini-project is the annotated solution above, rewritten by you. The four points are in order of difficulty: the first fixes the structures, the second the discard rules, the third the aggregation, the fourth makes you deliberately provoke the mojibake so you recognise it the next time you meet it on real data.

Challenge

• Easy. Write mediana(valori) with the same edge cases as media.
• Medium. Add to Vettore the scalar multiplication (__mul__) and the dot product as a method punto(altro).
• Hard. Write a decorator @cronometra that prints the execution time of a function. What must it return so as not to break the decorated function?
• Expert. Show with an example how pickle.loads can execute code (with an object that defines __reduce__), only in an isolated test environment, and report how you would avoid it.
• Enterprise. Write your team's standard for reading files: encoding, error handling, log of the discarded rows, minimum tests.

The five challenges start from a ten-line function and arrive at a team rule. Hard and Expert are the ones that teach most: the decorator forces you to understand what a closure returns, and the demonstration with pickle (only in a test environment) makes concrete a risk that would otherwise remain a line of warning.

What you can do now

You can write a function with an explicit signature, docstring and edge cases; you can explain why a file is made of bytes and recognise mojibake by looking at the first bytes; you can read a CSV and a JSON without losing the accents and without leaving files open; you can define a specific error that tells the row and the cause; and you know when an immutable dataclass is enough in place of a class. The next step is to understand why decimal numbers are not what they seem: floating point and IEEE 754.

The same series also includes: 09 - Python for Data 2: Lists, Tuples, Dictionaries, Sets (the previous lesson), 14 - Visualising Data with Matplotlib, 15 - scikit-learn: Estimators, Pipelines and Validation (which refer back to this lesson).

Sources