08 - Python for Data 1: Environment and Fundamentals

Python for Data, Part 1: Isolated Environment and Language Fundamentals

Machine Learning Engineer course · Module 01 · Python for data · Lesson 1

Goal. Put the reader in a position to write and run Python in a reproducible environment, with a correct mental model of what happens when they write b = a. By the end the reader can create a virtual environment, install packages without dirtying the system, and use variables, numbers, strings, conditions and loops knowing where the traps are.

False myth. "A variable is a box that contains a value." In Python a variable is a label attached to an object. Two labels can sit on the same object, and this is the source of the most common beginner bug.

Introduction

Python is the language in which machine learning is done: the libraries we will use (NumPy, Pandas, scikit-learn, and later the neural network frameworks) are written to be called from Python. You will not need to know every corner of it: you will need to know well the part that serves data. This lesson lays its foundations.

Roadmap. (1) Why virtual environments exist and how they are used. (2) Names and objects. (3) Numbers and operations, with integer division proved. (4) Truth and conditions. (5) Loops. (6) Beginner mistakes.

What it is, why it exists, when NOT to use it

Python is an interpreted language with dynamic typing: you write, you run, and the type of values is decided while the program runs. The advantage is the speed with which an idea can be tried. The price is that many errors appear only at run time, and that loops in pure Python are slow compared with compiled code: this is why data is processed with NumPy, which moves the work into compiled code (lesson 05).

When Python is NOT the right choice: intensive computation written by hand without libraries (write it in a compiled language or use NumPy); systems with rigid latency or memory requirements; native mobile applications. Python as "glue" that orchestrates fast libraries remains unbeatable for data.

Installation, isolated environment, check

Analogy: a toolbox per project

Imagine doing two jobs: one requires an old version of a tool, the other the new one. If you have a single shared toolbox, one job breaks the other. The virtual environment is one toolbox per project: a folder with a copy of the interpreter and of the packages, isolated from the rest of the system.

Why it is mandatory, not optional

On recent Linux distributions installing packages into the system Python with pip is refused, with the error externally-managed-environment: it is PEP 668, which protects the system from packages conflicting with those managed by the distribution's package manager. The virtual environment is the intended answer.

python3 --version                       # check: 3.12 or later
python3 -m venv .venv                   # creates the environment in the .venv folder
source .venv/bin/activate               # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install numpy pandas matplotlib scikit-learn
python -m pip freeze > requirements.txt # snapshot of the installed versions
python -c "import numpy, pandas, sklearn; print(numpy.__version__, pandas.__version__, sklearn.__version__)"

Three habits that avoid hours of trouble:

1. python -m pip, not pip: it guarantees that the package goes into the interpreter you are using.
2. One environment per project, the .venv folder excluded from version control.
3. A versioned requirements.txt (or equivalent): whoever comes after you recreates the same environment with python -m pip install -r requirements.txt.

The three habits fix three different faults: the first is the package installed into the wrong interpreter, the second is the conflict between projects, the third is the lack of reproducibility. They are ordered by frequency of incident: the first saves you in your very first week, the last when someone else tries to redo your work.

What the command python3 -m venv .venv really does. It creates a .venv folder containing a copy (or a link) of the Python interpreter, an empty site-packages folder where the installed packages will go, and a configuration file pyvenv.cfg that remembers which Python it was derived from. The activate command does not "install" anything: it modifies the PATH environment variable of your shell, so that typing python finds first the one inside .venv. Hence the check with which python. Inside Python you can check by yourself which environment you are in:

import sys

print("Python version:", sys.version_info[:3])
print("are you inside a virtual environment?", sys.prefix != sys.base_prefix)

# Output (outside a virtual environment, Python 3.14.4):
# Python version: (3, 14, 4)
# are you inside a virtual environment? False

Output obtained:

Python version: (3, 14, 4)
are you inside a virtual environment? False

How to read the code. sys.prefix is the root of the installation that is running the program; sys.base_prefix is the root of the base installation from which the environment was created. Inside a virtual environment the two paths differ; outside they are equal. Here the block was run outside an environment, so the answer is False: if you run it after activate it should become True. It is a check you can put at the beginning of a script to refuse to start in the wrong environment.

Check that it works: after activate, which python (or where python on Windows) must point inside .venv. If it points elsewhere, you are using the wrong Python.

Reproducibility, in practice. pip freeze writes the list of all installed packages with the exact version (numpy==2.3.5), even those you installed only indirectly as dependencies. It is fine to recreate an identical environment (for example on another computer or on a server); for a project that must remain readable, many people also keep a short list of just the packages they chose. The point is that the numbers produced by an analysis depend on the library versions: writing in the report "results obtained with NumPy 2.3 and scikit-learn 1.8" is not pedantry, it is what allows the reader to repeat the experiment and find the same numbers. We will do it in every lesson.

First example: names and objects

a = [1, 2, 3]
b = a                  # does NOT copy: attaches a second label to the same object
b.append(4)
print(a, a is b)       # a has changed, even though we only "touched" b

c = list(a)            # copy (shallow): a new object with the same elements
print(c is a, c == a)

# Output:
# [1, 2, 3, 4] True
# False True

Output: [1, 2, 3, 4] True and False True.

A second, contrasting example: names, functions and in-place changes

The b = a above shows two names on the same object. The same happens, less visibly, when you pass an object to a function: the parameter is a new name for the object you passed. What happens inside the function depends on a single distinction: does the function modify the object, or does it rebind the name to a new object?

def aggiungi_in_coda(lista, valore):
    lista.append(valore)                 # MODIFIES the received object: whoever passed the list will see it

def con_nuova_lista(lista):
    lista = lista + [99]                 # creates a NEW object and binds it to the local name
    return lista

dati = [1, 2, 3]
aggiungi_in_coda(dati, 4)
print("after aggiungi_in_coda:", dati)

nuova = con_nuova_lista(dati)
print("dati:", dati, "| nuova:", nuova, "| same object?", nuova is dati)

# the case of numbers: immutable, the += operator creates a new object
x = 10
y = x
y += 1
print("x =", x, "| y =", y)

# the contrasting case of lists: += modifies in place
a = [1, 2]
b = a
b += [3]
print("a =", a, "| b =", b, "| same object?", a is b)

Output obtained:

after aggiungi_in_coda: [1, 2, 3, 4]
dati: [1, 2, 3, 4] | nuova: [1, 2, 3, 4, 99] | same object? False
x = 10 | y = 11
a = [1, 2, 3] | b = [1, 2, 3] | same object? True

How to read the code. The first function calls append, a method that modifies the list: after the call dati also contains the 4, because the function and the caller share the same object. The second writes lista = lista + [99]: the + operator between lists builds a new list, and the assignment rebinds the local name lista to the new one; dati stays as it was and nuova is another object (is gives False). The case of numbers completes the picture: an int cannot be modified (it is immutable), so y += 1 means y = y + 1, that is a new number bound to y; x does not change. The final case is the trap: for lists b += [3] does not create a new list but modifies the existing one, and since a and b are the same object both change. The same line += has different meanings depending on whether the object is mutable or not.

The practical rule. A function that receives a mutable object (list, dictionary, array) and modifies it in place has a side effect visible outside. If you do not want it, work on a copy (copia = list(lista)) or return a new object and document it. "Pure" functions (which compute a result without modifying their arguments) are easier to test and understand, and are the style we will use for data transformations.

The mental model

Every object in Python has three properties (documentation of the data model): identity (never changes, it is a unique number while the object lives), type, value. The statement b = a creates a binding of the name b to the object already indicated by a. Two names, one object.

• is compares the identity: "are they the same object?".
• == compares the value: "are they equal?".

Two lists with the same elements are == but not is: they have the same content but are two distinct objects, with two identities. The value comparison is the one you want almost always; the identity comparison serves to know whether two names point at the same object, for example to discover an unwanted alias.

Real trap: writing if x is 1 instead of if x == 1. It works by chance with small numbers in some implementations, and fails elsewhere; the modern interpreter emits a SyntaxWarning precisely for this. Use is only for None (if x is None). On Python 3.14.4 the expression x is 1 run with -W error fails with SyntaxError: "is" with 'int' literal. Did you mean "=="?: the warning is a SyntaxWarning already emitted at compile time.

Why it matters for data: passing an array to a function that modifies it "in place" also alters the original. We will find it again with NumPy (views versus copies) and Pandas.

Reading an error: Python's messages are information

Beginners tend to be frightened when a "red screen" appears. In reality a Python error message is very precise information: it says what type of error it is, what caused it and where (the traceback lists the calls, from the outside to the inside, down to the offending line). It is read from the bottom: the last line gives the type and the message. Here are the six errors you will meet most often, provoked on purpose and caught:

casi = {
    "NameError": lambda: variabile_mai_definita,
    "TypeError": lambda: "3" + 4,
    "IndexError": lambda: [1, 2, 3][5],
    "KeyError": lambda: {"a": 1}["b"],
    "ValueError": lambda: int("12x"),
    "ZeroDivisionError": lambda: 1 / 0,
}
for nome, prova in casi.items():
    try:
        prova()
    except Exception as errore:
        print(f"{nome:<18} -> {errore}")

Output obtained:

NameError          -> name 'variabile_mai_definita' is not defined
TypeError          -> can only concatenate str (not "int") to str
IndexError         -> list index out of range
KeyError           -> 'b'
ValueError         -> invalid literal for int() with base 10: '12x'
ZeroDivisionError  -> division by zero

How to read the code. casi is a dictionary that associates the expected name of the error with an anonymous function (lambda) that provokes it. The loop runs each inside a try/except block: if the function raises an exception, execution jumps to except, which prints the message. The meaning of each: NameError = you used a name that does not exist (usually a typo or a missing assignment); TypeError = you combined incompatible types (here a text and a number); IndexError = you asked for a position beyond the end; KeyError = you asked for a key that the dictionary does not have; ValueError = the type is right but the value is not ("12x" is not an integer); ZeroDivisionError = division by zero. Learning to recognise the type immediately narrows where to look: a KeyError is searched for in dictionaries, an IndexError in lists.

Theory: numbers and their behaviour

Unlimited integers, limited decimals

Python's int has arbitrary precision: the size is limited only by memory.

x = 10**30 + 1
print(x, type(x))
print(7 / 2, type(7 / 2))        # "true" division: always float
print(7 // 2, 7 % 3)             # integer division and remainder

# Output:
# 1000000000000000000000000000001 <class 'int'>
# 3.5 <class 'float'>
# 3 1

Output: 1000000000000000000000000000001 <class 'int'>; 3.5 <class 'float'>; 3 1.

floats instead have finite precision (64 bits): lesson 04 explains their structure and consequences. For now remember: 0.1 + 0.2 == 0.3 is False.

Integer division and remainder: a demonstration of what they do with negatives

With positive numbers no surprise: 7 // 2 = 3, 7 % 2 = 1. With negatives, many languages give different answers, and the beginner is thrown. Python defines:

a // b is the floor of a / b (the largest integer ≤ the real quotient), and the remainder a % b is defined accordingly to preserve the identity

a = (a // b) · b + (a % b).

Check with a = −7, b = 3:

1. −7 / 3 = −2.333…, the floor is −3. So −7 // 3 = −3.
2. The remainder must satisfy −7 = (−3)·3 + r, that is r = −7 + 9 = 2. So −7 % 3 = 2.

The remainder always has the sign of the divisor (here positive). This choice makes % useful for "wrapping" indices: index % length is always between 0 and length − 1, even with negative indices.

print(-7 // 3, -7 % 3, divmod(-7, 3))
a, b = -7, 3
print(a == (a // b) * b + (a % b))

# Output:
# -3 2 (-3, 2)
# True

Output: -3 2 (-3, 2) and True.

Strings and formatting

Strings are immutable: every operation creates a new one. Modern formatting uses f-strings, which evaluate expressions between curly braces and accept format specifications after the colon.

nome, ricavo = "Marta", 1234567.891
print(f"{nome.upper()} billed {ricavo:,.2f} euros")   # thousands separator, 2 decimals
print(f"{0.4567:.1%}")                                 # percentage

# Output:
# MARTA billed 1,234,567.89 euros
# 45.7%

Output: MARTA billed 1,234,567.89 euros and 45.7%. (The thousands separator is the comma: it is a convention of the format, independent of the system language.)

Code: conditions and loops

Truthiness of values

In a condition, Python considers "false" (falsy): False, None, zero (0, 0.0), empty containers ("", [], {}, set()). Everything else is true. So if lista: means "if the list is not empty".

print(bool(0), bool(""), bool([]), bool([0]), bool(None))

# Output:
# False False False True False

Output: False False False True False. Note [0]: a list with one element (which is zero) is still non-empty, hence true.

Conditions

def classifica_voto(v):
    """Problem: translate a 0-30 grade into a band. Edge case: out-of-range grades."""
    if not 0 <= v <= 30:                 # chained comparison: readable as in mathematics
        raise ValueError(f"grade out of range: {v}")
    if v >= 28:
        return "excellent"
    elif v >= 24:
        return "good"
    elif v >= 18:
        return "sufficient"
    return "insufficient"

print([classifica_voto(v) for v in (30, 26, 18, 17)])

# Output:
# ['excellent', 'good', 'sufficient', 'insufficient']

Output: ['excellent', 'good', 'sufficient', 'insufficient'].

Points of style and correctness: the order of the branches matters (from the most restrictive); a raise branch at the beginning rejects invalid inputs early ("fail fast"); the comparison 0 <= v <= 30 is valid in Python and means 0 <= v and v <= 30.

for loops

for iterates over any iterable object. Two utilities you will use everywhere: enumerate (index-element pairs) and zip (iterating several sequences in parallel).

nomi = ["Anna", "Luca"]
voti = [30, 21]
for posizione, (nome, voto) in enumerate(zip(nomi, voti), start=1):
    print(posizione, nome, voto)

# Output:
# 1 Anna 30
# 2 Luca 21

Output: 1 Anna 30 and 2 Luca 21.

Trap: zip stops at the shortest sequence, silently. If nomi has 100 elements and voti has 99, you lose a row without errors. For aligned data use zip(a, b, strict=True), which raises an error if the lengths differ.

How to read the two previous loops. In the first, enumerate(zip(nomi, voti), start=1) produces pairs (position, (name, grade)): zip pairs the two lists element by element, enumerate adds a counter starting at 1, and the notation posizione, (nome, voto) unpacks the nested structure into three names. A for loop is always read like this: "for each element of the iterable, run the body with the name bound to that element". The advantage over for i in range(len(...)) is that you do not handle indices by hand, so you cannot get them wrong.

while loops

while repeats as long as the condition is true. It is used when you do not know in advance how many iterations are needed. Example: the Collatz sequence (if even divide by 2, if odd multiply by 3 and add 1) from the number 27.

n, passi = 27, 0
while n != 1:
    n = n // 2 if n % 2 == 0 else 3 * n + 1
    passi += 1
print(passi)

# Output:
# 111

Output: 111. The loop ends for 27, but whether it ends for every positive integer is an open mathematical problem (Collatz conjecture): a good reminder that a while requires reasoning about termination.

A loop with a numerical stopping criterion

while is the natural tool when you repeat until a quantity converges. We compute a square root with an iterative method (Newton's method applied to x² = n: you start from an estimate x and improve it with x ← (x + n/x)/2; the justification you will see in the calculus lesson). It stops when two successive estimates differ by less than a tolerance, and there is always a cap on the number of steps, because without a cap a reasoning error becomes an infinite loop.

import math

def radice(n, tolleranza=1e-12, tetto=60):
    x = n if n > 1 else 1.0                    # starting estimate
    for passo in range(1, tetto + 1):
        nuovo = 0.5 * (x + n / x)
        if abs(nuovo - x) <= tolleranza * max(1.0, abs(nuovo)):
            return nuovo, passo
        x = nuovo
    raise RuntimeError("the method does not converge within the step cap")

for n in (2, 9, 1e6, 0.0001):
    valore, passi = radice(n)
    print(f"radice({n}) = {valore:.12g} in {passi} steps | math.sqrt = {math.sqrt(n):.12g}")

Output obtained:

radice(2) = 1.41421356237 in 6 steps | math.sqrt = 1.41421356237
radice(9) = 3 in 7 steps | math.sqrt = 3
radice(1000000.0) = 1000 in 15 steps | math.sqrt = 1000
radice(0.0001) = 0.01 in 12 steps | math.sqrt = 0.01

How to read the code. The initial estimate is n (or 1 if n ≤ 1). At each step the mean of x and n/x is computed: if x is too large, n/x is too small, and the mean approaches the right value. The stopping test compares two successive estimates in a relative way (max(1, |nuovo|) prevents the absolute tolerance from becoming unsuitable for very large or very small numbers: lesson 4 on floating-point numbers). The method converges in few steps (from 6 to 15 for these values: it depends on how far the starting estimate is) because, once near the solution, at each iteration the number of correct digits roughly doubles. The final raise is the safety net: if for a pathological value it does not converge, the program says so instead of running forever.

Rule: every while must have an action inside the body that brings the condition closer to falsity, otherwise the program runs forever.

Sum of the first n integers: three ways, one theorem

n = 100
somma_ciclo = 0
for k in range(1, n + 1):
    somma_ciclo += k
print(somma_ciclo, sum(range(1, n + 1)), n * (n + 1) // 2)

# Output:
# 5050 5050 5050

Output: 5050 5050 5050. The third method is Gauss's formula, with a two-line proof: write the sum S = 1 + 2 + … + n and again backwards S = n + (n−1) + … + 1. Adding column by column, each pair is worth n + 1, and there are n pairs: 2S = n(n+1), so S = n(n+1)/2. The loop costs n operations, the formula one: a mathematical idea beats a loop. It is the spirit of the whole course, and the reason why NumPy, which expresses the computation on whole vectors, is faster.

The mini-project solution, commented

To close we put almost everything together: conditions, loops, exceptions, dictionaries and a bit of style. Here is a possible solution of the mini-project, with the three tests.

def classifica_voto(v):
    if not 0 <= v <= 30:
        raise ValueError(f"grade out of range: {v}")
    return v

def statistiche_voti(voti):
    validi, scartati = [], 0
    for v in voti:
        try:
            validi.append(classifica_voto(v))
        except ValueError:
            scartati += 1
    if not validi:                                   # empty list: no mean can be defined
        return {"valid": 0, "mean": None, "minimum": None, "maximum": None, "discarded": scartati}
    return {
        "valid": len(validi),
        "mean": sum(validi) / len(validi),
        "minimum": min(validi),
        "maximum": max(validi),
        "discarded": scartati,
    }

assert statistiche_voti([30, 24, 18]) == {"valid": 3, "mean": 24.0, "minimum": 18, "maximum": 30, "discarded": 0}
assert statistiche_voti([30, 99, -1, 20])["discarded"] == 2
assert statistiche_voti([])["mean"] is None
print(statistiche_voti([30, 99, -1, 20, 27]))
print("tests passed")

Output obtained:

{'valid': 3, 'mean': 25.666666666666668, 'minimum': 20, 'maximum': 30, 'discarded': 2}
tests passed

How to read the solution. classifica_voto repeats the validity check and raises ValueError for out-of-range grades. statistiche_voti scans the grades; for each one it tries to accept it inside a try: if the function raises a ValueError, the error is caught and the counter of discarded items is incremented, without the program stopping. The case of the empty list is decided explicitly: the mean of zero numbers does not exist, and returning None (and not, say, 0) is honest, because a zero would be a false value that might end up in a later calculation. The three asserts are the tests: if one fails, Python raises AssertionError indicating which. Writing the tests before or right after the function is the first form of verification; in the following lessons you will find it as a constant habit.

A minimal vocabulary

• Interpreter: the program that reads and executes Python code.
• Virtual environment: an isolated folder with its own interpreter and packages.
• Object / name: a value with identity, type and value / the label that designates it.
• Mutable / immutable: object that can / cannot be modified in place.
• Side effect: a change visible outside the function that caused it.
• Iterable: an object from which a for loop can take the elements one at a time.
• Traceback: the list of calls that led to an error.
• Falsy / truthy: values considered false / true in a condition.
• f-string: a string with expressions between braces, evaluated on the spot.

The nine entries fall into three blocks: the environment (interpreter, virtual environment), the data model (object and name, mutable, side effect, iterable) and reading errors (traceback, falsy, f-string). The middle block is the one the lesson keeps repeating: almost every beginner error on data comes from an unwanted alias or an unwanted in-place change.

Typical mistakes (symptom → cause → remedy)

Best practices

• Follow PEP 8: names in snake_case, readable lines, spaces around operators.
• Names that say what they contain (ricavo_netto, not x2).
• One environment per project; versioned requirements.txt.
• Error messages with the offending value (f"voto fuori scala: {v}").
• Read PEP 20 (import this): "Errors should never pass silently".
• Prefer built-in functions (sum, min, max, sorted) to hand-written loops.

The first three rules concern the order of the project, the fourth and fifth the way of failing usefully, the last speed and readability. The one that changes the quality of beginners' programs most is the error message with the offending value: it turns "something is wrong" into "grade 31 is out of range".

Mini-project

Write statistiche_voti.py that, given a list of grades (even with invalid values), returns a dictionary with the number of valid grades, mean, minimum, maximum and number of discarded ones.

1. Validate each grade with classifica_voto-like logic; discard (and count) those outside 0-30 by catching ValueError.
2. Compute the mean as sum / how many, handling the empty list case (what should it return?).
3. Add three asserts as tests: normal list, list with discarded values, empty list.

The three points go from the most mechanical to the most delicate: validation reuses the exception raised by the classification function, the mean asks you to decide what "no data" means, the tests fix the decisions. The commented solution above shows one choice (returning None), but it is not the only defensible one: what matters is that it is explicit.

Challenge

• Easy. Print the multiples of 7 below 100 with a single for loop and range.
• Medium. Write e_primo(n) with a loop up to int(n**0.5). Why is it enough to go as far as the square root? (If n = a·b with a ≤ b, then a ≤ √n.)
• Hard. Implement the greatest common divisor with Euclid's algorithm in a while, and prove termination: which quantity decreases at each step?
• Expert. Show with a numerical example that (a // b) * b + a % b == a for 1000 random pairs with mixed signs. Then explain why int(a / b) is not equivalent to a // b for negatives.
• Enterprise. Prepare a project template for a team: folder structure, .gitignore for .venv, requirements.txt, a README page with the three commands to recreate the environment.

The first two challenges are about calculation, the last three about reasoning: Hard asks you to prove termination and Expert to explain the difference between floor and truncation, which is exactly the trap of exercise 2. If you pick one, pick Expert: it repeats the most counter-intuitive point of the lesson with your own numbers.

Recap

Mind map. Virtual environment = a toolbox per project. Variable = label on an object. is = identity, == = value. // = floor, % = remainder with the sign of the divisor. Falsy = zero, empty, None. for for known sequences, while when a condition is needed, with proven termination.

3 concepts: isolated environment; names and objects; truthiness of values. 3 mistakes: pip into the system; aliases that modify; zip that truncates. 3 best practices: PEP 8; requirements.txt; fail fast with clear messages. 1 exercise: the mini-project of this lesson.

Next lesson: lists, tuples, dictionaries and sets: the structures in which you keep data.

The same series also includes: 07 - Cloud and Artificial Intelligence (the previous lesson), 09 - Python for Data 2: Lists, Tuples, Dictionaries, Sets (the next lesson).