102 - ML-Ops: Git, the time machine for your project

Git is a system that records the history of a folder: every time you decide, it takes a photograph of it (a commit) with a message and an author, and lets you go back to any past photograph, work on several versions in parallel and understand who changed what. In a machine learning project it serves one more purpose: when a model gets worse, it tells you in which change it happened. This lesson uses Git 2.53, but the commands you will see have been stable for years. It comes after the introduction to the module, which explains why the code is the first of the five implicit things to make explicit.

The vocabulary of Git

Git has a lexicon of its own, and without it you cannot read any error message. The repository (repo) is the project folder plus its hidden history, which Git keeps in a .git subfolder. A commit is a photograph of the project at one instant, with message, author and date; it is identified by a hash (like 119e2f5) computed from its content, so change one character and the hash changes, and its first digits are enough to point to it. The working tree is the files as they are now on disk; the staging area (or index) is the tray where you place the changes you want in the next photograph.

A branch is a line of commits that advances on its own, reduced to a label pointing to the latest commit; HEAD is Git's "you are here", that is, the branch (or commit) you are working on. A merge brings into one branch the changes made on another, the diff is the precise list of lines changed between two versions (+ added, - removed) and a remote is a copy of the repository on another machine, for example GitHub, covered in the next lesson.

The path of a change, from the file you touch to the shared copy, goes through four places. The diagram shows the forward arrows (the commands that move work towards the history) and the dashed backward ones (the commands for undoing):

The analogy: a video game's save slots

Think of a game with manual saves. You save before the boss, play, lose: you reload. You save in several slots to try two strategies without losing either. Git works like this, with one important difference: every save has a label that you write ("feat: add AUC threshold") and, when you look at it, it shows you what changed with respect to the previous one. Inside, as we will see below, every commit is a complete photograph that reuses the unchanged files, so you can have thousands of them without taking up useless space.

The mechanism: three areas, not two

Beginners' confusion comes from the fact that Git has three places where your work lives, not two. In the working tree are the files as you see and edit them. The staging area holds the list of changes you have chosen to include in the next photograph. The repository is the sequence of commits already taken. The staging area serves to decide what goes into a commit: if you have modified three files but two are unfinished work, you add to the photograph only the file that is ready. A commit should tell one thing only.

The minimal round, which you will use a thousand times, has five commands: you create the repository, look at the status, bring a file into staging, take the photograph and read the history.

git init -b main                 # creates the repository in the current folder
git status                       # what changed, what is in staging
git add src/churn/model.py       # brings ONE file into staging
git commit -m "feat: add the C parameter to the logistic regression"
git log --oneline                # the history, one line per commit

The flag -b main chooses the name of the first branch; without it, Git uses the init.defaultBranch setting, which depends on the machine. Before the first commit, set who you are, once per machine: without a name and address Git refuses to commit, because every commit carries an author.

git config --global user.name "Your name"
git config --global user.email "you@example.com"

What you put in the commit, and what you do not

The rule is simple: what goes in the repository is what can be regenerated or what is needed to regenerate. Code goes in, the dependency file, the tests, the documentation. What stays out is whatever is large, derived or secret:

• Raw data and large datasets: Git copies the whole history, and an 800 MB file modified ten times weighs on the repository forever. For data, dedicated tools are used, as explained in the article on versioning datasets and models with DVC and lakeFS.
• Trained models (.joblib, .pt): they are outputs, and are recorded in a registry (MLflow, model registry and aliases).
• Virtual environments (.venv/): they are recreated from the lockfile (UV).
• Secrets and .env files: a committed key must be considered compromised even if you delete it afterwards, because it stays in the history.

The four items have the same root: everything that is derived from something else or must not circulate stays out, and the repository stays small and publishable. The case of secrets differs from the other three, because it is not a problem of weight but of security, and for that reason it is the only irreversible one. You declare all this in a .gitignore file at the root. A reasonable example for an ML project:

.venv/
__pycache__/
.ipynb_checkpoints/
data/raw/
mlruns/
mlflow.db
*.joblib
.env

Branches: trying without breaking

A branch is a parallel line of development, as light as a label. To try an idea (for example a different regularization value) you create a branch, work, and if it works you bring it back to the main line; if it does not work you throw it away without having touched anything.

git switch -c experiment/c-regularization      # creates and switches to the new branch
# ... you edit, try, commit ...
git switch main                                # back to the main line
git merge experiment/c-regularization          # integrate, if the experiment is good
git branch -d experiment/c-regularization      # clean up

One branch per experiment is a good habit: the history stays readable and you can compare two ideas with git diff main..other-branch. The price is that branches pile up, and the ones already integrated must be pruned with git branch -d, which refuses to delete a branch not yet integrated and so protects you from a loss.

Going back, in three levels of severity

There is also git reset --hard, which rewrites the present by throwing away uncommitted work: use it only knowing what you are throwing away. The practical rule is that commands that add history (commit, revert) are safe, while those that rewrite it require care, especially if someone else has already received those commits.

A complete session, command by command

Theory is really understood by watching what Git prints. I ran the whole session that follows, twice, in a temporary folder with Git 2.53 and LC_ALL=C (to have the messages in English, uniform). To make the hashes equal too I fixed the date and identity of the commits; on your computer the dates will be different and so the hashes will not match the ones shown: what counts is the format and the structure, not the digits.

git init -b main
git config user.name "Test" && git config user.email "a@b.c"

git init -b main creates the repository (.git) and calls the first branch main. The two config lines tell Git who signs the commits; here I set them only for this repository (without --global).

mkdir src
printf 'C = 1.0\n' > src/params.py
printf '.venv/\n*.joblib\n.env\n' > .gitignore
git status --short
# ?? .gitignore
# ?? src/

The ?? means "untracked": Git sees the files but does not know them yet. --short is the compact form of git status: two columns, on the left the state in staging, on the right in the working area.

git add src/params.py .gitignore
git status --short
# A  .gitignore
# A  src/params.py
git commit -m "feat: C parameter"
git log --oneline
# a8d1d8b feat: C parameter

After the add, the A in the left column says "added to staging". The commit creates the photograph; git log --oneline shows one line per commit, and those seven characters are the beginning of the full hash.

printf 'C = 0.1\n' > src/params.py
git diff
# diff --git a/src/params.py b/src/params.py
# index 7d6dcb9..c844a8e 100644
# --- a/src/params.py
# +++ b/src/params.py
# (Git prints the hunk header here, omitted)
# -C = 1.0
# +C = 0.1
git add -A
git diff                      # prints nothing: the change has moved to staging
git diff --staged --stat
#  src/params.py | 2 +-
#  1 file changed, 1 insertion(+), 1 deletion(-)

The diff shows one line removed (-) and one added (+): you changed a number. After git add -A, git diff shows nothing any more, because it compares the working tree with staging; to see what is about to go into the commit you use git diff --staged. Rule: before every commit, read git diff --staged. You discover files included by mistake (a .env, an 800 MB file) when removing them costs nothing.

The .gitignore in action

git commit -m "tune: C to 0.1"
touch model.joblib .env
git status --short            # prints nothing: the two files are ignored
git check-ignore -v model.joblib .env
# .gitignore:2:*.joblib	model.joblib
# .gitignore:3:.env	.env

After creating the two files, git status --short prints nothing. git check-ignore -v tells you why: each line reports the file, the line number and the rule that excludes it. When a file "does not want" to enter Git, or when one got in that should not have, it is the right tool to understand. Careful: .gitignore does not remove files already tracked. For a secret already committed, adding it is not enough: you need to remove it from tracking and replace the secret.

What a commit really is

git cat-file -t HEAD
# commit
git cat-file -p HEAD
# tree ede37614c8d100b0f34433119d5638b2d9b67d84
# parent a8d1d8bfc3db2c9c68b7f9405ea148ef83bf195a
# author Test <a@b.c> 1794477720 +0000
# committer Test <a@b.c> 1794477720 +0000
#
# tune: C to 0.1

The second command prints the raw content: a tree line (the snapshot of the files), a parent line (the hash of the previous commit), author and committer with date, and the message. It is the proof that the history is a chain: each commit points to its parent. That is why changing an old commit changes all the following hashes, and it is the reason rewriting shared history is delicate.

Branch and merge: two forms

git switch -c experiment/big-c
printf 'C = 10.0\n' > src/params.py
git commit -am "try: C to 10"
git switch main
git merge experiment/big-c
# Updating 90de7bd..c85fc80
# Fast-forward
#  src/params.py | 2 +-
git log --oneline --graph
# * c85fc80 try: C to 10
# * 90de7bd tune: C to 0.1
# * a8d1d8b feat: C parameter

commit -am adds the already tracked files to staging and commits in one step. Since nothing had been done on main in the meantime, the merge is a fast-forward: Git simply moves the main label forward, and the graph stays a straight line. If instead main has had commits of its own in the meantime, Git creates a merge commit with two parents. To see it I created a second branch exp2 two commits back, added a commit on each of the two lines and ran git merge --no-edit exp2:

git log --oneline --graph
# *   4bfa1de Merge branch 'exp2'
# |\
# | * 8a4e75b try: C to 2
# * | 899e2bf docs: note
# * | 95c9cd7 Revert "try: C to 10"
# * | c85fc80 try: C to 10
# |/
# * 90de7bd tune: C to 0.1
# * a8d1d8b feat: C parameter

The graph is read from the top: the merge commit 4bfa1de joins two lines, that of exp2 (a single commit) and that of main, which in the meantime had accumulated three more commits. The two lines that split and rejoin are exactly "two people worked in parallel". The block already shows the commit Revert "try: C to 10", which comes from the command in the following section.

Undoing, with three different commands for three situations

git revert --no-edit HEAD
# [main 95c9cd7] Revert "try: C to 10"
#  1 file changed, 1 insertion(+), 1 deletion(-)

It created a new commit, Revert "try: C to 10", which applies the opposite of the indicated commit: the file went back to C = 0.1, and the history contains both the mistake and its correction. It is the right choice for commits already shared, because it rewrites nothing. --no-edit accepts the default message instead of opening the editor.

printf 'C = 5\n' > src/params.py
git add src/params.py
git restore --staged src/params.py     # takes it out of staging; the file stays modified
git status --short
#  M src/params.py
git restore src/params.py              # throws away the file's changes
cat src/params.py
# C = 2

These are the two levels of restore in the diagram: --staged moves the change backward from staging to the working area (the M state, with the space on the left, says "modified but not in staging"); without --staged, the file goes back to what it was in the last commit. Here C = 2 holds because it is the value the previous merge brought onto main. The second command destroys uncommitted changes: there is no recycle bin.

The safety net: reflog

Did you delete a branch by mistake, or run one reset too many? Git keeps a diary of where HEAD has pointed, the reflog. Here is the proof, step by step: I make a commit, "throw it away" with a reset, and find it again.

printf 'C = 77\n' > src/params.py
git commit -am "try: C to 77"
# [main e91336c] try: C to 77
git reset --hard HEAD~1            # I "throw it away": the branch goes back by one
git log --oneline -1
# 4bfa1de Merge branch 'exp2'      (the commit is no longer there)
git reflog -3
# 4bfa1de HEAD@{0}: reset: moving to HEAD~1
# e91336c HEAD@{1}: commit: try: C to 77
# 4bfa1de HEAD@{2}: checkout: moving from 899e2bf842c19591d81ed5f7b431a2778bb3e7f7 to main
git switch -c recovered e91336c    # recreates a branch from a hash found in the reflog
cat src/params.py
# C = 77

The second line of the reflog shows the "lost" commit, e91336c, with its description; creating a branch on it brings it back to life and the content is back. Three observations. First: it works because Git does not immediately delete objects that are no longer reachable; according to the git config documentation, reflog entries that are no longer reachable expire after 30 days (gc.reflogExpireUnreachable) and the others after 90 (gc.reflogExpire), unless set otherwise. Second: the reflog is local, it is not shared with remotes, so it saves you from your own mistake, not from somebody else's. Third: it works only for what had been committed; uncommitted changes deleted by reset --hard or by restore are not in the reflog, because they were never a photograph.

The ML case: finding when the model got worse with bisect

Imagine noticing that the AUC required by the check has dropped, but you do not know in which of the last fifty changes. Git can search for it by bisection: you give it a "good" commit and a "bad" one, it tries the one in the middle, you (or a script) say whether it is good, and in a few steps it isolates the culprit. With n commits you need about log₂(n) trials: on 50 commits six trials are enough, not fifty.

Here is the automatic version, run on a test repository with eight commits, in which from the fifth onwards the threshold in the cfg file goes from 0.70 to 0.50:

git bisect start HEAD $(git rev-list --max-parents=0 HEAD)   # bad = HEAD, good = first commit
# Bisecting: 3 revisions left to test after this (roughly 2 steps)
git bisect run sh -c 'grep -q "AUC_MIN=0.70" cfg'           # exit 0 = good, non-zero = bad
# ...
# c116d4704afcdc0bec2dde3e3853534360182487 is the first bad commit
# bisect found first bad commit
git bisect reset                                             # go back to where you were

Git tried commits 4, 6 and 5 (three trials on eight commits) and pointed to the fifth as the first bad commit, the one that brings the threshold to 0.50. In the real case the command inside bisect run would be your test: uv run pytest tests/test_model.py -q. The test is the tool that tells Git what "good" means: one more concrete reason to write tests, the subject of pytest and of how to test a model.

Trap: git bisect reset does not accept options such as -q; its optional argument is a commit, and with -q Git answers that it is not a valid commit (tried with Git 2.53). If you forget the reset you stay in a bisecting state.

The why of Git: the problem it really solves

Before Git, the normal way to "version" was to copy the folder and change its name. It works for solo, short-lived work, and breaks in three predictable ways. First, you do not know what distinguishes two copies without comparing them file by file. Second, you cannot work two at a time on the same project: either you take turns, or each has their own copy and then someone has to merge them by hand. Third, you cannot go back to an intermediate point that you did not copy on purpose. Git solves these three problems with a single idea: the project's history is a chain of photographs, each with a parent, and parallel work is a branching of that chain.

One property of Git has practical consequences that often go unnoticed: it is distributed. Every clone of a repository contains the entire history, not just the latest version. Three useful things follow: you can work without a network (commit, comparisons, bisect are all local); every clone is also a complete backup copy; and the "server" (GitHub, covered in the lesson on GitHub) is just a clone that everyone has agreed to consider shared, not a technical authority.

How Git stores things (and why branches cost almost nothing)

In the previous example, git cat-file -p HEAD showed a commit with a tree line and a parent line. Behind it there is a simple model with three types of objects, all identified by the hash of their content: the blob (the content of a file), the tree (the list of file and folder names with the hash of the respective blobs and sub-trees) and the commit (a pointer to a tree, plus the parent, the author and the message). A commit is not a "difference" from the previous one: it is a complete photograph of the file tree, but since unchanged files have the same hash and are reused, the photograph costs only what has changed. It is like an album in which photos that do not change are pasted as a reference to the previous page.

An important consequence follows: a branch is just a text file containing the hash of a commit. In the test session .git/refs/heads/main weighed 41 bytes (wc -c): forty hexadecimal digits plus the newline. Creating a branch copies nothing: it adds a label. That is why "one branch per experiment" costs practically zero. HEAD, in turn, is a pointer to the branch you are on: when you make a commit, Git moves the current branch's label onto the new commit.

The "detached HEAD" state and why it scares people

Placing yourself on a commit that is not the tip of a branch (git checkout HEAD~1) produces this message, with git status:

git checkout -q HEAD~1
git status
# HEAD detached at 899e2bf
# nothing to commit, working tree clean

"Detached HEAD" means you are looking at an old photograph without a branch label following you. It is a normal and useful state for inspecting the past (re-running the tests on an old version, for example). The danger is another: if in this state you make new commits, no branch points to them, and when you move elsewhere they seem to have vanished (they remain recoverable with the reflog, as seen above). Simple rule: if you want to work from an old point, git switch -c new-branch before making changes; if you only want to look, look and then git switch main.

Naming releases: tags, and how they tie a model to the code

A branch moves with every commit; a tag is a label that stays where it is. It is used to mark the points that matter: a released version, the code with which you trained the delivered model.

git tag -a v1.0 -m "first release"
git commit --allow-empty -m "chore: after the tag"
git describe --tags
# v1.0-1-g6b60af0
git tag -n1
# v1.0            first release

tag -a creates an "annotated" tag (with message, author and date); git tag -n1 lists them with the first line of the message. git describe --tags is very useful: after a commit following the tag it answers v1.0-1-g6b60af0, that is "one commit after v1.0, and the current hash starts with 6b60af0". It is a readable and unique description of the code version, which you can write in logs or as the tag of an experiment in MLflow (tracking experiments): from that string anyone can get back to the exact code.

A critical look: git stash and this project's rule

When you try to switch branches with uncommitted changes that conflict, Git answers like this (message copied from the run):

error: Your local changes to the following files would be overwritten by checkout:
	note.txt
Please commit your changes or stash them before you switch branches.
Aborting

The standard suggestion is git stash: a command that sets the changes aside in a temporary area to reapply them later. This project, in its working rules, forbids it: the stash moves work "to an area invisible to status and log", and the risk is forgetting it there. The alternative rule is simple: commit (even a provisional commit, to be tidied later) or copy the files with an explicit path. It is a project choice and not a universal truth (many teams use the stash), but it is a good example of how team rules work: they arise from a concrete cost (lost or forgotten work) and are written where everyone sees them. For an ML engineer who has half an hour of uncommitted training in the working folder, the principle holds in any case: what is not in a commit does not exist, for Git.

Writing commits that another person can use

The message is for the you of six months from now, who needs to understand why, not what (the what is shown by the diff). This repository uses the format type(scope): subject, with types like feat, fix, refactor, docs, test, chore. A real example from its history: fix(routes): redirectTo assoluti (commit 87c0ed473a, verified with git log --oneline). It is a convention, described by the Conventional Commits specification, which makes the history readable and filterable. A few criteria, with examples:

The last example is typical of machine learning: a commit that changes a parameter should report the metric before and after (the figures in the example are invented, for teaching purposes). When, six months from now, a git bisect takes you to that commit, the message will be the first thing you read.

Questions you ask yourself on day one

"I committed on the wrong branch." If you have not pushed yet: git switch -c right (creates the right branch from the current commit), then move main back with git switch main && git reset --hard HEAD~1. The reset --hard rewrites: use it only on commits never published, and only after checking with git log that the commit is also on the new branch.

"I got the message of the last commit wrong." git commit --amend -m "corrected message", once again only if it has not been published (it changes the hash).

"How often should I commit?" Every time you have completed a unit of work that makes sense on its own and leaves the project in a working state: a passing test, an added parameter, a fix. Small, frequent commits make bisect and revert effective, but require discipline in the message. If you cannot write the message in one line without using "and", it is probably two commits.

"Should I put notebooks in Git?" Yes, but they are difficult files: an .ipynb also contains the outputs (charts in base64), and a diff becomes unreadable. Common practices: clear the outputs before committing, or keep the logic in .py files and use the notebook only as an exploration interface.

What to do with files that are too large

The technical reason to keep data and models out of Git is the combination of two facts: Git keeps the complete history in every clone, and large binary files neither compress nor "diff" well. A 200 MB model committed ten times can weigh gigabytes in every clone, forever (even deleting it afterwards, because it stays in the history): this is an order of magnitude of the argument, not a measurement. The alternatives come in three families: tools that put a small pointer file in the repository and keep the content elsewhere (Git LFS, DVC, covered by the series on versioning datasets and models); an object store with versions; or, for models, the MLflow registry. In every case the principle is the same: Git versions the code that produces the artifact and a reference to it, not the artifact.

What you can do now

You can bring a file from the working tree into the history through staging, and read git diff --staged before every commit. You can decide what to keep out of the repository and declare it in .gitignore, try an idea on a branch and join it back with a merge. You can undo with the right command for the situation (restore, revert), find a "lost" commit with the reflog and find with git bisect run the change that broke a test. The next lesson takes this history to a shared server: GitHub, remote, protected branches and pull requests.

The same series also includes ML-Ops: using open source well, from licenses to contributing, ML-Ops: environment variables and secrets outside the code and ML-Ops: deployment, where the model runs and how to choose.