WIP MITInstructors

ML From Scratch

Implement regression, classification, clustering and pure neural networks in NumPy

Status
WIP
License
MIT
Category
Instructors
Updated
GitHub is not a UI string to be translated from Italian into English.

Project status

Description

ML From Scratch is the most challenging path in the series: 10 weeks of implementing machine learning algorithms in pure NumPy, without high-level dependencies (no TensorFlow, no PyTorch, no scikit-learn for core algorithms-only for dataset loading and baseline comparison). The philosophy is 'understand by building': if you can implement backpropagation manually with loops and manual gradients, you really understand how it works; if you only called `model.fit()`, you understood how to use the API. The path covers: linear and polynomial regression with normal equation and gradient descent, logistic binary and multinomial regression, validation (k-fold cross-validation, train/val/test split, learning curve), regularization (L1 Lasso, L2 Ridge, ElasticNet, early stopping), classification with k-NN and Naive Bayes (Gaussian + Multinomial), decision trees (Gini, entropy, pruning, random forest optional), clustering (k-means, k-means++, DBSCAN, hierarchical), dimensionality reduction (PCA with SVD, t-SNE conceptual), multi-layer perceptron neural networks with manual backpropagation (forward pass, gradient computation, weight update, mini-batch SGD, Adam optimizer from scratch), interpretability (feature importance, SHAP conceptual, confusion matrix). Each Jupyter notebook is 'live': the user executes cells and sees every step (mathematical formula → NumPy code → Matplotlib visualization of fit). The main dataset is Boston Housing (regression) and Iris (classification) for basic steps, MNIST (digit classification) for the final neural network. License MIT, clean educational code (no over-engineering), dense pedagogical comments.

Key Features

  • Linear, Polynomial, Regularized Regression

    Normal Equation, Gradient Descent, Batch/Mini-Batch/SGD, L1/L2/Elastic Net Regularization, K-Fold Cross-Validation.

  • Classification: Logistic Regression, K-Nearest Neighbors, Naive Bayes

    Binary logistic regression and softmax multinomial, k-NN with KD-tree, Gaussian and multinomial Naive Bayes, metrics (precision, recall, F1, ROC-AUC).

  • Decision Tree and Random Forest

    Tree induction with Gini and entropy, post-hoc pruning, feature importance, random forests with bagging, out-of-bag error estimation.

  • Unsupervised clustering

    K-means with K-means++, Elbow Method, Silhouette Score, DBSCAN Density-Based, Hierarchical Agglomerative Clustering with Dendrogram.

  • Dimensionality Reduction

    Principal Component Analysis with SVD decomposition, Scree plot, explained variance, Conceptual t-SNE visualization, Comparison to scikit-learn baseline.

  • Neural Networks from Scratch (MLP)

    Manual forward pass, backpropagation with chain rule, gradient check, mini-batch stochastic gradient descent, Adam optimizer implemented from scratch, MNIST baseline.

  • Jupyter Notebooks for Pedagogy

    10 progressive notebooks, each step includes mathematical formulas, NumPy code, Matplotlib plots, and optional exercises. No empty Markdown.

Angular + Spring Boot on top of a PostgreSQL database hosted in the cloud via AWS or Google Cloud Platform

  • NumPy
  • Python 3.12
  • Jupyter Lab
  • Matplotlib
  • pandas
  • scikit-learn (only for baseline comparison)

Use cases

  • University students (computer science, statistics, physics) who want to understand ML behind APIs.
  • Developers mid-senior who use scikit-learn but want to internalize math.
  • Open source data science training materials sought (no black box)
  • Self-learners who want to prepare for machine learning interviews (e.g. "Implement k-means from scratch")

Road Map

  1. v0.1.0Scheduled

    Regression and Classification Base

    • Notebook 1: Linear Regression with Normal Equation
    • Notebook 2: Gradient Descent and Variants
    • Notebook 3: Binary and Multinomial Logistic Regression
    • Notebook 5: Decision Trees and Random Forests with Cross Validation
    • Boston Housing and Iris Baseline Dataset
  2. v0.5.0Scheduled

    Trees + Clustering + Dimension Reduction

    • Notebook 5: Decision Tree and Random Forest
    • Notebook 6: k-means clustering with DBSCAN and hierarchical clustering.
    • Notebook 7: Conceptual PCA and t-SNE
    • Workouts highlight importance and interpretability
  3. v1.0.0Scheduled

    Neural Networks + MNIST Dataset

    • Notebook 8: Manual MLP Forward and Backward Propagation
    • Notebook 9: Adam optimizer from scratch
    • Notebook 10: End-to-End MNIST Classification
    • Mathematical Complete Derivations Documentation for MkDocs

Who is it for

  • Junior/ Mid-Level Developer
  • For Developer Senior

Quick start

Start the project locally with four commands, requires Docker and GitHub Desktop is not required but a git client like SourceTree or Visual Studio Code's built-in one will do just fine though. 

# Quick start (work-in-progress, repo non ancora pubblicato)
# git clone https://github.com/<tba>/ml-from-scratch.git
# cd ml-from-scratch
# python -m venv venv && source venv/bin/activate
# pip install -r requirements.txt
# jupyter lab notebooks/

Contribute to the project

ML From Scratch is open source. Open an issue, propose a pull request or leave a star on GitHub: every contribution, no matter how small, is welcome.

  • ⭐ Leave a star if it's useful
  • 🐛 Report bugs or suggest features via issue
  • 🔧 Send a pull request with fixes or improvements
  • 📣 Share the project if you find it interesting
Star on GitHubOpen an issuePull request
← Back to list⭐ View on GitHub