When there are too many features and most of them are redundant
An 8×8 image dataset has 64 features, a sensor or text dataset has thousands, and many of them are blurry copies of the others: neighbouring pixels, correlated measurements, words that appear together. Principal component analysis (PCA) finds the few directions along which the data vary the most and rewrites the data in those coordinates. The construction is pure linear algebra, and it uses everything in the article on linear algebra: the covariance, its eigenvalues and the singular value decomposition. In this article you derive it, compute it by hand on four points, check it with NumPy and scikit-learn, and measure on real data when it helps and when it does not: the result on a classifier, obtained in this run, is not what many tutorials promise.
Notation of the series: \mathbf{X} \in \mathbb{R}^{n \times d} is the data matrix, with n samples on the rows and d features on the columns, \mathbf{C} is the covariance, and \lambda_i are the eigenvalues in decreasing order.
The curse of dimensionality, measured
In many dimensions the notion of "near" weakens: the distances between random points tend to have almost the same value. You can check it with an experiment. Draw 2000 uniform points in a cube [0,1]^d and measure, from a query point, the ratio (d_{\max} - d_{\min})/d_{\min} between the largest and the smallest distance. If the ratio is much larger than one, the nearest point is clearly nearer than the others; if it tends to zero, all the points are almost equidistant and a distance-based method (k-NN, clustering) loses discriminating power.
import numpy as np
rng = np.random.default_rng(0)
print(" d min distance max (max - min) / min")
for d in (2, 10, 100, 1000):
X = rng.random((2000, d)) # 2000 uniform points in the cube [0,1]^d
q = rng.random(d) # a query point
dist = np.linalg.norm(X - q, axis=1)
print(f"{d:4d} {dist.min():12.3f} {dist.max():7.3f} {(dist.max() - dist.min()) / dist.min():10.2f}")
# Output:
# d min distance max (max - min) / min
# 2 0.003 0.765 266.64
# 10 0.529 2.089 2.95
# 100 3.224 4.846 0.50
# 1000 12.028 13.452 0.12
The ratio drops from 266 in two dimensions to 2.95 with d = 10, 0.50 with d = 100 and 0.12 with d = 1000: at a thousand dimensions the farthest point is only 12% farther than the nearest. The value at d = 2 is dominated by chance (one very close point), so the trend matters, not the figure. Note that these points are uniform and independent; real data often lie close to a subspace of much lower dimension, and that is precisely the structure PCA tries to exploit.
Derivation: the direction of maximum variance is an eigenvector
Start from centred data (zero mean per column) and from the covariance \mathbf{C} = \frac{1}{n-1}\mathbf{X}^T\mathbf{X}. Projecting the data onto a unit direction \mathbf{w} gives the numbers \mathbf{X}\mathbf{w}, whose variance is \mathbf{w}^T\mathbf{C}\mathbf{w}. The first principal component is the direction that maximises this quantity under the constraint \|\mathbf{w}\| = 1. With a Lagrange multiplier you look for the stationary point of \mathbf{w}^T\mathbf{C}\mathbf{w} - \lambda(\mathbf{w}^T\mathbf{w} - 1); differentiating with respect to \mathbf{w} gives \mathbf{C}\mathbf{w} = \lambda\mathbf{w}. The direction you are after is therefore an eigenvector of \mathbf{C}, and since the variance along it is \mathbf{w}^T\mathbf{C}\mathbf{w} = \lambda\,\mathbf{w}^T\mathbf{w} = \lambda, the maximum is obtained by choosing the largest eigenvalue. The second component is the second eigenvector, orthogonal to the first, and so on (Deisenroth, Faisal and Ong, ch. 10).
With k components the reduction is \mathbf{Z} = \mathbf{X}\mathbf{W}_k, where \mathbf{W}_k \in \mathbb{R}^{d \times k} has the first k eigenvectors as columns, and the approximate reconstruction is \hat{\mathbf{X}} = \mathbf{Z}\mathbf{W}_k^T. The fraction of variance retained is \sum_{i \leq k}\lambda_i / \sum_{i \leq d}\lambda_i. A useful consequence: the squared reconstruction error is the sum of the discarded eigenvalues (multiplied by n-1 with the chosen normalisation), so choosing how many components to keep means deciding how much error to accept.
The steps, in the order in which they must be carried out, are four:
- centre the data by subtracting the mean of each column (and standardise them if the features have different units)
- compute the covariance \mathbf{C} or, equivalently, the SVD of the centred matrix
- sort the eigenvalues in decreasing order and choose k
- project with \mathbf{Z} = \mathbf{X}\mathbf{W}_k, keeping the mean and the components so you can apply the same transformation to new data
The order matters: without centring, the first "component" points towards the mean of the data and not towards the direction of maximum variation, and the eigenvalues must be sorted before choosing k, because np.linalg.eigh returns them in increasing order. The last step is the one that gets forgotten: the mean and the matrix \mathbf{W}_k computed on the training set must be reused on the test set, never recomputed.
A worked example on four points
Four samples, two features, already centred: (2, 2), (0, 1), (-1, -1), (-1, -2). With n = 4, \mathbf{X}^T\mathbf{X} = \begin{pmatrix} 6 & 7 \\ 7 & 10 \end{pmatrix} and dividing by n - 1 = 3 you get \mathbf{C} = \begin{pmatrix} 2 & 7/3 \\ 7/3 & 10/3 \end{pmatrix}. The trace is 16/3 \approx 5.3333 and the determinant 20/3 - 49/9 = 11/9, so the eigenvalues solve \lambda^2 - \tfrac{16}{3}\lambda + \tfrac{11}{9} = 0: \lambda_1 \approx 5.0934 and \lambda_2 \approx 0.2400. The first component explains 5.0934/5.3333 \approx 95.5\% of the variance. The eigenvector for \lambda_1 solves (2 - \lambda_1)v_1 + \tfrac{7}{3}v_2 = 0, that is v_2 \approx 1.3257\,v_1, and normalised it gives \mathbf{w}_1 \approx (0.6022;\ 0.7983). The first point is projected to 2 \cdot 0.6022 + 2 \cdot 0.7983 \approx 2.801, and the overall reconstruction error on the four points is (n-1)\lambda_2 = 3 \cdot 0.24 = 0.72.
import numpy as np
from sklearn.decomposition import PCA
X = np.array([[ 2.0, 2.0],
[ 0.0, 1.0],
[-1.0, -1.0],
[-1.0, -2.0]]) # 4 samples, 2 features, already centred
n = len(X)
C = X.T @ X / (n - 1)
print("C =\n", np.round(C, 4))
vals, vecs = np.linalg.eigh(C)
vals, vecs = vals[::-1], vecs[:, ::-1] # decreasing order
print("eigenvalues:", np.round(vals, 4), " trace =", round(np.trace(C), 4))
print("explained variance:", np.round(vals / vals.sum(), 4))
w1 = vecs[:, 0] * np.sign(vecs[0, 0])
print("first component w1 =", np.round(w1, 4))
z = X @ w1 # projection to 1 dimension
X_hat = np.outer(z, w1) # reconstruction
print("projections z =", np.round(z, 4))
print(f"reconstruction error (sum of squares) = {np.sum((X - X_hat) ** 2):.4f} (n - 1) * lambda_2 = {(n - 1) * vals[1]:.4f}")
# Same result from the SVD of X, without ever forming the covariance
U, s, Vt = np.linalg.svd(X, full_matrices=False)
print("sigma^2 / (n - 1):", np.round(s ** 2 / (n - 1), 4))
pca = PCA(n_components=1).fit(X)
print("scikit-learn: explained variance =", np.round(pca.explained_variance_, 4), " component =", np.round(pca.components_[0] * np.sign(pca.components_[0][0]), 4))
# Output:
# C =
# [[2. 2.3333]
# [2.3333 3.3333]]
# eigenvalues: [5.0934 0.24 ] trace = 5.3333
# explained variance: [0.955 0.045]
# first component w1 = [0.6022 0.7983]
# projections z = [ 2.8011 0.7983 -1.4005 -2.1989]
# reconstruction error (sum of squares) = 0.7199 (n - 1) * lambda_2 = 0.7199
# sigma^2 / (n - 1): [5.0934 0.24 ]
# scikit-learn: explained variance = [5.0934] component = [0.6022 0.7983]
The listing reproduces the calculations: same eigenvalues, same component \mathbf{w}_1, and the sum of the squared reconstruction errors coincides with (n-1)\lambda_2 = 0.7199 (the 0.72 of the hand calculation, with the unrounded figures). The most important part is the second one: the SVD of the data matrix, without ever forming the covariance, gives the same numbers through the rule \lambda_i = \sigma_i^2/(n-1). It is the route scikit-learn takes (the documentation of PCA says it applies the SVD to the centred data) and it is numerically more stable: forming \mathbf{X}^T\mathbf{X} squares the condition number of the matrix. The sign of an eigenvector is arbitrary, which is why the listing normalises it before comparing.
The scale of the features changes the result
PCA looks for variance, and variance depends on the unit of measurement: a feature expressed in grams has a variance a million times larger than the same value in kilograms. If the features are not on the same scale, the principal component points to the feature with the largest numbers, not to the most informative one. In the scikit-learn wine dataset (178 samples, 13 chemical properties) the feature proline has a variance of the order of a hundred thousand against hundredths for others.
import numpy as np
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
wine = load_wine()
X = wine.data # 178 samples, 13 features in different units
var = X.var(axis=0, ddof=1)
big = var.argmax()
print(f"feature with the largest variance: {wine.feature_names[big]} (variance {var[big]:.0f}); smallest: {var.min():.4f}")
raw = PCA().fit(X).explained_variance_ratio_
std = PCA().fit(StandardScaler().fit_transform(X)).explained_variance_ratio_
print("without standardising: PC1 =", round(raw[0], 4), " PC2 =", round(raw[1], 4))
print("after standardisation: PC1 =", round(std[0], 4), " PC2 =", round(std[1], 4))
print("weight of the dominant feature on PC1 (without standardising):", round(abs(PCA().fit(X).components_[0][big]), 4))
# Output:
# feature with the largest variance: proline (variance 99167); smallest: 0.0155
# without standardising: PC1 = 0.9981 PC2 = 0.0017
# after standardisation: PC1 = 0.362 PC2 = 0.1921
# weight of the dominant feature on PC1 (without standardising): 0.9998
Without standardising, the first component explains 99.8\% of the variance but coincides practically with a single feature (its weight is 0.9998): PCA has redescribed the dataset with its largest column. After standardisation (mean 0, variance 1 per column) the first component explains 36.2\% and the second 19.2\%: the variance is spread over several directions, and the components really do mix the features. Standardising is not always right, though: if the features are already in the same unit and the difference in variance is informative, standardising wipes it out. It is a decision to take by looking at the data.
How many components to keep, and what a classifier gains
The most common criterion is the cumulative fraction of explained variance: you choose the smallest k that exceeds a threshold, usually 90–99\%. The threshold is a convention, not a truth, and the real usefulness of k is judged on the final task. The listing does this on the digits dataset (1797 8×8 images, 64 features). Scikit-learn lets you give the threshold directly (n_components=0.95); here an integer is used to see the effect of each k. Standardisation and PCA sit inside a Pipeline, so that both are learned on the training set only.
import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_digits(return_X_y=True) # 1797 8x8 images = 64 features
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
full = PCA().fit(StandardScaler().fit_transform(X_tr))
cum = np.cumsum(full.explained_variance_ratio_)
for target in (0.80, 0.90, 0.95, 0.99):
print(f"components to explain {target:.0%} of the variance: {int(np.searchsorted(cum, target) + 1)} out of 64")
base = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
base.fit(X_tr, y_tr)
print(f"\nwithout PCA (64 features): test accuracy = {base.score(X_te, y_te):.4f}")
for k in (10, 20, 40):
m = make_pipeline(StandardScaler(), PCA(n_components=k), LogisticRegression(max_iter=5000))
m.fit(X_tr, y_tr)
print(f"with PCA, k = {k:2d}: test accuracy = {m.score(X_te, y_te):.4f}")
# Output:
# components to explain 80% of the variance: 21 out of 64
# components to explain 90% of the variance: 31 out of 64
# components to explain 95% of the variance: 40 out of 64
# components to explain 99% of the variance: 54 out of 64
#
# without PCA (64 features): test accuracy = 0.9722
# with PCA, k = 10: test accuracy = 0.8778
# with PCA, k = 20: test accuracy = 0.9417
# with PCA, k = 40: test accuracy = 0.9528
You need 21 components out of 64 for 80\% of the variance, 31 for 90\%, 40 for 95\% and 54 for 99\%: despite the correlations between neighbouring pixels, in these images the variance is spread over many directions. The result on the classifier runs against what tutorials suggest: logistic regression on the 64 features reaches a test accuracy of 0.9722, with PCA at 40 components (95\%) it falls to 0.9528, with 20 to 0.9417, with 10 to 0.8778. PCA did not improve generalisation. The reason is that logistic regression does not suffer from many features as much as a distance-based method does, and the low-variance directions that PCA discards still contain information about the class: variance is not the same thing as discriminating power. The gain to expect from PCA lies elsewhere: less memory, less time, visualisation, and methods sensitive to dimension. With a single split and a single seed, the difference between 0.9722 and 0.9528 (7 images out of 360) is not a precise measurement anyway.
Beyond PCA: t-SNE and UMAP are for looking, not for transforming
PCA is linear. When the data lie on a curved surface, non-linear methods are used to visualise them. t-SNE (van der Maaten and Hinton, 2008) builds neighbourhood probabilities between points in the original space and in the reduced one and minimises the KL divergence between the two, a concept covered in the article on information theory. UMAP (McInnes et al., 2018) starts from a theoretical framework of Riemannian geometry and algebraic topology and, according to its authors, is competitive with t-SNE in visualisation quality, with better running times and, in their judgement, more global structure preserved. UMAP is not installed in the environment used for this article, so it was not run.
import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE, trustworthiness
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
X, y = load_digits(return_X_y=True)
emb = {"PCA": PCA(n_components=2).fit_transform(X),
"t-SNE": TSNE(n_components=2, perplexity=30, init="pca", random_state=0).fit_transform(X)}
for name, Z in emb.items():
acc = cross_val_score(KNeighborsClassifier(5), Z, y, cv=5).mean()
tw = trustworthiness(X, Z, n_neighbors=10)
print(f"{name:6s} 2D: 5-NN accuracy (5 folds) = {acc:.3f} trustworthiness = {tw:.3f}")
# Output:
# PCA 2D: 5-NN accuracy (5 folds) = 0.603 trustworthiness = 0.830
# t-SNE 2D: 5-NN accuracy (5 folds) = 0.976 trustworthiness = 0.993
On the same 1797 digits the listing projects to 2 dimensions with PCA and with t-SNE and measures two things: the accuracy of a 5-fold 5-NN on the 2D coordinates, and the trustworthiness, the fraction of neighbours preserved (from 0 to 1). 2D PCA preserves little (0.603 and 0.830), t-SNE much more (0.976 and 0.993): local neighbours are preserved. But the guide by Wattenberg, Viégas and Johnson (How to Use t-SNE Effectively) warns about what the plot does not say: cluster sizes in a t-SNE plot carry no meaning, distances between clusters may mean nothing, and the result depends on the perplexity, for which you need to look at several values. Moreover t-SNE does not produce a transformation that can be applied to new data. This is why PCA remains the preprocessing tool (it is invertible and interpretable) and t-SNE and UMAP the exploration tools.
What you can do now
You can derive PCA as a constrained maximisation problem and recognise in \lambda_i the variance of component i. You can compute it by hand on a few points, check it with NumPy both from the covariance and from the SVD, and decide whether to standardise by looking at the scales. You can choose the number of components from the cumulative variance and, above all, you know the value has to be confirmed on the final task: on digits PCA lowered the accuracy of logistic regression instead of improving it. The next article changes subject but not thread: loss functions are what the gradient of the optimizer reduces.
Sources
- Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapter 10 (Dimensionality Reduction with PCA) - §10.2 and §10.3: derivation as variance maximisation, lost variance equal to the sum of the discarded eigenvalues (equation 10.25), reconstruction error perspective; PDF read, consulted on 2026-11-12
- scikit-learn: sklearn.decomposition.PCA - the class centres but does not scale the data and uses the SVD; page read, consulted on 2026-11-12
- McInnes, Healy, Melville, UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction (arXiv 1802.03426) - the authors' claims about UMAP (global structure and running times); abstract read, consulted on 2026-11-12
- Wattenberg, Viégas, Johnson, How to Use t-SNE Effectively, Distill 2016 - cluster sizes and distances in t-SNE and the effect of perplexity; article read, consulted on 2026-11-12
- van der Maaten, Hinton, Visualizing Data using t-SNE, JMLR 9, 2008 - the original t-SNE paper; page reached, text not read, consulted on 2026-11-12
Commenti
Caricamento commenti...
Sign in to comment