A neural network layer is a matrix multiplication: how much algebra you really need
When a model classifies an image or generates a prediction, the numerical work is almost always the same: a vector of numbers comes in, it is multiplied by a matrix of weights, a bias is added and a nonlinear function is applied. To read machine learning code, and to understand why it sometimes explodes or fails to converge, four tools are enough: the dot product, the matrix product, eigenvalues and the singular value decomposition (SVD). This article builds them one at a time, each with its formula, its meaning, a small calculation worked by hand and the check with NumPy. If the NumPy result and the hand result did not agree, it would be the text that is wrong.
This is the first of ten articles. The second, differential calculus for deep learning, uses everything that follows to explain how gradients are computed through these same matrix products. The PCA of the article on dimensionality reduction is instead a direct application of eigenvalues and SVD, and it is best approached after reading the section on the decomposition below.
The notation we will use throughout the series
Formulas are readable only if a symbol always means the same thing. The conventions hold for all ten articles; when one introduces a new convention it defines it at first use.
- a scalar is an italic letter (x, \eta); a vector is a bold lowercase letter (\mathbf{x}) and is always a column; a matrix is a bold capital (\mathbf{W})
- a dataset is a matrix \mathbf{X} \in \mathbb{R}^{N \times d}: N samples on the rows, d features on the columns; the weights of a layer that turns d values into m are \mathbf{W} \in \mathbb{R}^{m \times d}
- \mathbf{a}^T\mathbf{b} is the dot product and \mathbf{A}^T the transpose; \|\mathbf{v}\|_2 and \|\mathbf{v}\|_1 are the L2 and L1 norms
- the parameters of a model are \theta, the loss function is L, the learning step is \eta, the expected value is \mathbb{E}[\cdot], the probability is P(\cdot)
The choice with the most practical consequences is the first: vectors are columns and the dataset has one sample per row. This is why a layer is written \mathbf{W}\mathbf{x} when you talk about one sample and \mathbf{X}\mathbf{W}^T when you process a whole batch, while the code in this series writes X @ W with the weights already transposed, as a d \times m matrix. The two ways are equivalent, but frameworks do not all agree on which to adopt, and if you do not know which one you have in front of you, shape errors are guaranteed.
Vectors and the dot product: measuring length and similarity
A vector \mathbf{v} \in \mathbb{R}^d is an ordered list of d numbers. In practice it is a sample: the features of a customer, the pixels of an image, the embedding of a sentence. Its length is measured with a norm. The L2 norm is the Euclidean distance from the origin, the L1 is the sum of the absolute values:
The dot product of two vectors multiplies the corresponding components and adds them up, and it has a geometric reading worth keeping in mind:
where \theta is the angle between the two vectors. If they point in the same direction the cosine is 1, if they are orthogonal it is 0, if they point in opposite directions it is -1. Dividing the dot product by the two lengths removes the effect of scale and leaves only the orientation: it is the cosine similarity of recommender systems and semantic search.
A small example. With \mathbf{a} = (2, -1, 4) and \mathbf{b} = (1, 3, 2) the dot product is 2 \cdot 1 + (-1) \cdot 3 + 4 \cdot 2 = 7. The L2 norm of \mathbf{a} is \sqrt{4 + 1 + 16} = \sqrt{21} \approx 4.5826 and its L1 norm is 2 + 1 + 4 = 7. The norm of \mathbf{b} is \sqrt{1 + 9 + 4} = \sqrt{14}, so the cosine is 7 / \sqrt{21 \cdot 14} = 7/\sqrt{294} \approx 0.4082: two vectors that resemble each other little. In the listing the last line tests the property that makes the cosine useful: \mathbf{a} and 2\mathbf{a} have different lengths but the same direction, and the cosine is exactly 1.
import numpy as np
a = np.array([2.0, -1.0, 4.0])
b = np.array([1.0, 3.0, 2.0])
dot = a @ b # 2*1 + (-1)*3 + 4*2
l2 = np.linalg.norm(a) # sqrt(4 + 1 + 16)
l1 = np.linalg.norm(a, ord=1) # 2 + 1 + 4
cos = dot / (np.linalg.norm(a) * np.linalg.norm(b))
print(f"a.b = {dot}")
print(f"L2 norm of a = {l2:.4f}")
print(f"L1 norm of a = {l1}")
print(f"cosine(a, b) = {cos:.4f}")
print(f"cosine(a, 2a) = {(a @ (2 * a)) / (np.linalg.norm(a) * np.linalg.norm(2 * a)):.4f}")
# Output:
# a.b = 7.0
# L2 norm of a = 4.5826
# L1 norm of a = 7.0
# cosine(a, b) = 0.4082
# cosine(a, 2a) = 1.0000
The dot product is written with the @ operator, which for two 1-D vectors coincides with np.dot; the alternative a * b would give the vector of component-by-component products, without the final sum. A neuron does exactly this: it computes \sigma(\mathbf{w}^T\mathbf{x} + b), that is, it measures how well the input aligns with the weight vector, adds a bias b and applies an activation \sigma.
Why L1 and L2 produce different weights
Regularization adds to the loss a penalty on the norm of the weights. With the L2 penalty (Ridge) the weights shrink but do not vanish; with L1 (Lasso) many become exactly zero. You can already see it in a single dimension. We look for the w that minimizes (w - a)^2 plus the penalty, with a = 0.3 and strength \lambda = 1. With the penalty \lambda w^2 the derivative vanishes at w = a/(1+\lambda) = 0.15: the weight is halved. With the penalty \lambda |w| the derivative to the right of zero is 2(w - a) + \lambda, which vanishes at w = a - \lambda/2 = -0.2, a negative value and therefore outside the branch; to the left the zero is at a + \lambda/2 = 0.8, positive and therefore outside its own branch. The minimum then sits at the corner, w = 0.
import numpy as np
a, lam = 0.3, 1.0
w = np.linspace(-1, 1, 200001)
ridge = (w - a) ** 2 + lam * w ** 2
lasso = (w - a) ** 2 + lam * np.abs(w)
print(f"ridge: minimum at w = {w[ridge.argmin()]:.4f} (a/(1+lam) = {a / (1 + lam):.4f})")
print(f"lasso: minimum at w = {w[lasso.argmin()]:.4f} (max(|a| - lam/2, 0) = {max(abs(a) - lam / 2, 0):.4f})")
# Output:
# ridge: minimum at w = 0.1500 (a/(1+lam) = 0.1500)
# lasso: minimum at w = 0.0000 (max(|a| - lam/2, 0) = 0.0000)
The grid search in the listing does not know the formulas: it scans 200001 values of w and takes the best one, and the agreement with 0.15 and 0 confirms the hand calculation. The reason L1 performs feature selection is therefore geometric: the penalty has a corner at the origin, and when |a| \leq \lambda/2 the minimum stays stuck there. L2 is smooth at the origin and so never gets there, unless a is zero.
Matrices: linear transformations, transpose, determinant and rank
A matrix \mathbf{A} \in \mathbb{R}^{m \times n} has m rows and n columns and does one of two things: it holds a dataset, or it represents a transformation that maps vectors of \mathbb{R}^n to vectors of \mathbb{R}^m. The product \mathbf{y} = \mathbf{A}\mathbf{x} has components y_i = \sum_{j} A_{ij} x_j: each component of the output is the dot product of one row of \mathbf{A} with \mathbf{x}. A complete layer adds the bias and the activation, \mathbf{h} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b}).
Hand calculation with \mathbf{W} = \begin{pmatrix} 0.5 & -0.3 & 0.8 \\ 0.2 & 0.7 & -0.4 \end{pmatrix} and \mathbf{x} = (1, 2, 3). The first row gives 0.5 - 0.6 + 2.4 = 2.3, the second 0.2 + 1.4 - 1.2 = 0.4. With a bias \mathbf{b} = (-3, 0.1) you get (-0.7;\ 0.5), and the ReLU, which zeroes negative values, leaves (0;\ 0.5). The first neuron stays off. The same listing also contains the determinant: for \begin{pmatrix} 3 & 1 \\ 2 & 4 \end{pmatrix} it is 3 \cdot 4 - 1 \cdot 2 = 10, and it is the factor by which the transformation multiplies areas: the unit square becomes a parallelogram of area 10.
import numpy as np
W = np.array([[0.5, -0.3, 0.8],
[0.2, 0.7, -0.4]])
x = np.array([1.0, 2.0, 3.0])
b = np.array([-3.0, 0.1])
z = W @ x
print("Wx =", np.round(z, 4))
print("Wx + b =", np.round(z + b, 4))
print("ReLU(Wx + b) =", np.round(np.maximum(0, z + b), 4))
print("shapes: W", W.shape, " x", x.shape, " W^T", W.T.shape)
A = np.array([[3.0, 1.0],
[2.0, 4.0]])
print("det(A) =", round(np.linalg.det(A), 4), " (3*4 - 1*2)")
# Output:
# Wx = [2.3 0.4]
# Wx + b = [-0.7 0.5]
# ReLU(Wx + b) = [0. 0.5]
# shapes: W (2, 3) x (3,) W^T (3, 2)
# det(A) = 10.0 (3*4 - 1*2)
The printed shapes are the cheapest check there is: a multiplication \mathbf{W}\mathbf{x} is defined only if the columns of \mathbf{W} (3) match the length of \mathbf{x} (3), and the result has as many elements as \mathbf{W} has rows (2). When a shape error shows up in a real model, it is almost always this rule being violated, usually because a matrix is stored transposed with respect to what the formula assumes.
An example we will use for the rest: three points on a line
To show covariance, rank, eigenvalues and SVD with the same numbers we take three samples with two features: (1, 2), (3, 6) and (5, 10). The second feature is always twice the first, so the points lie on a line. The means are 3 and 6; subtracting them (centering) gives the matrix \mathbf{X}_c with rows (-2, -4), (0, 0), (2, 4). The sample covariance is
with N = 3. On the diagonal are the variances of the two features (4 and 16), off the diagonal their covariance (8). The formula holds only after centering, and the factor N-1 is the one used by np.cov; dividing by N is a different choice, legitimate but to be declared.
The determinant is 4 \cdot 16 - 8 \cdot 8 = 0. A matrix with zero determinant is singular: it squashes space onto a lower dimension and cannot be inverted. It is the algebraic signal of collinearity between features: in linear regression the normal equations require inverting a matrix of this kind, and with duplicated or proportional features the inverse does not exist. The rank, the number of linearly independent rows, measures the same thing in a more robust way: here it is 1, because the second row is twice the first. The data live in a subspace of dimension 1, and this is the idea dimensionality reduction starts from.
Eigenvalues and eigenvectors: the directions the transformation does not rotate
An eigenvector of a square matrix \mathbf{A} is a direction that the transformation does not rotate but only scales: \mathbf{A}\mathbf{v} = \lambda\mathbf{v}. The factor \lambda is the eigenvalue. Eigenvalues are found by solving the characteristic equation \det(\mathbf{A} - \lambda\mathbf{I}) = 0. For the covariance of the example it becomes (4 - \lambda)(16 - \lambda) - 64 = \lambda^2 - 20\lambda = 0, so \lambda = 20 or \lambda = 0. For \lambda = 20 the equation (\mathbf{C} - 20\mathbf{I})\mathbf{v} = 0 gives -16 v_1 + 8 v_2 = 0, that is v_2 = 2v_1: the direction (1, 2)/\sqrt{5} \approx (0.4472;\ 0.8944), exactly the line on which the data lie.
The meaning for machine learning is this: the eigenvalues of the covariance are the variances along the principal directions. Here all the variance (20, that is 4 + 16, the trace of the matrix) lies along (1, 2)/\sqrt{5} and none in the orthogonal direction. In real data the eigenvalues are non-negative and very different from one another, and keeping only the eigenvectors of the largest ones is PCA, covered in the article on dimensionality reduction.
import numpy as np
X = np.array([[1.0, 2.0],
[3.0, 6.0],
[5.0, 10.0]])
N = X.shape[0]
Xc = X - X.mean(axis=0) # centering: zero mean per column
C = Xc.T @ Xc / (N - 1) # sample covariance
print("Xc =\n", Xc)
print("C =\n", C)
print("matches np.cov:", np.allclose(C, np.cov(X, rowvar=False)))
print("det(C) =", round(np.linalg.det(C), 6))
print("rank(C) =", np.linalg.matrix_rank(C))
try:
np.linalg.inv(C)
except np.linalg.LinAlgError as e:
print("inv(C) fails:", e)
vals, vecs = np.linalg.eigh(C) # eigh: symmetric matrix, ascending eigenvalues
print("eigenvalues:", np.round(vals, 6))
v = vecs[:, -1]
print("principal eigenvector:", np.round(v * np.sign(v[0]), 4))
print("(1, 2)/sqrt(5) :", np.round(np.array([1, 2]) / np.sqrt(5), 4))
print("C v = lambda v:", np.allclose(C @ v, vals[-1] * v))
s = np.linalg.svd(Xc, compute_uv=False)
print("singular values of Xc:", np.round(s, 4), " sqrt(40) =", round(np.sqrt(40), 4))
print("sigma^2 / (N-1):", np.round(s ** 2 / (N - 1), 6))
# Output:
# Xc =
# [[-2. -4.]
# [ 0. 0.]
# [ 2. 4.]]
# C =
# [[ 4. 8.]
# [ 8. 16.]]
# matches np.cov: True
# det(C) = 0.0
# rank(C) = 1
# inv(C) fails: Singular matrix
# eigenvalues: [ 0. 20.]
# principal eigenvector: [0.4472 0.8944]
# (1, 2)/sqrt(5) : [0.4472 0.8944]
# C v = lambda v: True
# singular values of Xc: [6.3246 0. ] sqrt(40) = 6.3246
# sigma^2 / (N-1): [20. 0.]
Here is how to read it. np.linalg.eigh is the function for symmetric matrices: it returns real eigenvalues in ascending order (which is why the principal eigenvector is the last column, vecs[:, -1]) and orthonormal eigenvectors; for non-symmetric matrices you need np.linalg.eig, which can give complex results. The sign of an eigenvector is arbitrary, \mathbf{v} and -\mathbf{v} are both valid, and that is why the listing normalizes it before printing. The inversion fails with LinAlgError: Singular matrix, as the zero determinant predicted. The last two lines anticipate the next section: the singular values of \mathbf{X}_c are the square roots of the eigenvalues of \mathbf{X}_c^T\mathbf{X}_c (\sqrt{40} \approx 6.3246), and dividing their squares by N-1 gives back the eigenvalues of the covariance, 20 and 0.
A warning about the determinant as a singularity test: here it is exactly zero because the numbers are small integers. With real data a nearly collinear feature gives a small but nonzero determinant, and the value depends on the scale of the features. The rank computed by np.linalg.matrix_rank uses the SVD with a tolerance, and is more reliable; the condition number (np.linalg.cond) tells you how close a matrix is to becoming singular.
SVD: the decomposition that works for any matrix
Eigenvalues exist only for square matrices, and data are almost always rectangular. The singular value decomposition holds for any \mathbf{A} \in \mathbb{R}^{m \times n}:
with \mathbf{U} \in \mathbb{R}^{m \times m} and \mathbf{V} \in \mathbb{R}^{n \times n} orthogonal, and \boldsymbol{\Sigma} \in \mathbb{R}^{m \times n} diagonal with the singular values \sigma_1 \geq \sigma_2 \geq \dots \geq 0. Applying the matrix to a vector amounts to three steps: rotate with \mathbf{V}^T, scale each axis by a \sigma_i, rotate again with \mathbf{U}. The squares of the singular values are the eigenvalues of \mathbf{A}^T\mathbf{A}, the link seen in the previous listing.
SVD is useful above all for approximating. Keeping the first k singular values gives \mathbf{A}_k = \sum_{i=1}^{k} \sigma_i \mathbf{u}_i \mathbf{v}_i^T, and the Eckart–Young theorem (1936) says that among all matrices of rank k this is the closest to \mathbf{A}, with an error in spectral norm equal to \sigma_{k+1} (Deisenroth, Faisal and Ong, Mathematics for Machine Learning, theorem 4.25). The Frobenius error is easy too: its square is the sum of the discarded \sigma_i^2.
Hand example. The 4 \times 4 ratings matrix in the listing has two blocks \begin{pmatrix} 5 & 4 \\ 4 & 5 \end{pmatrix}: two groups of users rating two groups of products. Each block is symmetric with eigenvalues 5 + 4 = 9 (direction (1, 1)/\sqrt{2}) and 5 - 4 = 1 (direction (1, -1)/\sqrt{2}), so the singular values are 9, 9, 1, 1. Keeping k = 2 leaves the term worth 9 of each block, which equals 9 \cdot \tfrac{1}{2}\begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}, that is 4.5 everywhere in the block: the mean of the block. The Frobenius error is \sqrt{1^2 + 1^2} = \sqrt{2}, the spectral one is \sigma_3 = 1.
import numpy as np
A = np.array([[5.0, 4.0, 0.0, 0.0],
[4.0, 5.0, 0.0, 0.0],
[0.0, 0.0, 4.0, 5.0],
[0.0, 0.0, 5.0, 4.0]])
U, s, Vt = np.linalg.svd(A) # s is a vector, Vt is already transposed
print("shapes of U, s, Vt:", U.shape, s.shape, Vt.shape)
print("singular values:", np.round(s, 4))
k = 2
Ak = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]
print("rank-2:\n", np.round(Ak, 2))
print("Frobenius error:", round(np.linalg.norm(A - Ak, "fro"), 4), " sqrt(1 + 1) =", round(2 ** 0.5, 4))
print("spectral error :", round(np.linalg.norm(A - Ak, 2), 4), " sigma_3 =", round(s[k], 4))
print(f"share of ||A||_F^2 captured: {np.sum(s[:k] ** 2) / np.sum(s ** 2):.4f} (162/164 = {162 / 164:.4f})")
# Output:
# shapes of U, s, Vt: (4, 4) (4,) (4, 4)
# singular values: [9. 9. 1. 1.]
# rank-2:
# [[4.5 4.5 0. 0. ]
# [4.5 4.5 0. 0. ]
# [0. 0. 4.5 4.5]
# [0. 0. 4.5 4.5]]
# Frobenius error: 1.4142 sqrt(1 + 1) = 1.4142
# spectral error : 1.0 sigma_3 = 1.0
# share of ||A||_F^2 captured: 0.9878 (162/164 = 0.9878)
np.linalg.svd returns three things that are not those of the formula: the singular values as a one-dimensional vector s (not as the matrix \boldsymbol{\Sigma}, which you rebuild with np.diag) and Vt already transposed. The rank-2 matrix in the listing reproduces the 4.5 computed by hand. The share 0.9878 is 162/164, the fraction of the squared Frobenius norm that remains; calling it «explained variance» is correct only if the matrix was centered first, otherwise it is simply energy. With repeated singular values (9 and 9) the singular vectors are not unique, but the approximation \mathbf{A}_2 is, because it depends only on the subspace.
Broadcasting: NumPy's way of pairing different shapes
Subtracting the mean from a dataset of 3 rows and 2 columns looks like an operation between incompatible shapes: (3, 2) minus (2,). NumPy aligns shapes from the right: a missing dimension, or one equal to 1, is virtually repeated, without copying the data. The mean of shape (2,) is therefore applied to every row. This is what makes centering or standardization a single line of code.
import numpy as np
X = np.array([[1.0, 2.0],
[3.0, 6.0],
[5.0, 10.0]])
mean = X.mean(axis=0) # shape (2,)
std = X.std(axis=0) # ddof=0: divides by N
print("X - mean =\n", X - mean) # (3,2) - (2,): the (2,) row repeats for every row
print("standardized =\n", np.round((X - mean) / std, 4))
y = np.array([1.0, 2.0, 3.0]) # shape (3,)
y_hat = np.array([[1.0], [2.0], [3.0]]) # shape (3,1): perfect predictions
print("shape of y_hat - y:", (y_hat - y).shape)
print("wrong MSE:", round(((y_hat - y) ** 2).mean(), 4))
print("correct MSE:", round(((y_hat.ravel() - y) ** 2).mean(), 4))
a = np.array([[1, 2], [3, 4]])
b = np.array([[5, 6], [7, 8]])
print("element-wise:\n", a * b)
print("matrix product:\n", a @ b)
# Output:
# X - mean =
# [[-2. -4.]
# [ 0. 0.]
# [ 2. 4.]]
# standardized =
# [[-1.2247 -1.2247]
# [ 0. 0. ]
# [ 1.2247 1.2247]]
# shape of y_hat - y: (3, 3)
# wrong MSE: 1.3333
# correct MSE: 0.0
# element-wise:
# [[ 5 12]
# [21 32]]
# matrix product:
# [[19 22]
# [43 50]]
The block shows the useful case and the dangerous one. Standardization, (\mathbf{X} - \text{mean})/\text{std}, gives \pm 1.2247 because the deviations -2, 0, 2 divided by \sqrt{8/3} \approx 1.633 equal \mp 1.2247; note that X.std divides by N by default, unlike np.cov. The dangerous case is the difference between y_hat of shape (3, 1) and y of shape (3,): broadcasting does not complain, it produces a 3 \times 3 matrix of all the cross differences, and the mean of the squares gives 1.3333 instead of 0, even though the predictions are perfect. No error, just a wrong number. The defense is to check the array shapes before a loss, or to flatten them with ravel(). The last two lines recall the difference between the component-by-component product (*: 1 \cdot 5 = 5, 2 \cdot 6 = 12) and the matrix product (@: 1 \cdot 5 + 2 \cdot 7 = 19).
Putting the pieces together: the forward pass of a two-layer network
With the matrix product, transpose and broadcasting you can write a network's inference: \mathbf{h} = \text{ReLU}(\mathbf{x}\mathbf{W}_1 + \mathbf{b}_1) and \mathbf{p} = \text{softmax}(\mathbf{h}\mathbf{W}_2 + \mathbf{b}_2), with the softmax \text{softmax}(\mathbf{z})_i = e^{z_i} / \sum_j e^{z_j} turning three scores into three probabilities. By hand, for \mathbf{z} = (1, 2, 3): the exponentials are 2.718;\ 7.389;\ 20.086, the sum is 30.193, and the probabilities are 0.0900;\ 0.2447;\ 0.6652. The softmax is invariant if you add the same constant to all the scores, and this is the property that allows subtracting the maximum: with scores of 1000 the exponential overflows and produces nan, while after the subtraction the result is identical to that of (0, 1, 2).
import numpy as np
def softmax(z):
e = np.exp(z - z.max(axis=-1, keepdims=True)) # subtracting the maximum does not change the result
return e / e.sum(axis=-1, keepdims=True)
print("softmax([1, 2, 3]) =", np.round(softmax(np.array([1.0, 2.0, 3.0])), 4))
big = np.array([1000.0, 1001.0, 1002.0])
with np.errstate(over="ignore", invalid="ignore"):
naive = np.exp(big) / np.exp(big).sum()
print("without subtracting the max =", naive)
print("with the subtraction =", np.round(softmax(big), 4))
rng = np.random.default_rng(42)
W1, b1 = rng.normal(0, 0.1, (4, 8)), np.zeros(8)
W2, b2 = rng.normal(0, 0.1, (8, 3)), np.zeros(3)
X = rng.normal(size=(5, 4)) # 5 samples, 4 features
h = np.maximum(0, X @ W1 + b1) # (5,4)@(4,8) -> (5,8)
P = softmax(h @ W2 + b2) # (5,8)@(8,3) -> (5,3)
print("shape of P:", P.shape, " row sums:", np.round(P.sum(axis=1), 6))
P_loop = np.vstack([softmax(np.maximum(0, x @ W1 + b1) @ W2 + b2) for x in X])
print("batch = one sample at a time:", np.allclose(P, P_loop))
# Output:
# softmax([1, 2, 3]) = [0.09 0.2447 0.6652]
# without subtracting the max = [nan nan nan]
# with the subtraction = [0.09 0.2447 0.6652]
# shape of P: (5, 3) row sums: [1. 1. 1. 1. 1.]
# batch = one sample at a time: True
The listing makes three checks. The first reproduces the softmax hand numbers. The second shows the nan of the naive version. The third checks that the batch of 5 samples, computed with two matrix products, gives the same result as a loop that processes one sample at a time: this is what justifies the matrix notation, which is equivalent but much faster, because a single call to an optimized library replaces the Python loop. The generator has seed 42, so the outputs are reproducible on the same version of NumPy.
What you can do now
You can compute by hand a dot product, a cosine, a layer and the covariance of a small dataset, and you can verify every result with a few lines of NumPy. You can read a zero determinant as collinearity, an eigenvalue as variance along a direction, a singular value as the weight of a component in the approximation, and you can recognize the silent broadcasting that produces a wrong number without an error. The same operations come back in information theory and in the mathematics of Transformers. The next step is to understand how the output of these operations changes when the weights change: it is the subject of differential calculus and backpropagation, where these same matrix products are differentiated.
Sources
- Deisenroth, Faisal, Ong, Mathematics for Machine Learning (Cambridge University Press, 2020) - norms, dot product and cosine (ch. 3), SVD and the Eckart-Young theorem 4.25 (ch. 4); PDF read, passages of theorems 4.24 and 4.25 checked, accessed on 2026-11-12
- Goodfellow, Bengio, Courville, Deep Learning, chapter 2 (Linear Algebra) - reference for layer notation and for the use of SVD in learning, page reached, accessed on 2026-11-12
- NumPy: numpy.linalg.svd - the function returns the singular values as a vector and V already transposed, accessed on 2026-11-12
- NumPy: Broadcasting - the rules for aligning shapes from the right, accessed on 2026-11-12
- MIT OpenCourseWare, 18.06 Linear Algebra (Gilbert Strang) - reference course for determinant, rank, eigenvalues and SVD, page reached, accessed on 2026-11-12
Commenti
Caricamento commenti...
Sign in to comment