Why the same model converges with one step size and diverges with another
Training a model means solving \min_\theta L(\theta), and the gradient only says which direction to descend in, not by how much. The step \eta (the learning rate) decides whether the descent converges, oscillates forever or explodes, and the variants of the algorithm (SGD, momentum, Adam, step schedules) are different ways of choosing or correcting that step. In this article each variant is tried on the same simple function, so that every behavior can be explained with a hand calculation. The gradient comes from the article on differential calculus; the function to minimize, in classification problems, is the negative log-likelihood seen in the article on probability and statistics. Notation: \theta are the parameters, \eta the step, g_t = \nabla_\theta L(\theta_t) the gradient at step t.
Gradient descent: the right step, too small, too large
The rule is \theta_{t+1} = \theta_t - \eta\,g_t. When all the data go into the gradient computation we speak of batch gradient descent. Try it on f(x) = x^4 - 3x^2 + 2, which has two minima at x = \pm\sqrt{1.5} \approx \pm 1.2247, where f = -0.25, and a local maximum at x = 0. The gradient is f'(x) = 4x^3 - 6x and at x = 2 it equals 32 - 12 = 20. With \eta = 0.1 the first step lands exactly on 2 - 0.1 \cdot 20 = 0, that is, on the maximum, where the gradient is zero and the method stops. With \eta = 0.2 the first step lands on -2, where the gradient is -20, and the second goes back to 2: a cycle.
import numpy as np
f = lambda x: x ** 4 - 3 * x ** 2 + 2 # two minima at x = +-sqrt(1.5) = +-1.2247, f = -0.25
g = lambda x: 4 * x ** 3 - 6 * x
print("gradient at x = 2:", g(2.0), " -> with lr = 0.1 the first step leads to", 2.0 - 0.1 * g(2.0))
for lr in (0.01, 0.05, 0.1, 0.2, 0.3):
x = 2.0
traj = [x]
for _ in range(100):
x = x - lr * g(x)
traj.append(x)
if abs(x) > 1e6:
break
tail = ", ".join(f"{v:.4g}" for v in traj[-3:])
print(f"lr = {lr:4}: last points {tail} f = {f(x):.4g}")
# Output:
# gradient at x = 2: 20.0 -> with lr = 0.1 the first step leads to 0.0
# lr = 0.01: last points 1.225, 1.225, 1.225 f = -0.25
# lr = 0.05: last points 1.225, 1.225, 1.225 f = -0.25
# lr = 0.1: last points 0, 0, 0 f = 2
# lr = 0.2: last points 2, -2, 2 f = 6
# lr = 0.3: last points 65.6, -3.386e+05, 4.658e+16 f = 4.706e+66
The lines of the listing show four regimes for the same starting point: with \eta = 0.01 and 0.05 you reach 1.225, the minimum; with 0.1 you stay stuck on the maximum, with 0.2 you bounce forever between \pm 2, with 0.3 the sequence explodes. Two lessons. The first is that a step that is too large does not merely make convergence unstable: it can lead to a stationary point that is not a minimum. The second is that the maximum acceptable step depends on the curvature: the function is steeper far from the origin, and a step that is fine near the minimum can be too large at the start. The fact that \eta = 0.01 and 0.05 end up in the same minimum does not mean they are equivalent: the second gets there much sooner.
SGD and mini-batches: an estimated gradient is cheaper and noisier
Computing the gradient over millions of samples at every step is expensive. Stochastic descent estimates it on a subset B (mini-batch):
The estimate is unbiased, because its expected value is the full gradient, but noisy: its standard deviation falls as 1/\sqrt{|B|}, so to halve the noise you must quadruple the batch. The listing measures it on a regression y = 3x + 1 + \text{noise} with 10000 samples, evaluating the gradient at the point (w, b) = (0, 0), where the true value is about (-6, -2) (for the model y \approx wx + b with standardized x, \partial L/\partial w = -2\,\mathbb{E}[xy] = -6 and \partial L/\partial b = -2\,\mathbb{E}[y] = -2).
import numpy as np
rng = np.random.default_rng(0)
N = 10_000
x = rng.normal(size=N)
y = 3 * x + 1 + rng.normal(scale=0.5, size=N) # data: y = 3x + 1 + noise
w, b = 0.0, 0.0
def grad(idx): # gradient of the mean squared error on idx
err = (w * x[idx] + b) - y[idx]
return np.array([2 * np.mean(err * x[idx]), 2 * np.mean(err)])
full = grad(np.arange(N))
print("full gradient (N = 10000):", np.round(full, 3), " expected about (-6, -2)")
for B in (1, 4, 16, 64, 256):
est = np.array([grad(rng.integers(0, N, size=B)) for _ in range(4000)])
print(f"B = {B:3d}: mean of the estimates {np.round(est.mean(axis=0), 2)} std of dL/dw = {est[:, 0].std():.3f}")
# Output:
# full gradient (N = 10000): [-5.975 -2.041] expected about (-6, -2)
# B = 1: mean of the estimates [-5.87 -2.04] std of dL/dw = 8.512
# B = 4: mean of the estimates [-6.08 -1.98] std of dL/dw = 4.321
# B = 16: mean of the estimates [-5.97 -2.07] std of dL/dw = 2.178
# B = 64: mean of the estimates [-5.98 -2.04] std of dL/dw = 1.076
# B = 256: mean of the estimates [-5.96 -2.03] std of dL/dw = 0.531
The means of the estimates stay close to (-6, -2) for every B: no bias. The standard deviation of the first component falls from 8.5 with B = 1 to 4.3 with B = 4, 2.2 with B = 16, 1.1 with B = 64 and 0.53 with B = 256: each time the batch is quadrupled, it halves, as predicted. In practice batches of tens or hundreds of samples are chosen (Ruder indicates they usually range from 50 to 256, with variations by application), a compromise between cost and noise. The noise is not only a flaw: it gives a bit of random movement that helps leave stationary points, but a step that is too large turns it into instability.
Momentum: remembering the repeated direction
On a function that is narrow in one direction and wide in the other, like an elongated valley, the gradient oscillates on the steep walls and advances slowly along the floor. Momentum accumulates past gradients into a «velocity»: v_t = \beta v_{t-1} + g_t, \theta_{t+1} = \theta_t - \eta\,v_t, with \beta typically 0.9 (Ruder). Components that change sign at every step cancel out in the sum, those that always point the same way add up. Another convention, v_t = \beta v_{t-1} + (1-\beta)g_t, differs only by a factor 1 - \beta absorbed in the step. Nesterov (NAG) computes the gradient at the point already shifted by the velocity, \nabla L(\theta_t - \eta\beta v_{t-1}), to correct the direction in advance.
The test bed is f(x, y) = x^2 + 10y^2, with gradient (2x, 20y): the curvatures are 2 and 20. One step of descent multiplies y by 1 - 20\eta and x by 1 - 2\eta. To converge we need |1 - 20\eta| < 1, that is \eta < 0.1, and the optimal value for plain descent, 2/(L + \mu) = 2/22 \approx 0.0909, gives a per-step factor (L-\mu)/(L+\mu) = 18/22 \approx 0.818. For momentum on a quadratic function the optimal values (a classic result of Polyak's «heavy ball», which I do not verify here in a source: the listing measures its effect) are a step \eta = 4/(\sqrt{L}+\sqrt{\mu})^2 and \beta = \left(\frac{\sqrt{L/\mu}-1}{\sqrt{L/\mu}+1}\right)^2, with a per-step factor of about 0.52.
import numpy as np
f = lambda p: p[0] ** 2 + 10 * p[1] ** 2 # curvatures 2 and 20: narrow valley
g = lambda p: np.array([2 * p[0], 20 * p[1]])
p0 = np.array([5.0, 5.0])
def gd(lr, steps):
p = p0.copy()
for _ in range(steps):
p = p - lr * g(p)
return f(p)
def momentum(lr, beta, steps): # v <- beta*v + g ; p <- p - lr*v
p, v = p0.copy(), np.zeros(2)
for _ in range(steps):
v = beta * v + g(p)
p = p - lr * v
return f(p)
print("contraction coefficient on y per step: |1 - 20*lr|")
for lr in (0.09, 0.099, 0.1, 0.105):
print(f" lr = {lr:5}: |1 - 20*lr| = {abs(1 - 20 * lr):.3f} f after 100 steps = {gd(lr, 100):.3e}")
print("at fixed lr 0.09: f after 10 / 50 / 100 steps =", [f"{gd(0.09, s):.2e}" for s in (10, 50, 100)])
print("momentum beta = 0.9, lr = 0.02: f after 10 / 50 / 100 steps =", [f"{momentum(0.02, 0.9, s):.2e}" for s in (10, 50, 100)])
L, mu = 20.0, 2.0 # maximum and minimum curvature
lr_t = 4 / (np.sqrt(L) + np.sqrt(mu)) ** 2
beta_t = ((np.sqrt(L / mu) - 1) / (np.sqrt(L / mu) + 1)) ** 2
print(f"tuned momentum (lr = {lr_t:.4f}, beta = {beta_t:.4f}): f after 10 / 50 / 100 steps =", [f"{momentum(lr_t, beta_t, s):.2e}" for s in (10, 50, 100)])
# Output:
# contraction coefficient on y per step: |1 - 20*lr|
# lr = 0.09: |1 - 20*lr| = 0.800 f after 100 steps = 1.552e-16
# lr = 0.099: |1 - 20*lr| = 0.980 f after 100 steps = 4.397e+00
# lr = 0.1: |1 - 20*lr| = 1.000 f after 100 steps = 2.500e+02
# lr = 0.105: |1 - 20*lr| = 1.100 f after 100 steps = 4.748e+10
# at fixed lr 0.09: f after 10 / 50 / 100 steps = ['3.35e+00', '1.11e-07', '1.55e-16']
# momentum beta = 0.9, lr = 0.02: f after 10 / 50 / 100 steps = ['6.73e+01', '1.81e-01', '7.13e-03']
# tuned momentum (lr = 0.1154, beta = 0.2699): f after 10 / 50 / 100 steps = ['1.36e-01', '5.41e-23', '7.72e-51']
The first four lines verify the stability threshold: at \eta = 0.1 the contraction coefficient on y is exactly 1 and the loss stays at 250 (the term 10y^2 with y = \pm 5), above the threshold it explodes. Then the comparison. With \eta = 0.09 plain descent reaches 10^{-16} in a hundred steps. Momentum with the «classic» values \beta = 0.9, \eta = 0.02 is slower (7 \cdot 10^{-3}), because the accumulated velocity overshoots the target and has to be reabsorbed. Only with parameters tuned to the curvatures does momentum win, reaching 10^{-51}. The gain is real but requires choosing \eta and \beta together; on a not very elongated valley like this one (curvature ratio 10) there is an advantage over well-tuned descent, but it is obtained with a tuning that in real problems is not known.
RMSprop and Adam: a different step for every parameter
If the curvatures differ greatly between parameters, a single \eta is awkward: it has to stay under the threshold of the steepest direction. RMSprop (proposed by Hinton in a lecture, not in a paper) divides the gradient by the square root of a moving average of its squares, s_t = \beta s_{t-1} + (1-\beta)g_t^2, \theta_{t+1} = \theta_t - \eta\,g_t/(\sqrt{s_t} + \epsilon): parameters with large gradients take smaller steps and vice versa. Adam (Kingma and Ba, 2015) combines this normalization with a moving average of the gradient, which acts as momentum:
The hyperparameters proposed by the paper are \eta = 0.001, \beta_1 = 0.9, \beta_2 = 0.999 and \epsilon = 10^{-8}.
The bias correction is needed because m and v start from zero. At the first step, with gradient g, we have m_1 = (1-\beta_1)g = 0.1g and v_1 = 0.001g^2; without correction the step is \eta \cdot 0.1g / (0.0316|g|) \approx 3.16\,\eta, with correction \hat m_1 = g, \hat v_1 = g^2 and the step is \eta\,g/|g| = \eta: every parameter takes a first step of size \eta, whatever the scale of its gradient.
import numpy as np
def adam_step(state, p, grad, lr=0.1, b1=0.9, b2=0.999, eps=1e-8, correct=True):
t, m, v = state
t += 1
m = b1 * m + (1 - b1) * grad # moving average of the gradient
v = b2 * v + (1 - b2) * grad ** 2 # moving average of the squared gradient
m_hat = m / (1 - b1 ** t) if correct else m # bias correction due to starting at zero
v_hat = v / (1 - b2 ** t) if correct else v
return (t, m, v), p - lr * m_hat / (np.sqrt(v_hat) + eps)
f = lambda p: p[0] ** 2 + 10 * p[1] ** 2
g = lambda p: np.array([2 * p[0], 20 * p[1]])
p0 = np.array([5.0, 5.0])
for correct in (True, False):
_, p1 = adam_step((0, np.zeros(2), np.zeros(2)), p0, g(p0), correct=correct)
print(f"gradient {g(p0)}, bias correction {correct!s:5}: first step = {np.round(p0 - p1, 4)}")
state, pa = (0, np.zeros(2), np.zeros(2)), p0.copy()
for _ in range(100):
state, pa = adam_step(state, pa, g(pa))
print(f"x^2 + 10 y^2, Adam (lr 0.1) after 100 steps: f = {f(pa):.3e}")
# very different scales: f = x^2 + 1000 y^2 (curvatures 2 and 2000)
f2 = lambda p: p[0] ** 2 + 1000 * p[1] ** 2
g2 = lambda p: np.array([2 * p[0], 2000 * p[1]])
pg = p0.copy()
for _ in range(200):
pg = pg - 0.0009 * g2(pg) # the maximum stable lr for GD is 2/2000 = 0.001
state, pa = (0, np.zeros(2), np.zeros(2)), p0.copy()
for _ in range(200):
state, pa = adam_step(state, pa, g2(pa))
print(f"after 200 steps: GD (lr 0.0009) f = {f2(pg):.3e} Adam (lr 0.1) f = {f2(pa):.3e}")
# Output:
# gradient [ 10. 100.], bias correction True : first step = [0.1 0.1]
# gradient [ 10. 100.], bias correction False: first step = [0.3162 0.3162]
# x^2 + 10 y^2, Adam (lr 0.1) after 100 steps: f = 1.673e-02
# after 200 steps: GD (lr 0.0009) f = 1.216e+01 Adam (lr 0.1) f = 1.201e-05
The first comparison in the listing verifies the hand calculation: with gradient (10, 100) both parameters move by 0.1 with the correction and by 0.316 without, the expected factor \sqrt{10}. In the second, the function x^2 + 1000y^2 has curvatures 2 and 2000: plain descent must use \eta < 0.001 to avoid diverging, and with 0.0009 after 200 steps x has moved little (f = 12.2); Adam with \eta = 0.1, tuning nothing, reaches 1.2 \cdot 10^{-5}. This is its advantage: robustness to scale. It is not a universal advantage, because on the previous function, well conditioned and with a tuned \eta, plain descent beat Adam (at 100 steps: f \approx 0.017 for Adam against 1.6 \cdot 10^{-16} from the previous listing), and Adam with a fixed step oscillates around the minimum with amplitude of the order of \eta.
Step schedules: warmup, step decay and cosine
A constant step is a compromise: large enough to make progress at the start, too large to settle near the minimum. A schedule reduces it over time. Step decay multiplies \eta by \gamma every s steps, \eta_t = \eta_0\gamma^{\lfloor t/s\rfloor}. Cosine annealing follows half a period of a cosine, \eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})(1 + \cos(\pi t/T)) (Loshchilov and Hutter, 2017, in the SGDR paper where it is combined with periodic restarts). Warmup makes the step grow linearly from zero for the first steps: it is used when a full step from the very beginning destabilizes training, for example because Adam's moment estimates still rest on few gradients (a frequent explanation that is not verified here). In the original Transformer (Vaswani et al., 2017) the step grows linearly for 4000 steps and then decreases as the inverse of the square root of the step: it is not a cosine curve.
import numpy as np
def step_decay(t, eta0=1e-3, gamma=0.1, s=300):
return eta0 * gamma ** (t // s)
def cosine(t, T, eta_min=1e-6, eta_max=1e-3):
return eta_min + 0.5 * (eta_max - eta_min) * (1 + np.cos(np.pi * t / T))
def warmup_cosine(t, warm, T, eta_max=1e-3):
if t < warm:
return eta_max * t / warm
return eta_max * 0.5 * (1 + np.cos(np.pi * (t - warm) / (T - warm)))
def transformer(step, d_model=512, warm=4000): # formula (3) of Vaswani et al.
step = max(step, 1)
return d_model ** -0.5 * min(step ** -0.5, step * warm ** -1.5)
T = 1000
print(" t step_decay cosine warmup+cosine")
for t in (0, 100, 300, 500, 600, 900, 1000):
print(f"{t:4d} {step_decay(t):.2e} {cosine(t, T):.2e} {warmup_cosine(t, 100, T):.2e}")
print("cosine at the halfway point:", cosine(500, T), " (eta_min + eta_max)/2 =", (1e-6 + 1e-3) / 2)
print("Transformer schedule: peak at step 4000 =", f"{transformer(4000):.3e}", " at step 16000 =", f"{transformer(16000):.3e}")
# Output:
# t step_decay cosine warmup+cosine
# 0 1.00e-03 1.00e-03 0.00e+00
# 100 1.00e-03 9.76e-04 1.00e-03
# 300 1.00e-04 7.94e-04 8.83e-04
# 500 1.00e-04 5.01e-04 5.87e-04
# 600 1.00e-05 3.46e-04 4.13e-04
# 900 1.00e-06 2.54e-05 3.02e-05
# 1000 1.00e-06 1.00e-06 0.00e+00
# cosine at the halfway point: 0.0005005000000000001 (eta_min + eta_max)/2 = 0.0005005
# Transformer schedule: peak at step 4000 = 6.988e-04 at step 16000 = 3.494e-04
The listing implements the four curves and prints their values. A hand check: at the halfway point the cosine equals the mean of the two extremes, (10^{-6} + 10^{-3})/2 = 5.005 \cdot 10^{-4}, and the listing gives exactly that value. The Transformer schedule, with d_{\text{model}} = 512, has a peak 512^{-0.5} \cdot 4000^{-0.5} \approx 7 \cdot 10^{-4} at step 4000 and then falls: at 16000 it is half, because \sqrt{4}=2. Warmup followed by a cosine is a frequent combination in practice, but none of the sources cited here presents it as a rule, and a schedule should be chosen by looking at the loss curve.
Saddle points: noise and negative curvature
In problems with a great many parameters the typical difficulty is not local minima but saddle points and the plateaus around them, as argued by Dauphin and colleagues (2014), already seen in the article on differential calculus. At a saddle point the gradient is zero, and descent stops if it starts exactly there. The simplest case is f(x, y) = x^2 - y^2, with gradient (2x, -2y): the coordinate y is multiplied at each step by 1 + 2\eta, so a small displacement grows exponentially. With \eta = 0.1 the factor is 1.2 and going from 10^{-6} to 1 takes \ln(10^6)/\ln(1.2) \approx 75.8 steps.
import numpy as np
# f(x, y) = x^2 - y^2: saddle at (0,0). Gradient (2x, -2y). From (1, y0) gradient descent...
lr = 0.1
for y0 in (0.0, 1e-6):
p = np.array([1.0, y0])
steps = 0
while abs(p[1]) < 1.0 and steps < 1000:
p = p - lr * np.array([2 * p[0], -2 * p[1]])
steps += 1
print(f"y0 = {y0:g}: after {steps} steps |y| = {abs(p[1]):.3g}, x = {p[0]:.2e}")
print("growth factor of y per step: 1 + 2*lr =", 1 + 2 * lr, "; expected steps for 1e-6 -> 1:", round(np.log(1e6) / np.log(1.2), 1))
# Output:
# y0 = 0: after 1000 steps |y| = 0, x = 1.23e-97
# y0 = 1e-06: after 76 steps |y| = 1.04, x = 4.31e-08
# growth factor of y per step: 1 + 2*lr = 1.2 ; expected steps for 1e-6 -> 1: 75.8
The listing confirms the calculation: starting from y = 10^{-6} you leave the saddle in 76 steps, while from exactly y = 0 you never leave (after a thousand steps y is still zero). In practice an exactly symmetric starting point does not occur and the mini-batch noise gives a small displacement anyway, but the time to escape grows only with the logarithm of the initial distance from the escape direction (76 steps for six orders of magnitude). Where the negative curvature is weak the growth factor is close to 1 and the steps needed become many more: this is the plateau Dauphin and colleagues talk about.
What you can do now
You can compute by hand the stability threshold of the step for a quadratic function, predict whether a choice of \eta converges, oscillates or diverges, and understand why Adam takes a first step of size \eta. You can estimate the noise of a mini-batch and decide whether it is worth increasing it, and you can recognize that momentum and Adam help when the curvatures are very different but are not always faster than well-tuned plain descent. The next step is to ask what is being minimized: the article on information theory defines entropy and cross-entropy, which are the most widely used loss in classification.
Sources
- Kingma, Ba, Adam: A Method for Stochastic Optimization, ICLR 2015 (arXiv 1412.6980) - algorithm 1, bias correction and proposed hyperparameters (eta 0.001, beta1 0.9, beta2 0.999, epsilon 1e-8); PDF read, accessed on 2026-11-12
- Ruder, An overview of gradient descent optimization algorithms (arXiv 1609.04747) - mini-batches between 50 and 256, momentum 0.9, Nesterov, RMSprop as Hinton's proposal; PDF read, accessed on 2026-11-12
- Loshchilov, Hutter, SGDR: Stochastic Gradient Descent with Warm Restarts (arXiv 1608.03983) - cosine annealing formula (equation 5); PDF read, accessed on 2026-11-12
- Vaswani et al., Attention Is All You Need (arXiv 1706.03762), section 5.3 - Adam with beta2 0.98, 4000-step warmup and decrease with the inverse square root; PDF read, accessed on 2026-11-12
- Dauphin et al., Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (arXiv 1406.2572) - saddle points and plateaus as the main difficulty in high dimension; abstract read, accessed on 2026-11-12
Commenti
Caricamento commenti...
Sign in to comment