80 - Deep Q-Learning: From Table to Neural Network, and Why Training Is Unstable

In the previous lesson, Gymnasium, Taxi and FrozenLake, the table worked beautifully because there were few states: 500 in Taxi, 16 in FrozenLake. Now the problem changes nature. In CartPole the state is made of four real numbers, and between any two real values there are always infinitely many others: a table with one row per state cannot even be written down. Deep Q-Learning (DQN, Mnih and colleagues) replaces the table with a neural network that estimates Q(s, a) for any state, even one never seen. The price is that training becomes unstable, and this lesson measures that: you build a complete DQN, remove one at a time the two devices that keep it standing, and watch what happens when you drop it into a simulator whose physics has been changed.

A note on method, because the numbers on this page are few and must be read for what they are. Every DQN training run here lasts 20,000 steps, with three seeds per configuration, on CPU and with a fixed number of threads. These are small experiments, chosen so that each can be redone in a few minutes, not a definitive evaluation: where the difference between two configurations is within the variability between seeds, the text says so and draws no conclusions. An earlier version of this lesson reported tables over 40,000 steps with more configurations; those numbers were not reproduced and have been removed, because the outputs you read here all come from listings that were re-run.

CartPole in brief

CartPole is a cart on a track with a pole hinged on top. The observation is a vector of four real numbers (cart position and velocity, pole angle and angular velocity), there are two actions (push left or right) and you receive +1 for every step, including the one in which the episode ends. According to the Gymnasium documentation the episode terminates (terminated) if the pole goes beyond ±12° (0.2095 radians) or the centre of the cart leaves ±2.4, and it is truncated (truncated) when it exceeds 500 steps: 500 is therefore the maximum return.

From the Q-Learning lesson you remember the distinction between the two ways an episode can end: the bootstrap is zeroed only for terminated, never for a truncation due to time, because the state after a truncation is not really without a future. The code below respects this by storing only terminated in the buffer. The rest of the vocabulary is needed as we go: the replay buffer is a circular archive of the latest transitions from which random batches are drawn; the target network is a frozen copy of the network, updated now and then, that computes the target; the Huber loss is quadratic for small errors and linear for large ones, so it is less sensitive to outlying errors than the squared error.

The limit of the table, measured

Before inventing a network, it is worth trying the obvious remedy: keep the table and discretise the state, that is, split each real number into intervals and use the combination of the four intervals as the row number. The listing below uses the Q-Learning of the lesson on Q-Learning and SARSA with 3, 6, 10 and 16 intervals per dimension, so tables from 81 up to 65,536 rows, and reports the mean return of the last 200 training episodes (with ε still at 0.05, so this is not a greedy evaluation).

import numpy as np
import gymnasium as gym

# What if we keep the TABLE and chop the continuous state into bins?
def make_binner(bins_per_dim):
    low = np.array([-2.4, -3.0, -0.21, -3.5]); high = -low       # cart position, cart velocity, pole angle, pole angular velocity
    edges = [np.linspace(l, h, bins_per_dim + 1)[1:-1] for l, h in zip(low, high)]
    def to_state(obs):
        idx = [int(np.digitize(o, e)) for o, e in zip(obs, edges)]
        s = 0
        for i in idx:
            s = s * bins_per_dim + i
        return s
    return to_state, bins_per_dim ** 4

def tabular_cartpole(bins, episodes=3000, alpha=0.1, gamma=0.99, seed=0):
    to_state, n_states = make_binner(bins)
    env = gym.make("CartPole-v1"); rng = np.random.default_rng(seed)
    Q = np.zeros((n_states, 2)); returns = []
    for ep in range(episodes):
        eps = max(0.05, 1.0 - ep / (episodes / 2))
        obs, _ = env.reset(seed=int(rng.integers(1 << 30))); s = to_state(obs); total, done = 0.0, False
        while not done:
            a = int(rng.integers(2)) if rng.random() < eps else int(np.argmax(Q[s]))
            obs, r, terminated, truncated, _ = env.step(a); s2 = to_state(obs)
            target = r + (0.0 if terminated else gamma * Q[s2].max())
            Q[s, a] += alpha * (target - Q[s, a]); s = s2; total += r; done = terminated or truncated
        returns.append(total)
    return np.mean(returns[-200:]), n_states

for bins in (3, 6, 10, 16):
    m, n = tabular_cartpole(bins)
    print(f"{bins:2d} bins per dimension -> {n:6d} table rows: mean return over the last 200 training episodes = {m:6.1f}")

#  3 bins per dimension ->     81 table rows: mean return over the last 200 training episodes =  111.5
#  6 bins per dimension ->   1296 table rows: mean return over the last 200 training episodes =  132.4
# 10 bins per dimension ->  10000 table rows: mean return over the last 200 training episodes =  168.7
# 16 bins per dimension ->  65536 table rows: mean return over the last 200 training episodes =  142.9

How to read the code: make_binner defines plausible extremes for each dimension and bins − 1 evenly spaced thresholds; np.digitize assigns each value to its interval and the four labels are combined into an index in base bins, so the table has bins⁴ rows. The output says two things. No table gets anywhere near 500: the maximum is 169, with 10,000 rows. And making the grid finer does not help: from 10,000 to 65,536 rows the value drops from 169 to 143, because more rows mean fewer visits per row, so sparser data. With few intervals different states are confused in the same cell; with many, each cell learns too little. This is the dilemma the network sidesteps by generalising, because nearby states share weights. The listing was run twice with identical output.

From table to network: the loss

The network Q(s, a; θ) receives the state and returns one value per action (two numbers in CartPole). The tabular Q-Learning rule, Q(s, a) ← Q(s, a) + α [ r + γ max Q(s′, ·) − Q(s, a) ], is a gradient-descent step on a squared error in which the target is treated as a constant. With a network you do the same: you define the loss below, and you take a gradient step only on θ, without differentiating the target. This is called semi-gradient: if you also differentiated through y, the estimate and the target would move together and the gradient would no longer represent the error you want to reduce.

L(θ) = ( y − Q(s, a; θ) )²       con   y = r + γ · max_{a′} Q(s′, a′; θ⁻)       (y = r se s′ è terminale)

The symbol θ⁻ indicates that the target is computed with a frozen copy of the weights, the target network, and not with θ: we will see shortly why. In practice the squared error is replaced by the Huber loss, which here is PyTorch's smooth_l1_loss. Note also what is missing: in the table every row updates by itself, whereas here a single gradient step moves the weights shared by all states. That is the source of everything that follows.

Why the naive version does not work

Taking the formula and training the network step by step on the latest transition, with θ⁻ = θ, seems natural and runs into two distinct problems, which DQN solves with two distinct devices. Sutton and Barto, in section 11.3 of their book, frame them as the deadly triad: instability and divergence can appear when you combine function approximation, bootstrapping (targets that contain existing estimates) and off-policy training. DQN has all three. The book adds that with two out of three instability can be avoided, but giving up approximation is not realistic, as you saw above with the table.

First problem: correlated samples. The data from a trajectory are consecutive and similar: if the pole is falling to the right, the next ten transitions tell the same story, and a network trained on a long stretch like that tends to forget the rest. The remedy is the replay buffer: you store the latest 50,000 transitions and at every step you train on a random batch of 64, which contains very different moments. The idea of sampling at random among past transitions is in the 2013 paper by Mnih and colleagues (Playing Atari with Deep Reinforcement Learning), which presents it precisely to attenuate correlated data and non-stationary distributions.

Second problem: the target that chases itself. In tabular Q-Learning, updating one cell changes the target of another row only if the two are connected. With a network every step changes the weights, and therefore changes at the same time the estimate Q(s, a; θ) and the target y, which depends on Q(s′, ·) computed with the same weights. It is like chasing a target that moves at every move you make. The remedy is the target network: the weights θ⁻ are copied from θ only every N steps (here 500) and between one copy and the next the target stands still. The target network appears in the 2015 Nature paper, which in its extended data also has a table on the effects of replay and of the target network, separately, on Atari games: here you redo the experiment on CartPole.

The complete code

Here is the whole DQN: replay buffer, network and training loop, plus the evaluation function. It contains three switches (use_replay, use_target, double) that serve the measurements. There is also a detail that is missing from many versions in circulation and is the cause of non-repeatable outputs: env.action_space.seed(seed). Without it action_space.sample(), used for exploration, draws from a generator that does not depend on your seed, and two identical runs give different results. Torch, NumPy, random, the environment and the action space are all seeded, and the number of PyTorch threads is fixed at 2.

import copy
import random
from collections import deque
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import gymnasium as gym

torch.set_num_threads(2)      # fixed number of threads: the machine used for the printed numbers was shared with other jobs

class ReplayBuffer:
    """Stores the last `capacity` transitions; sampling a random batch breaks the correlation between consecutive steps."""
    def __init__(self, capacity, obs_dim):
        self.s = np.zeros((capacity, obs_dim), np.float32); self.s2 = np.zeros((capacity, obs_dim), np.float32)
        self.a = np.zeros(capacity, np.int64); self.r = np.zeros(capacity, np.float32); self.done = np.zeros(capacity, np.float32)
        self.capacity, self.size, self.pos = capacity, 0, 0
    def add(self, s, a, r, s2, terminated):
        i = self.pos
        self.s[i], self.a[i], self.r[i], self.s2[i], self.done[i] = s, a, r, s2, float(terminated)
        self.pos = (self.pos + 1) % self.capacity
        self.size = min(self.size + 1, self.capacity)
    def sample(self, batch):
        idx = np.random.randint(0, self.size, batch)
        return (torch.from_numpy(self.s[idx]), torch.from_numpy(self.a[idx]), torch.from_numpy(self.r[idx]),
                torch.from_numpy(self.s2[idx]), torch.from_numpy(self.done[idx]))

class QNetwork(nn.Module):
    """Q(s, .) for ALL actions at once: input = the 4 numbers of the CartPole state, output = one value per action."""
    def __init__(self, obs_dim, n_actions, hidden=128):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(obs_dim, hidden), nn.ReLU(), nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, n_actions))
    def forward(self, x):
        return self.net(x)

def dqn(steps=40_000, use_replay=True, use_target=True, double=False, gamma=0.99, lr=1e-3, batch=64,
        buffer_size=50_000, learn_start=1_000, target_every=500, eps_end=0.05, eps_decay_steps=10_000,
        seed=0, env_factory=None, eval_every=None, eval_fn=None, keep_best=False):
    torch.manual_seed(seed); np.random.seed(seed); random.seed(seed)
    env = env_factory() if env_factory else gym.make("CartPole-v1")
    obs_dim, n_actions = env.observation_space.shape[0], env.action_space.n
    q, q_target = QNetwork(obs_dim, n_actions), QNetwork(obs_dim, n_actions)
    q_target.load_state_dict(q.state_dict())
    opt = torch.optim.Adam(q.parameters(), lr=lr)
    buf = ReplayBuffer(buffer_size, obs_dim)
    env.action_space.seed(seed)                              # seed of action_space.sample(), otherwise the output is not reproducible
    s, _ = env.reset(seed=seed)
    episode_return, episode_returns, checkpoints = 0.0, [], []
    best_score, best_state = -1.0, None
    for t in range(1, steps + 1):
        eps = max(eps_end, 1.0 - (1.0 - eps_end) * t / eps_decay_steps)
        if random.random() < eps:
            a = env.action_space.sample()
        else:
            with torch.no_grad():
                a = int(q(torch.from_numpy(s).float().unsqueeze(0)).argmax())
        s2, r, terminated, truncated, _ = env.step(a)
        buf.add(s, a, r, s2, terminated)                      # `terminated` only: a time-limit truncation is not a real end
        s, episode_return = s2, episode_return + r
        if terminated or truncated:
            episode_returns.append(episode_return); episode_return = 0.0
            s, _ = env.reset()
        if t >= learn_start:
            if use_replay:
                bs, ba, br, bs2, bd = buf.sample(batch)
            else:                                              # no replay: learn only from the LAST transition
                bs, ba, br, bs2, bd = [torch.from_numpy(x[(buf.pos - 1) % buf.capacity][None]) for x in (buf.s, buf.a, buf.r, buf.s2, buf.done)]
            with torch.no_grad():
                net_for_target = q_target if use_target else q
                if double:                                     # Double DQN: the online net CHOOSES the action, the target net EVALUATES it
                    best = q(bs2).argmax(dim=1, keepdim=True)
                    next_q = net_for_target(bs2).gather(1, best).squeeze(1)
                else:
                    next_q = net_for_target(bs2).max(dim=1).values
                target = br + gamma * (1.0 - bd) * next_q
            current = q(bs).gather(1, ba.unsqueeze(1)).squeeze(1)
            loss = F.smooth_l1_loss(current, target)            # Huber loss: gentler than MSE for large errors
            opt.zero_grad(); loss.backward()
            nn.utils.clip_grad_norm_(q.parameters(), 10.0)
            opt.step()
            if use_target and t % target_every == 0:
                q_target.load_state_dict(q.state_dict())        # periodic hard copy of the weights
        if eval_every and t % eval_every == 0 and eval_fn:
            score = eval_fn(q)
            checkpoints.append((t, score))
            if score > best_score:                              # remember the best checkpoint (early stopping on a validation score)
                best_score, best_state = score, copy.deepcopy(q.state_dict())
    if keep_best and best_state is not None:
        q.load_state_dict(best_state)
    return q, episode_returns, checkpoints

def evaluate_greedy(q, episodes=30, seed=10_000, env_factory=None):
    env = env_factory() if env_factory else gym.make("CartPole-v1")
    totals = []
    for i in range(episodes):
        s, _ = env.reset(seed=seed + i); done, total = False, 0.0
        while not done:
            with torch.no_grad():
                a = int(q(torch.from_numpy(s).float().unsqueeze(0)).argmax())
            s, r, terminated, truncated, _ = env.step(a); total += r; done = terminated or truncated
        totals.append(total)
    return float(np.mean(totals))

Walk through the code from the top. ReplayBuffer is a circular archive of five arrays (state, action, reward, next state, end flag): add writes at the current position and advances the pointer with % capacity, overwriting the oldest entries when full; sample draws random indices. It stores float(terminated), not terminated or truncated, for the reason seen above. QNetwork is a perceptron with two hidden layers of 128 units (4 → 128 → 128 → 2) that returns Q(s, left) and Q(s, right) in one go, so to choose the action it is enough to take the argmax of the output.

In the loop, ε falls linearly from 1 to 0.05 over the first 10,000 steps; after learn_start = 1000 steps, to have some data, at every step a minibatch of 64 is drawn and the target is computed inside torch.no_grad(), that is, without gradient: target = r + γ (1 − done) next_q, where (1 − done) cancels the future for terminal transitions. The estimate is q(bs).gather(1, ba), the value of the action actually taken; the loss is smooth_l1_loss; the gradient is clipped to norm 10 before the Adam step, to avoid enormous steps when the error is large. Every target_every steps the online network is copied into the target one.

The switches each change one line. use_replay=False replaces the minibatch with the last transition only; use_target=False computes the target with the online network; double=True makes the online network choose the action and the target network evaluate it, as explained further on. evaluate_greedy is the evaluation protocol of the previous lesson: ε = 0, new episodes with seeds never used in training, mean return. A warning about the keep_best line: it is used later to keep the best checkpoint, and the best checkpoint is chosen on a noisy measurement, so it is a useful criterion but not an innocent one.

The ablation: removing one of the two devices

An ablation removes one piece at a time to see what changes. The four configurations are: full DQN, no target network, no replay buffer, neither of the two. For each, three seeds, 20,000 steps and a greedy evaluation of 20 episodes every 5,000 steps, so four checkpoints per training run and 12 per configuration. The 12 training runs took a time that depends on the load of the machine and I do not report it; the order of magnitude was a few minutes each, on CPU with two threads.

# The ablation: the same training run with replay and/or target network switched off.
# 3 seeds, 20 000 steps, a greedy evaluation (20 episodes, epsilon = 0) every 5 000 steps.
configs = {
    "DQN (replay + target)": dict(use_replay=True,  use_target=True),
    "no target network":     dict(use_replay=True,  use_target=False),
    "no replay buffer":      dict(use_replay=False, use_target=True),
    "neither":               dict(use_replay=False, use_target=False),
}
for name, kwargs in configs.items():
    for seed in range(3):
        q, er, cp = dqn(steps=20_000, seed=seed, eval_every=5_000, eval_fn=lambda q: evaluate_greedy(q, 20), **kwargs)
        sc = [round(c[1]) for c in cp]
        print(f"{name:22s} seed {seed}: {sc}", flush=True)

# DQN (replay + target)  seed 0: [143, 88, 123, 123]
# DQN (replay + target)  seed 1: [225, 276, 168, 143]
# DQN (replay + target)  seed 2: [115, 83, 110, 130]
# no target network      seed 0: [9, 9, 9, 9]
# no target network      seed 1: [9, 9, 9, 9]
# no target network      seed 2: [9, 9, 9, 9]
# no replay buffer       seed 0: [17, 11, 11, 9]
# no replay buffer       seed 1: [50, 16, 62, 10]
# no replay buffer       seed 2: [15, 10, 9, 9]
# neither                seed 0: [9, 9, 9, 9]
# neither                seed 1: [9, 9, 9, 9]
# neither                seed 2: [9, 9, 9, 9]
#
#
# configuration                  mean   median   ckpt <= 15   ckpt >= 400
# DQN (replay + target)         143.9    126.5         0.0%      0.0%
# no target network               9.0      9.0       100.0%      0.0%
# no replay buffer               19.1     11.0        66.7%      0.0%
# neither                         9.0      9.0       100.0%      0.0%

The code loops over the configurations and prints, for each seed, the mean return of the greedy policy at the four checkpoints; the table at the end is not printed by the listing: I computed it separately, with a short script, from the same 12 evaluations per configuration. The reading is clear-cut, at least for the extreme case. Without the target network the return is 9 at every checkpoint of every seed, and the same holds for the configuration that removes both. The full DQN never collapses (no checkpoint ≤ 15) but does not reach 500: the maximum observed is 276 and the mean 144. Without replay it sits between the two: mean 19, and 66.7% of the checkpoints are ≤ 15.

What does 9 mean? I checked in the case without target network, seed 0, with a separate run of 20,000 steps: the greedy policy always chooses action 1 (push right) and the pole falls after 9 steps. The Q values estimated in the four situations I tried are of the order of 5 · 10⁷, whereas the maximum possible discounted return with γ = 0.99 over 500 steps is (1 − 0.99⁵⁰⁰)/0.01 ≈ 99.3. It is a true divergence, not a slightly wrong value: the target was chasing itself, the values exploded and the two outputs became almost equal. The check concerns a single seed: in the other two I see the same return of 9, but I did not inspect their networks. For the case without replay I did not measure the cause.

Practical conclusion: in the configuration tried, the target network is the difference between an agent that does not learn and one that does, and replay is the difference between an agent that learns very little and one that learns. But the full DQN you have just seen is anything but a solved agent: mean return 144 out of a maximum of 500, with values that oscillate from one checkpoint to the next. The point is that the absence of collapse is not convergence, and the next experiment serves to see it.

Overestimation and Double DQN

There is a known statistical flaw in the target r + γ max Q(s′, ·): the maximum of noisy estimates is on average larger than the maximum of the true values. If all actions are worth 0 but the estimates are noisy, the highest of the estimates is positive. It is best measured in a case where the truth is known: Gaussian estimates with standard deviation 1 around 0, for 2, 5 and 10 actions, 200,000 trials each. The remedy proposed by van Hasselt and colleagues in 2015 (Double Q-learning applied to DQN) is to choose the action with one estimate and evaluate it with another, independent one.

import numpy as np
rng = np.random.default_rng(0)
# All actions are EQUALLY good (true value 0), but our estimates are noisy (standard deviation 1).
for n_actions in (2, 5, 10):
    noisy = rng.normal(0, 1, (200_000, n_actions))                  # the estimates Q(s', a') for each action
    other = rng.normal(0, 1, (200_000, n_actions))                  # an INDEPENDENT second set of estimates
    max_estimate = noisy.max(axis=1).mean()                          # what DQN uses:  max_a Q(s', a)
    double_estimate = other[np.arange(len(other)), noisy.argmax(axis=1)].mean()   # Double DQN: pick with one set, evaluate with the other
    print(f"{n_actions:2d} actions: E[max of noisy estimates] = {max_estimate:5.2f}   |   pick with one, evaluate with the other = {double_estimate:5.2f}   (truth: 0)")

#  2 actions: E[max of noisy estimates] =  0.57   |   pick with one, evaluate with the other = -0.00   (truth: 0)
#  5 actions: E[max of noisy estimates] =  1.16   |   pick with one, evaluate with the other = -0.00   (truth: 0)
# 10 actions: E[max of noisy estimates] =  1.54   |   pick with one, evaluate with the other = -0.00   (truth: 0)

The bias of the maximum grows with the number of actions (0.57, 1.16, 1.54) and the "choose with one, evaluate with the other" method cancels it (−0.00, that is, zero within sampling noise). In DQN the two estimates are the two networks already present: the online network chooses a* = argmax Q(s′, ·; θ) and the target network evaluates its value Q(s′, a*; θ⁻), and this is the double=True branch of the code. The estimates of the online and target networks are not independent as in the simulation (the target is an old copy of the online one), so the real effect is probably weaker than the ideal one. The abstract of van Hasselt's paper states that DQN suffers from substantial overestimations in some Atari games: here we check it on CartPole.

Instability and overestimation on CartPole

The listing defines overestimation, which compares, over 30 episodes, the value Q(s0, a*) predicted by the network for the initial state with the discounted return actually obtained by following the greedy policy, and then applies it to two configurations with learning rate 5e-4: standard DQN and Double DQN, three seeds each. The step halved with respect to 10⁻³ is the one that the previous version of this lesson indicated as more stable: here we check whether it holds.

# Does the network OVERESTIMATE?  Compare what it predicts for the start state with what the greedy policy really collects.
@torch.no_grad()
def overestimation(q, episodes=30, gamma=0.99, seed=20_000):
    env = gym.make("CartPole-v1")
    predicted, realised = [], []
    for i in range(episodes):
        s, _ = env.reset(seed=seed + i)
        predicted.append(q(torch.from_numpy(s).float().unsqueeze(0)).max().item())   # Q(s0, best action)
        rewards, done = [], False
        while not done:
            a = int(q(torch.from_numpy(s).float().unsqueeze(0)).argmax())
            s, r, terminated, truncated, _ = env.step(a); rewards.append(r); done = terminated or truncated
        G = 0.0
        for r in reversed(rewards):
            G = r + gamma * G                                  # the discounted return that was REALLY obtained
        realised.append(G)
    return np.mean(predicted), np.mean(realised)

configs = {
    "lr 5e-4":             dict(lr=5e-4),
    "lr 5e-4, Double DQN": dict(lr=5e-4, double=True),
}
for name, kwargs in configs.items():
    for seed in range(3):
        q, er, cp = dqn(steps=20_000, seed=seed, eval_every=5_000, eval_fn=lambda q: evaluate_greedy(q, 20), **kwargs)
        sc = [round(c[1]) for c in cp]
        print(f"{name:22s} seed {seed}: {sc}", flush=True)
        p, r = overestimation(q)
        print(f"   overestimation {name} seed {seed}: predicted {p:.1f} realised {r:.1f}", flush=True)

# lr 5e-4                seed 0: [239, 184, 129, 141]
#    overestimation lr 5e-4 seed 0: predicted 33.7 realised 75.5
# lr 5e-4                seed 1: [219, 117, 125, 104]
#    overestimation lr 5e-4 seed 1: predicted 34.5 realised 64.6
# lr 5e-4                seed 2: [189, 145, 97, 99]
#    overestimation lr 5e-4 seed 2: predicted 34.1 realised 63.1
# lr 5e-4, Double DQN    seed 0: [173, 443, 152, 106]
#    overestimation lr 5e-4, Double DQN seed 0: predicted 32.8 realised 65.4
# lr 5e-4, Double DQN    seed 1: [120, 90, 106, 109]
#    overestimation lr 5e-4, Double DQN seed 1: predicted 33.4 realised 66.3
# lr 5e-4, Double DQN    seed 2: [197, 115, 103, 109]
#    overestimation lr 5e-4, Double DQN seed 2: predicted 34.0 realised 67.4

First reading, honest: overestimation does not appear. The predicted value for the initial state is about 34 in all six networks, whereas the discounted return actually obtained is 63-76 (mean 68 for standard DQN, 66 for Double DQN). The network underestimates, probably because at 20,000 steps it has not yet extended the values towards the long episodes (I did not verify this explanation); the bias of the maximum is real (you measured it above) but here it is dominated by other errors. And the two variants cannot be told apart: I have not shown that Double DQN is needed in this problem, nor that it is not needed on harder problems or longer training runs, where the literature documents it. It costs one line and did no harm: a reasonable choice as a default, not a result of this measurement.

Second reading, on stability. The mean of the 12 checkpoints is 149 for standard DQN and 152 for Double DQN, against 144 with the 10⁻³ step of the ablation: differences well within the noise of three seeds, so I cannot say that halving the step helps, as the previous version claimed. What is seen in all three configurations, however, is that the first checkpoint (5,000 steps) is often the best: mean of the first checkpoint 161 (step 10⁻³), 216 (step 5e-4), 163 (Double) against 132, 115 and 108 for the last. Double DQN with seed 0 also shows the typical jump: return 173, then 443, then 152 and 106, with no parameter changed between one checkpoint and the next.

Why does the policy get worse while it keeps training? A plausible explanation, which I did not verify by isolating the buffer: a better policy fills the buffer with long, easy episodes, the cases where the pole falls leave the buffer, the network forgets them and the next policy gets worse. The mechanism is that of the deadly triad cited above, in a mild form: no explosive divergence, but no monotonic improvement either. This is why in practice one evaluates at intervals and keeps the best checkpoint instead of the last, trains with more seeds and reports mean and dispersion. I did not try here the "soft" copy of the target network (θ⁻ ← τθ + (1 − τ)θ⁻) nor prioritized replay: they are remedies cited, not measured.

The different simulator: overfitting and domain randomisation

Let us close with the warning of the lesson on Gymnasium and overfitting, now with a network. Does a DQN trained on CartPole with the standard constants (half pole length 0.5, cart mass 1.0, gravity 9.8) still work if the physics changes? Two kinds of agent are trained, three seeds each: (a) nominal physics only; (b) physics randomised at every episode, with pole length between 0.25 and 1.0 and cart mass between 0.5 and 2.0. For each, the best checkpoint is chosen on the training regime (keep_best=True) and it is evaluated in five conditions, one of which was never randomised: gravity 15.

class Physics(gym.Wrapper):
    """Change the physics of CartPole. `sampler` is called at every reset and returns a dict of parameters."""
    def __init__(self, env, sampler):
        super().__init__(env); self.sampler = sampler
    def reset(self, **kwargs):
        u = self.env.unwrapped
        p = self.sampler()
        u.length, u.masscart, u.gravity = p.get("length", 0.5), p.get("masscart", 1.0), p.get("gravity", 9.8)
        u.total_mass = u.masspole + u.masscart                    # derived quantities must be kept consistent
        u.polemass_length = u.masspole * u.length
        return self.env.reset(**kwargs)

nominal = lambda: gym.make("CartPole-v1")
def fixed(**params):
    return lambda: Physics(gym.make("CartPole-v1"), lambda: params)
def randomised():
    rng = np.random.default_rng(123)
    return lambda: Physics(gym.make("CartPole-v1"), lambda: dict(length=rng.uniform(0.25, 1.0), masscart=rng.uniform(0.5, 2.0)))

tests = {
    "nominal":              fixed(),
    "short pole (0.25)":    fixed(length=0.25),
    "long pole (1.0)":      fixed(length=1.0),
    "heavy cart (3.0)":     fixed(masscart=3.0),
    "gravity 15 (unseen kind)": fixed(gravity=15.0),
}
agents = {"nominal": nominal, "randomised": randomised()}

def run_regime(regime, seeds=3):
    factory = agents[regime]
    table = {name: [] for name in tests}
    for seed in range(seeds):
        q, _, _ = dqn(steps=20_000, seed=seed, lr=5e-4, env_factory=factory, eval_every=5_000,
                      eval_fn=lambda q: evaluate_greedy(q, 20, env_factory=factory), keep_best=True)   # best checkpoint on the TRAINING regime
        for name, tf in tests.items():
            table[name].append(evaluate_greedy(q, 30, env_factory=tf))
    return {name: float(np.mean(v)) for name, v in table.items()}

for regime in ('nominal', 'randomised'):
    print(regime, {k: round(v, 1) for k, v in run_regime(regime).items()})

# nominal {'nominal': 203.8, 'short pole (0.25)': 213.1, 'long pole (1.0)': 45.1, 'heavy cart (3.0)': 195.0, 'gravity 15 (unseen kind)': 186.9}
# randomised {'nominal': 336.6, 'short pole (0.25)': 387.9, 'long pole (1.0)': 270.6, 'heavy cart (3.0)': 260.9, 'gravity 15 (unseen kind)': 278.4}

The Physics class is a Gymnasium wrapper that at every reset asks a function for the new parameters and writes them into the environment, also updating the derived quantities total_mass and polemass_length: if you did not, the physics would be inconsistent. Mind the generator of the randomisation: randomised() uses a default_rng(123) shared among all the episodes of that regime, so the parameters drawn depend on the order of the calls and remain reproducible only if you re-run the whole listing. Each cell of the table is the mean of 30 episodes per seed and then of 3 seeds.

Reading. The agent trained on nominal physics scores 204 on the nominal one and 213 with the short pole, but 45 with the long pole (−78%), 195 with the heavy cart and 187 with gravity 15. The randomised agent does better in all five conditions: 337 on the nominal one, 388 with the short pole, 271 with the long pole, 261 with the heavy cart, 278 with the never-seen gravity. This is the opposite of what the previous version of this lesson reported (40,000 steps, three seeds), where randomisation worsened the nominal case without giving robustness. The two results cannot be reconciled, and I have no experiment that says which is more credible: with three seeds and checkpoints chosen on 20 evaluation episodes the variance is large, and a mean of three training runs can be dragged by a single lucky agent.

What this measurement supports with reasonable confidence is more modest: the nominal agent is much more fragile when the pole gets longer (45 against 204 on the nominal one), that is, there is overfitting to the simulator, visible even with a network; and at this budget randomisation did not worsen the agent, because it improved all the conditions measured. It does not follow that randomising is always advantageous: it means that at 20,000 steps, with these seeds, I did not find a price to pay. The defence that stays valid in both versions is to evaluate the agent on held-out variants, never used to train it.

Trade-offs and traps

Some choices have no right value. A large replay buffer offers more varied but older samples, referring to past policies; a small one is more recent but more correlated. A target network copied rarely gives a stable target and slow learning; copied often it is more reactive and more unstable, and in this lesson I did not vary it. A single environment is fast but fragile; many environments give robustness in theory and cost more. Finally, DQN is not the only family: policy-gradient methods (REINFORCE, actor-critic) learn the policy directly and handle continuous actions better, but we do not cover them here.

The traps this module leaves you with are seven, and it is worth having them as a checklist:

1. Evaluating with the last checkpoint. The data above show the first checkpoint often better than the last (216 against 115 for a whole configuration).
2. A single seed. With three the variance is already enormous; with one, any conclusion is an anecdote. A result opposite to the one previously published shows it.
3. Not seeding action_space.sample(). Outputs are not repeatable and you do not notice until you re-run.
4. Zeroing the bootstrap on a truncation. An episode interrupted by the 500-step limit has not really ended: store only terminated.
5. Forgetting torch.no_grad() in the target, or differentiating through y: the idea of semi-gradient breaks.
6. Not normalising the inputs. CartPole is scaled well enough to work like this; other environments, with components on very different scales, are not.
7. Mistaking the low training ε for an evaluation. Evaluation is something else: ε = 0, new seeds.

If I have to choose the two that weigh most, they are the first and the second: in deep RL the evaluation noise is so high that a result without several seeds and without a rule for choosing the checkpoint does not count as a measurement. The other five are code errors that produce symptoms far from the cause (a return stuck at 9, a curve that does not go down), which is why it is worth putting them in a checklist before blaming the algorithm.

Exercises with solutions

1. Why is the target network used in the target and not the online network? Because every step changes the online weights and with them the target: the network chases a goal that moves. The frozen copy keeps it still between one copy and the next.
2. What is the replay buffer for? To break the correlation between consecutive samples, training on random batches of transitions from different moments, and to reuse each experience several times.
3. In the ablation, what does the constant return of 9 indicate? That the greedy policy always chooses the same action and the pole falls in 9 steps; in the controlled case the Q values had exploded to about 5 · 10⁷.
4. With 2 actions of true value 0 and estimates N(0, 1), what is the mean of the maximum? About 0.57, as in the simulation: positive, even though the truth is 0.

The exercises check three ideas that are worth being able to say in words before opening the code: why two devices are needed and not one, what a constant value in a learning curve means, and why a maximum of noisy numbers is not the maximum of the true numbers. If the fourth answer surprises you, re-run the simulation listing changing the number of actions and watch how the bias grows.

Frequently asked questions

Why is it not enough to train the network on the last transition?

Because consecutive transitions are very similar and the gradient computed on a single one of them is a poor estimator. In the ablation, without replay 66.7% of the checkpoints have a return ≤ 15 and the maximum observed is 62.

Is Double DQN always better than DQN?

I have not shown it here: with three seeds and 20,000 steps the mean returns (152 against 149) cannot be told apart and overestimation does not appear in either. It corrects a real bias of the maximum, measured in the simulation, at the cost of one line.

How many seeds do you need to trust a result?

More than three. Here three seeds have already given a result on randomisation opposite to that of an earlier, longer trial. To compare two configurations you need enough seeds to be able to say whether the difference exceeds the dispersion between seeds.

What you can do now

You can build a DQN on CartPole with replay buffer, target network, Huber loss and optional Double DQN, fixing all its seeds so that the output is repeatable, and you can use the four switches to run an ablation. You have seen that without the target network the Q values explode (the agent always chooses the same action and stays at 9), that without replay learning is almost nil, and that with both the agent learns but oscillates, with the first checkpoint often better than the last. You have measured the bias of the maximum in a controlled case and not found it in a real short-trained network, and you have seen an agent trained on a single physics lose 78% of its return with the long pole.

The path of the module ends here: the Markov process is the language of the problem, Bellman is the law the value must obey, Q-Learning solves it without a model, DQN solves it when the states are too many. Each step removes a limit of the previous one and adds one of its own: in RL the real work is in the measurements.

In the series, the next one is What an AI Agent Actually Is (and When You Don't Need One).