63 - RNN, LSTM and GRU Compared: A Controlled Experiment Over Several Seeds

"The LSTM is better than the plain RNN" is a sentence you read everywhere. In this lesson we put it to the test with a controlled experiment on a task that requires real memory, using three seeds for each configuration, and the result is not what a textbook would lead you to expect: under some conditions the LSTM, with the standard initialisation, does worse than the plain RNN. The reason can be understood, and it is instructive, because it takes us back to the lessons on gradient and gates. The second goal is methodological: learning to compare models without fooling yourself.

The words of this lesson

The seed is the number that initialises the random number generator: changing it changes the initial weights and the order of the data, and two trainings that are identical except for the seed can give very different results. The variance between seeds measures how different: if it is large, a single number tells you nothing reliable about the model. A hyperparameter is a choice that is not learned but decided by you beforehand (state size, learning rate, number of steps). The chance level is the accuracy of someone guessing: with 8 equiprobable classes it is 1/8 = 0.125, and a result close to that value means "the network has learned nothing". Finally a fair comparison uses the same data, the same training budget, the same metrics and the same amount of tuning for every model.

The analogy: three students, same test, different days

If you want to know which of three students is best at remembering, one test each is not enough: one of them might have slept badly. Repeat the test on different days and look at the distribution of the scores. The seeds are the "different days" of your model. A model that wins once out of three and scores zero the other two times is not "the best": it is unstable.

The task: remembering the first symbol

The network reads a sequence of T symbols (integers from 0 to 9). The first symbol is always a "class" from 0 to 7; the following T − 1 are noise. At the end the network must say what the first one was. There is no shortcut: the information is only in the first position and must survive all the noise. For each symbol the network first derives a vector (an nn.Embedding layer, which associates with every integer a vector of 16 learned numbers) and then passes it to the recurrent part.

import torch
import torch.nn as nn

VOCAB, CLASSES = 10, 8

def recall_batch(batch, T):
    """Sequence of random tokens. The label is the FIRST token (always in 0..7).
    The network must carry it through T-1 distracting steps."""
    x = torch.randint(0, VOCAB, (batch, T))
    x[:, 0] = torch.randint(0, CLASSES, (batch,))
    return x, x[:, 0].clone()

class Classifier(nn.Module):
    def __init__(self, kind, hidden=64):
        super().__init__()
        self.emb = nn.Embedding(VOCAB, 16)
        self.rec = {"RNN": nn.RNN, "LSTM": nn.LSTM, "GRU": nn.GRU}[kind](16, hidden, batch_first=True)
        self.head = nn.Linear(hidden, CLASSES)
    def forward(self, x):
        out, _ = self.rec(self.emb(x))
        return self.head(out[:, -1])

def train_and_eval(kind, T, seed, steps=3000):
    torch.manual_seed(seed)
    net = Classifier(kind)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for _ in range(steps):
        x, y = recall_batch(64, T)
        loss = nn.functional.cross_entropy(net(x), y)
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(net.parameters(), 1.0)
        opt.step()
    with torch.no_grad():
        x, y = recall_batch(2000, T)
        return (net(x).argmax(-1) == y).float().mean().item()

for T in (10, 20, 40):
    for kind in ("RNN", "LSTM", "GRU"):
        accs = [train_and_eval(kind, T, seed) for seed in range(3)]
        print(f"T={T:2d} {kind:5s} accuracy per seed: {[round(a, 2) for a in accs]}")
print("chance level:", 1 / CLASSES)

Explaining the code. recall_batch generates the data on the fly: random symbols, with the first position overwritten by a class 0-7; the label is exactly that. Classifier accepts the cell type as a string ("RNN", "LSTM", "GRU") and keeps everything else identical: same embedding size, same state size (64), same linear head. This is the heart of the fair comparison: only one thing changes. train_and_eval trains for 3000 steps with Adam and clipping, the same for everyone, and measures the accuracy on 2000 new sequences. The double loop at the bottom tries three lengths (T = 10, 20, 40) and three seeds per cell. No cell gets preferential treatment: the LSTM here uses PyTorch's default initialisation (forget-gate bias at 0), which is also what you find in a lot of real code.

Measured results

Accuracy on 2000 new sequences, one value per seed (0, 1, 2). Chance level: 0.125.

             seed 0   seed 1   seed 2
T=10  RNN     0.62     0.87     1.00
T=10  LSTM    1.00     1.00     1.00
T=10  GRU     1.00     1.00     1.00

T=20  RNN     1.00     0.51     0.23
T=20  LSTM    0.14     0.12     0.13
T=20  GRU     1.00     1.00     1.00

T=40  RNN     0.13     0.12     0.90
T=40  LSTM    0.12     0.12     0.14
T=40  GRU     0.11     0.11     0.13

What you see, without embellishment. At T=10 the LSTM and the GRU always solve it (3 seeds out of 3), while the plain RNN works irregularly (0.62, 0.87, 1.00). At T=20 the GRU still solves it 3 times out of 3; the plain RNN wins in one seed (1.00), then gets to half (0.51) and to a quarter (0.23), so one time out of three it is perfect, but you cannot count on it; the LSTM with the standard initialisation stays at chance level in all three seeds. At T=40 no cell solves it reliably in 3000 steps: one seed of the plain RNN reaches 0.90, the other two do not, and the GRU and the LSTM stay at chance.

The surprising datum is the LSTM at T=20: the cell considered "the solution" to the memory problem is the one that fails most clearly. It is not a defect of the architecture, and the next paragraph shows why.

Why did the LSTM stay at chance at T=20?

The plausible explanation goes through the forget gate. In PyTorch all the weights and biases of an LSTM start from a uniform distribution between −1/√H and +1/√H (with H = 64: ±0.125), so at the beginning the forget-gate bias is close to zero and the gate is about f = σ(0) = 0.5. After 20 steps the factor multiplying the memory is about 0.5²⁰ ≈ 10⁻⁶: the signal of the first symbol is gone before training has a chance to teach the gate to "keep". This is a hypothesis, not a proof: the GRU starts from a more favourable condition on this task, but its structure is different and here the cause has not been isolated. The hypothesis about the LSTM can, however, be put to the test: exactly the same experiment, with the forget-gate bias raised to 1, the remedy proposed by Jozefowicz and colleagues (see the sources).

def lstm_forget_bias(net, value=1.0):
    h = net.rec.hidden_size
    with torch.no_grad():
        net.rec.bias_ih_l0[h:2 * h] = value       # bias layout [i | f | g | o]

def train_and_eval_fb(T, seed, steps=3000, forget_bias=1.0):
    torch.manual_seed(seed)
    net = Classifier("LSTM")
    lstm_forget_bias(net, forget_bias)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for _ in range(steps):
        x, y = recall_batch(64, T)
        loss = nn.functional.cross_entropy(net(x), y)
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(net.parameters(), 1.0)
        opt.step()
    with torch.no_grad():
        x, y = recall_batch(2000, T)
        return (net(x).argmax(-1) == y).float().mean().item()

for T in (20, 40):
    accs = [train_and_eval_fb(T, seed) for seed in range(3)]
    print(f"T={T:2d} LSTM (forget bias 1) accuracy per seed: {[round(a, 2) for a in accs]}")

T=20 LSTM (forget bias 1) accuracy per seed: [1.0, 0.25, 1.0]
T=40 LSTM (forget bias 1) accuracy per seed: [0.12, 0.12, 0.13]

At T=20 with bias 1 the LSTM goes from "chance in all three seeds" to "perfect in two seeds out of three" (the third reaches 0.25: it has started to learn, it has not finished). A single hyperparameter, a single line of code, made the difference. At T=40 nothing changes within 3000 steps: for longer lengths you need more steps or other measures. The code of lstm_forget_bias uses the same [hidden : 2*hidden] slice as the lesson on the LSTM, because in PyTorch's layout the bias vector is [i | f | g | o].

What can and cannot be concluded

We can conclude that, on this task and with this budget, initialisation matters as much as architecture: with everything else equal, "LSTM or GRU" gave different results, and a single line of code changed the outcome of the LSTM. We cannot conclude that the GRU is "better" than the LSTM, because with bias 1 the LSTM recovered in two seeds out of three, and with more steps other configurations would change. Three seeds are enough to see that the variance is large, not to estimate a stable mean. Nor can we say that the plain RNN is good: it reached full accuracy in two of the six trainings at T=10 and T=20, and in four out of six it did not, so it is unstable.

The useful message is this: every comparison between architectures is also a comparison between their initialisations, their learning rates and their budgets. If you do not control them, you are measuring something else.

Summarising with mean, standard deviation and "solved out of three"

A table of nine rows of three numbers is readable but cannot be compared at a glance. To summarise honestly you need two decisions taken before looking at the results: which threshold means "solved" (here accuracy ≥ 95%) and which statistics to report. The code below takes the numbers from the table and computes the mean, the standard deviation (with ddof=1, the correct formula for a small sample), the minimum and how many seeds out of three exceed the threshold.

import numpy as np

# Results of the recall experiment, copied from the table above (accuracy per seed).
results = {
    ("RNN", 10): [0.62, 0.87, 1.00],  ("LSTM", 10): [1.00, 1.00, 1.00],  ("GRU", 10): [1.00, 1.00, 1.00],
    ("RNN", 20): [1.00, 0.51, 0.23],  ("LSTM", 20): [0.14, 0.12, 0.13],  ("GRU", 20): [1.00, 1.00, 1.00],
    ("RNN", 40): [0.13, 0.12, 0.90],  ("LSTM", 40): [0.12, 0.12, 0.14],  ("GRU", 40): [0.11, 0.11, 0.13],
}

SOLVED = 0.95        # decide BEFORE looking: "solved" means accuracy >= 95%
print(f"{'cell':5s} {'T':>3s}  {'mean':>5s}  {'std':>5s}  {'min':>5s}  solved/seeds")
for (cell, T), accs in results.items():
    a = np.array(accs)
    print(f"{cell:5s} {T:3d}  {a.mean():5.2f}  {a.std(ddof=1):5.2f}  {a.min():5.2f}  {(a >= SOLVED).sum()}/{len(a)}")

cell    T   mean    std    min  solved/seeds
RNN    10   0.83   0.19   0.62  1/3
LSTM   10   1.00   0.00   1.00  3/3
GRU    10   1.00   0.00   1.00  3/3
RNN    20   0.58   0.39   0.23  1/3
LSTM   20   0.13   0.01   0.12  0/3
GRU    20   1.00   0.00   1.00  3/3
RNN    40   0.38   0.45   0.12  0/3
LSTM   40   0.13   0.01   0.12  0/3
GRU    40   0.12   0.01   0.11  0/3

The most revealing column is the standard deviation. The plain RNN at T = 20 has mean 0.58 but deviation 0.39: a mean that describes no real seed (the three values were 1.00, 0.51, 0.23). When the deviation is comparable to the mean, the mean is almost useless and the "solved out of three" column says more: 1 out of 3. The ddof=1 is needed because with three seeds the deviation computed by dividing by n would underestimate the true one (you divide by n − 1, Bessel's correction).

A second test bench: the adding problem, with steps to solve

A result on a single task can be a fluke. Let us redo the comparison on the adding problem (the definition of adding_batch is repeated in the block, so the listing is self-contained), which has a different structure (continuous values, two markers, regression). The metric is "steps to solve" (threshold 0.05, a third of the baseline), with three seeds and a maximum budget of 5000 steps; None means "not solved within the budget". The LSTM has the forget-gate bias at 1: we treat it in the most favourable way possible.

def adding_batch(batch, T):
    """Adding problem: channel 0 = random values in [0,1], channel 1 = two markers; target = sum of the two marked values."""
    values = torch.rand(batch, T)
    markers = torch.zeros(batch, T)
    for b in range(batch):
        markers[b, torch.randperm(T)[:2]] = 1.0
    return torch.stack([values, markers], dim=-1), (values * markers).sum(dim=1)

class AnyNet3(nn.Module):
    def __init__(self, kind, hidden=64):
        super().__init__()
        self.rec = {"RNN": nn.RNN, "GRU": nn.GRU, "LSTM": nn.LSTM}[kind](2, hidden, batch_first=True)
        self.head = nn.Linear(hidden, 1)
        if kind == "LSTM":
            with torch.no_grad():
                self.rec.bias_ih_l0[hidden:2 * hidden] = 1.0       # fair to the LSTM: forget bias 1
    def forward(self, x):
        out, _ = self.rec(x)
        return self.head(out[:, -1]).squeeze(-1)

def adding_steps(kind, T, seed, max_steps=5000, threshold=0.05):
    torch.manual_seed(seed)
    net = AnyNet3(kind)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    recent = []
    for step in range(1, max_steps + 1):
        x, y = adding_batch(64, T)
        loss = nn.functional.mse_loss(net(x), y)
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(net.parameters(), 1.0)
        opt.step()
        recent.append(loss.item()); recent = recent[-100:]
        if step >= 100 and sum(recent) / 100 < threshold:
            return step
    return None                                  # not solved within the budget

for T in (30, 100):
    for kind in ("RNN", "GRU", "LSTM"):
        print(f"T={T:3d} {kind:4s} steps to solve (3 seeds): {[adding_steps(kind, T, s) for s in range(3)]}")

T= 30 RNN  steps to solve (3 seeds): [2082, 1245, None]
T= 30 GRU  steps to solve (3 seeds): [335, 315, 341]
T= 30 LSTM steps to solve (3 seeds): [776, 651, 580]
T=100 RNN  steps to solve (3 seeds): [None, None, None]
T=100 GRU  steps to solve (3 seeds): [556, 556, 575]
T=100 LSTM steps to solve (3 seeds): [1800, 1524, 1423]

Reading. AnyNet3 builds the network (RNN, GRU or LSTM) with the same state size and the same head; adding_steps trains until the moving average of the loss drops below the threshold and returns the step, or None. The plain RNN manages it at T = 30 in only two seeds out of three, and slowly (1245 and 2082 steps), and at T = 100 it never manages it within 5000 steps. The GRU always solves it and with minimal variability: 315-341 steps at T = 30 and 556-575 at T = 100. The LSTM always solves it but with about two to three times the steps of the GRU (580-776 at T = 30, 1423-1800 at T = 100). This picture differs from that of the recall task, where the LSTM with bias 1 had recovered: there the LSTM was still competitive at T = 20. Two tasks, two pictures that are not identical: this is exactly why a single benchmark is not enough.

What the two experiments say together

The GRU was the most reliable cell overall: it won at T=20 in the recall task and in all the adding tests, with low variability between seeds. It is not always the best, though: at T=40 in the recall task it stayed at chance, while one seed of the plain RNN reached 0.90. This is a fact about our experiments, not a law. The LSTM needs care (forget-gate bias, more steps) to give its best: with the care it always solves, without it sometimes it does not. The plain RNN is unreliable beyond short dependencies, so much so that at T = 100 it never solves the adding problem. One limit remains that has to be stated: all the tasks here are synthetic and pure memory, so we have not measured generalisation on real data. On language tasks or on series with a complex structure the order may change, and that is why the advice remains "start from the GRU for speed, compare with the LSTM if needed, measure on several seeds".

How to choose: an operational criterion

The diagram summarises three lessons. The first question concerns legitimacy: if at prediction time you do not have the future, bidirectionality is forbidden. The second concerns the reach of the memory. In the middle (tens of steps) the choice between GRU and LSTM is not made with a rule: you try both with the same budget, on several seeds, and you look at stability too, not only at the mean.

A checklist for honest comparisons

An honest comparison starts from the same budget (steps, epochs, learning rate) for everyone, or from tuning of equal effort for each model. Use several seeds, at least 3 and better 5 or more, and report all the values or the mean and standard deviation, not just the best one. Write an explicit baseline (chance, persistence, a trivial model) before looking at the results and evaluate on new data: here they are generated on the fly, so leakage is not possible. Change one thing at a time, otherwise you do not know what to attribute the difference to. Finally look at the cost: in the recurrent layer, with state 64 and input 16, the parameters are 5,248 for the RNN, 15,744 for the GRU and 20,992 for the LSTM (values obtained from sum(p.numel() for p in rec.parameters()), checked in the run), while the time per step depends on the implementation and on your hardware and has not been compared here.

Real traps

The most common is publishing the lucky seed: you try ten seeds and report the best. You decide beforehand how many seeds to use. The second is concluding from "it does not learn" that the model is unsuitable, whereas as we have seen it can be a problem of initialisation or of steps. The third is comparing models with very different parameter counts without saying so: the LSTM has four times the parameters of the RNN, so at equal hidden_size it is not the same capacity budget. The last is using a task that does not require memory: if the last few steps are enough, all models "win" and the comparison does not measure what you want to know.

A worked example by hand: why three seeds prove nothing

Let us take the three values of the RNN at T = 20 again: 1.00, 0.51, 0.23. The mean is (1.00 + 0.51 + 0.23)/3 = 0.58. The deviations from the mean are +0.42, −0.07, −0.35; their squares 0.176, 0.005, 0.123, whose sum is 0.304. Dividing by n − 1 = 2 gives the sample variance 0.152 and the standard deviation 0.39 (as in the table). The standard error of the mean, that is the uncertainty on the estimated mean, is 0.39 / √3 = 0.225: an approximate interval for the "true" mean accuracy is 0.58 ± 0.45 (two standard errors), that is from 0.13 to 1.03: practically the whole possible space. With only three samples the correct multiplier is not 2 but 4.30 (the value of Student's t distribution with 2 degrees of freedom, computed with scipy.stats.t.ppf(0.975, 2)), and the interval becomes even wider. With three seeds you can say almost nothing about the mean of the RNN at T = 20, and this is the important information: the deviation is high, so more seeds are needed to tighten the estimate.

How many? The standard error scales as 1/√n: to halve it you need four times the seeds. From 3 to 12 seeds the standard error goes from 0.225 to about 0.11 (at equal deviation). For an RNN this unstable it is worth reporting the number of successful seeds as well ("1 out of 3"), which does not depend on mean estimates.

Frequently asked questions

Why not always use 100 seeds? Because every training costs something; you choose the number of seeds according to the cost and the observed variance. For diagnostic experiments like these 3-5 are enough to see large differences; for fine comparisons many more are needed.

Can I compare models on the same set of seeds? Yes, and it is advisable: at equal seed the data and the random initialisations are (partly) shared and the comparisons are more "paired". Be careful, though, that the same seed does not produce the same random numbers in different architectures.

Is a statistical test useful? With few seeds and non-normal distributions, a classical test is weak. It is better to show all the values (as in the table) and use criteria such as "solved out of n".

Is variance between seeds a defect of the model? It is information about the model: an unstable model (RNN: 1.00, 0.51, 0.23) is less reliable in production than a stable one (GRU: 1.00, 1.00, 1.00), at equal mean.

How do I choose the "solved" threshold? Before looking at the results, according to the use. Here 95% accuracy, but for a safety system it could be 99.9%. Changing the threshold after seeing the data is a form of self-deception.

Does a fair comparison require the same number of parameters? It depends on the question: "at equal hidden_size" or "at equal parameters". They are two different questions; here we answered the first (same hidden_size, so the LSTM has more parameters) and not the second.

How to re-check the calculations

To fix the concepts, recompute by hand the sample standard deviation (ddof = 1) of 0.9, 1.0, 0.2: the mean is 0.70, the deviations are 0.2, 0.3, −0.5, the squares 0.04, 0.09, 0.25, the sum 0.38, divided by 2 gives 0.19 and the root is 0.436 (the value coincides with np.std(..., ddof=1)). If with 3 seeds the standard error of the mean is 0.25, to bring it to 0.125 you need 4 times the seeds, that is 12. And if a cell solves 2 seeds out of 3, the exact 95% interval (Clopper-Pearson, scipy.stats.binomtest(2, 3).proportion_ci()) goes from about 0.09 to 0.99: with three seeds the reliability of a cell is almost undetermined, and more seeds are needed before trusting it.

What you can do now

You can set up a fair comparison between RNN, LSTM and GRU (same data, same state, same budget, several seeds), report the results with mean, deviation and "solved out of three", and recognise when a model "does not learn" only because it starts badly. On a pure memory task, at length 20, the GRU solved it in all three seeds, the LSTM with bias 1 in two out of three, the plain RNN in one out of three and the LSTM with standard initialisation in none; at length 40 no cell solved it within 3000 steps. The result you take away is not which cell wins, but that initialisation, budget and seeds move the results as much as the architecture. In the next lesson we apply these cells to a generative task: producing text character by character.

In the series, the previous lesson is Bidirectional RNNs: Reading the Sequence Both Ways; the next one is Generating Text Character by Character with an LSTM.