56 - Recurrent Neural Networks (RNN): Giving the Network a Memory

A recurrent neural network (RNN) is a network that reads a sequence one element at a time and carries a small vector with it, the hidden state, which summarises what it has read so far. At every step it combines the new element with the previous state and produces the next state, always using the same weights. That is why an RNN handles sequences of any length with a fixed number of parameters, and why it is the starting point of this module: LSTM, GRU and bidirectional networks are variants of the same idea. In this article we build one with NumPy, from a single multiplication up to a complete training run with hand-written gradients, and we check every number by running the code.

The vocabulary you need

A sequence is an ordered list in which order matters (the words of a sentence, the readings of a sensor, the notes of a tune); each element is a time step with index t, from t = 1 up to t = T, and the "time" can be real time or just the position in the list. Before entering the network, each element becomes a fixed-length vector of size d, and the network is made of weights, numbers it learns during training and that live in matrices that turn one vector into another. The hidden state h is the vector of length n that the network updates at every step: it is its working memory, "hidden" because it is neither the input nor the output and nobody has labelled it, and n is a hyperparameter you choose.

At every step the values of the state go through the tanh, a nonlinear function that squashes any value between −1 and 1; finally, a batch is a group of sequences processed together, which is just one more dimension in the tensors.

The problem: a feedforward network has no notion of "before" and "after"

A dense network receives an input of fixed shape and turns it into an output. If you give it a 5-word sentence and then a 12-word one, the first layer does not know what to do with it: it expects an exact number of inputs. You can force everything to a fixed length with padding, but two defects remain. The first is that order is not in the structure: "the dog bites the man" and "the man bites the dog" become two different vectors, and the network has no way to know that the words are the same in different positions.

The second is that nothing is shared: the meaning of "not" in front of an adjective is the same at position 2 and at position 40, but with separate weights for every position the network has to learn the same rule forty times. Convolution solves the sharing but only looks at a local window; an RNN, instead, carries information along the whole sequence.

What not sharing weights costs: a concrete count

Let us put numbers on the idea "learn the same thing forty times". Take a sequence of T = 100 steps with d = 50 numbers per step (for example 100 words, each described by 50 numbers) and a layer of n = 128 units. A dense network must receive the sequence "stretched" into a single vector of 100 × 50 = 5000 numbers, and every unit has one weight for each of them. An RNN reads one step at a time and reuses the same two matrices. The listing counts the parameters in the two cases with shape arithmetic (no deep learning library is needed to count) and then runs an RNN written in NumPy on three different lengths, with the same weights, to verify that the final state always has the same shape.

import numpy as np

T, d, n = 100, 50, 128                    # 100 steps, 50 features per step, 128 hidden units

dense_params = (T * d) * n + n            # dense layer fed with the whole flattened sequence
rnn_params = n * d + n * n + n            # W_xh, W_hh, b_h of one recurrent layer
print("dense:", dense_params)
print("rnn  :", rnn_params)
print("ratio:", round(dense_params / rnn_params, 1))

rng = np.random.default_rng(0)
W_xh = rng.normal(0, 0.1, (n, d))
W_hh = rng.normal(0, 0.1, (n, n))
b_h = np.zeros(n)

def run(sequence):
    h = np.zeros(n)
    states = []
    for x_t in sequence:
        h = np.tanh(W_xh @ x_t + W_hh @ h + b_h)
        states.append(h)
    return np.array(states), h

# The same weights read ANY length; the final state always has shape (n,).
for length in (5, 100, 1000):
    states, h_last = run(rng.normal(size=(length, d)))
    print(length, states.shape, h_last.shape)

# Output:
# dense: 640128
# rnn  : 22912
# ratio: 27.9
# 5 (5, 128) (128,)
# 100 (100, 128) (128,)
# 1000 (1000, 128) (128,)

The dense layer has 640,128 parameters (5000 × 128 weights plus 128 biases), the recurrent layer 22,912 (128 × 50 input weights, 128 × 128 recurrent weights, 128 biases): about 28 times fewer for the same number of units.

One detail explains an apparent difference: torch.nn.RNN keeps two separate bias vectors, bias_ih and bias_hh, which its code adds together (you can read it in the PyTorch source), so there the count is 23,040; mathematically one bias is enough, and we use one.

Fewer parameters is not just a memory saving: with fewer parameters and the same data you are less at risk of overfitting, that is, learning the examples by heart instead of a general rule. The last three lines of the output show the property that matters: with lengths 5, 100 and 1000 the intermediate states grow with the sequence, but the final state stays a vector of 128 numbers. That constant shape is what lets you attach a fixed-size head and classify sentences of any length. The cost you pay is a different one, and you can see it in the loop of run: step t cannot start before step t-1 has finished.

The analogy: reading a book with a notebook

Imagine reading a novel with a notebook of a single page. After every sentence you do the same operation: you look at the new sentence, you look at your current notes, and you rewrite the page. You cannot keep everything, so you decide what to keep and what to throw away. The notebook is the hidden state h; the rules you use to rewrite are the weights, and they are the same on page 3 and on page 300. At the end, to answer "who is the murderer?", you consult only the notebook. The analogy already announces the main defect: a page of fixed size, rewritten continuously, forgets. We will see it in the article on backpropagation through time.

The maths, step by step

Write x_t for the element at time t (a vector of size d) and h_t for the hidden state (size n). The initial state h_0 is the zero vector. The "classic" RNN, called Elman's after his 1990 work, is:

h_t = tanh( W_xh · x_t + W_hh · h_{t-1} + b_h )
y_t = W_hy · h_t + b_y

x_t  : (d,)        W_xh : (n, d)
h_t  : (n,)        W_hh : (n, n)      b_h : (n,)
y_t  : (k,)        W_hy : (k, n)      b_y : (k,)

The two lines at the top are the rule; the shapes below tell you which dimensions must match for the multiplication to be possible.

Three consequences matter more than the formula. The first concerns the tanh: without a nonlinearity the whole sequence would be a product of matrices and the state would grow or die exponentially, while the tanh keeps every component in (−1, 1). The second concerns the shared weights: W_xh, W_hh and b_h are the same at every step, so the parameters of the layer are n·d + n·n + n whatever the length; with d = 3 and n = 4 they are 12 + 16 + 4 = 32.

The third is the idea of unrolling the network: if you draw the computation step by step you get a chain of identical copies of the same cell, linked by h, that is, a network as deep as the sequence is long with the weights shared among the layers. This picture explains how it is trained (backpropagation through time) and why it breaks (gradients that vanish or explode).

How to read the drawing. At the bottom the elements x1 … x4 come in, one per step. Each box is the same cell repeated, not four different cells: that is the meaning of "shared weights". The horizontal orange arrow is the hidden state h, which leaves one step and enters the next, and it is the only channel through which the past reaches the present. At the top, if needed, an output y comes out for every step; if you only care about the final verdict (for example "positive review") you use only the last output.

One complete step with real matrices (2 inputs, 2 units)

To remove any doubt about what the shapes mean, we run two steps with d = 2 and n = 2. W_xh is 2×2 (two units, two features), W_hh is 2×2 (two units, two state units) and b has two components. The variable pre is the pre-activation, that is, the sum before the tanh; printing it lets you check every step by hand.

import numpy as np

W_xh = np.array([[0.5, -0.3],
                 [0.2,  0.8]])
W_hh = np.array([[ 0.1, 0.4],
                 [-0.6, 0.3]])
b = np.array([0.1, -0.1])

h = np.zeros(2)
for t, x in enumerate([np.array([1.0, 2.0]), np.array([0.0, 1.0])], start=1):
    pre = W_xh @ x + W_hh @ h + b          # the "pre-activation" a_t
    h = np.tanh(pre)
    print(f"t={t}  W_xh@x={W_xh @ x}  pre-activation a={pre.round(4)}  h={h.round(4)}")

# Output:
# t=1  W_xh@x=[-0.1  1.8]  pre-activation a=[0.  1.7]  h=[0.     0.9354]
# t=2  W_xh@x=[-0.3  0.8]  pre-activation a=[0.1742 0.9806]  h=[0.1724 0.7533]

Let us check the first step with a pencil. With x_1 = (1, 2), the first component of W_xh x_1 is 0.5·1 + (−0.3)·2 = −0.1; the second is 0.2·1 + 0.8·2 = 1.8. The initial state is zero, so W_hh h_0 = (0, 0). Adding the bias (0.1, −0.1) gives a = (0.0, 1.7), and the tanh, which acts component by component, gives tanh(0) = 0 and tanh(1.7) = 0.9354.

At the second step, with x_2 = (0, 1), the input contributes (−0.3, 0.8) and the memory contributes W_hh h_1 = (0.1·0 + 0.4·0.9354, −0.6·0 + 0.3·0.9354) = (0.3742, 0.2806). With the bias: (−0.3 + 0.3742 + 0.1, 0.8 + 0.2806 − 0.1) = (0.1742, 0.9806), and after the tanh h_2 = (0.1724, 0.7533). Notice that the second unit inherited from step 1 a contribution (0.2806) that depends on an input that is by now out of sight: this is memory at work.

What you can ask of this structure

• Many-to-one: you read the whole sequence and use only the last state, for example to classify the sentiment of a review.
• Synchronous many-to-many: one output for each input, for example the part of speech of every word or the forecast of the next value of a series.
• One-to-many: a single input, a sequence as output, for example a caption generated from a vector that represents an image.
• Asynchronous many-to-many (encoder-decoder): you read one sequence and produce another of a different length, as in machine translation; it needs two connected networks and is not the subject of this series.

The four forms differ only in which states are read and which inputs are supplied: the cell is the same. That is why, faced with a new problem, the first question is where you need the output (at the end, at every step, or in another sequence) and not which cell to use. In the rest of the article we work on the many-to-one case, the simplest to train and to check.

A tiny example to follow with a calculator

Before the bigger code, the smallest possible network: one input, a state made of one number, three parameters. We choose W_xh = 0.5, W_hh = 0.8, b = 0 and feed in 1, 0, 0, 0: an impulse at the first step, then silence. This way we see what the network remembers of that impulse.

import numpy as np

# The smallest possible RNN: one input, one state, three numbers.
W_xh, W_hh, b = 0.5, 0.8, 0.0
h = 0.0
for t, x in enumerate([1.0, 0.0, 0.0, 0.0], start=1):
    h = np.tanh(W_xh * x + W_hh * h + b)
    print(f"t={t}  x={x}  h={h:.4f}")

# Output:
# t=1  x=1.0  h=0.4621
# t=2  x=0.0  h=0.3537
# t=3  x=0.0  h=0.2757
# t=4  x=0.0  h=0.2170

At the first step the state was 0, so h = tanh(0.5·1 + 0.8·0) = tanh(0.5) = 0.4621. At the second the input is zero but the state is not: h = tanh(0.8·0.4621) = tanh(0.3697) = 0.3537, then 0.2757, then 0.2170. The impulse leaves a trace that decays at every step, because the recurrent weight 0.8 is less than 1 and the tanh compresses further. With a W_hh greater than 1 the trace would decay more slowly or even grow, until the tanh stopped it near ±1. This factor that multiplies the state at every step is the root of what we will see in the next article: the memory of a simple RNN lasts as long as a power of W_hh lasts.

The forward pass by hand, in NumPy

The forward pass of an RNN fits in ten lines: a for loop and a state that gets updated. Writing it by hand serves to see that there is nothing mysterious. The listing also has a second part, which rewrites the same computation as a matrix product over a batch of two sequences (the second is the first read backwards): it shows how it parallelises over the batch, not over time, and it checks that the two ways give the same function.

import numpy as np

rng = np.random.default_rng(0)
input_size, hidden_size = 3, 4

# The same three parameters are reused at every time step.
W_xh = rng.normal(0, 0.5, (hidden_size, input_size))
W_hh = rng.normal(0, 0.5, (hidden_size, hidden_size))
b_h = np.zeros(hidden_size)

def rnn_forward(sequence):
    """sequence: array (T, input_size). Returns all hidden states (T, hidden_size)."""
    h = np.zeros(hidden_size)           # h_0: the "empty memory"
    states = []
    for x_t in sequence:                # one step per element, in order
        h = np.tanh(W_xh @ x_t + W_hh @ h + b_h)
        states.append(h)
    return np.array(states)

sequence = rng.normal(size=(5, input_size))
H = rnn_forward(sequence)
print(H.shape)          # (5, 4)
print(H[-1].round(3))   # the state after reading everything

# The same function, written as a matrix product over a whole batch of 2 sequences.
batch = np.stack([sequence, sequence[::-1]])              # (batch=2, T=5, 3)
h = np.zeros((2, hidden_size))
for t in range(5):
    h = np.tanh(batch[:, t, :] @ W_xh.T + h @ W_hh.T + b_h)
print(np.abs(h[0] - H[-1]).max())                          # same function, 0.0
print(np.abs(h[1] - rnn_forward(sequence[::-1])[-1]).max())

# Output:
# (5, 4)
# [ 0.397 -0.681 -0.183  0.777]
# 0.0
# 0.0

Four lines matter. rng.normal(0, 0.5, ...) creates small random weights: you never initialise to zero, because all the units would do the same computation and would stay identical forever (the symmetry problem). h = np.zeros(hidden_size) is h_0, the "empty memory" before the first element. Inside the loop, W_xh @ x_t transforms the input, W_hh @ h transforms the old memory, the sum merges them and np.tanh compresses them. states.append(h) keeps every state, because they will be needed to give an output at every step and to compute the gradients.

The two maximum differences printed at the end are 0.0: the batch version computes exactly the same numbers, because it performs the same operations in the same order; it is the same function written with the shapes (batch, time, feature), the convention you find in libraries when you pass batch_first=True.

The library does the same thing, and you can check it. The block below continues the previous one (it uses W_xh, W_hh, b_h, sequence and H): it copies our weights into torch.nn.RNN, with the name correspondence taken from the module's source, and compares its outputs with our states. It was run with PyTorch 2.12 on CPU, 2 threads, twice with the same output. A check worth doing the first time you use a library is exactly this: copy your weights into the module and compare its outputs with yours.

# Continues the previous block: it uses W_xh, W_hh, b_h, sequence and H.
import torch, torch.nn as nn

torch.set_num_threads(2)
x = torch.tensor(sequence, dtype=torch.float32).unsqueeze(0)   # (batch=1, T=5, 3)

rnn = nn.RNN(input_size=3, hidden_size=4, batch_first=True, nonlinearity="tanh")
with torch.no_grad():
    rnn.weight_ih_l0.copy_(torch.tensor(W_xh, dtype=torch.float32))   # our W_xh
    rnn.weight_hh_l0.copy_(torch.tensor(W_hh, dtype=torch.float32))   # our W_hh
    rnn.bias_ih_l0.copy_(torch.tensor(b_h, dtype=torch.float32))      # our b_h
    rnn.bias_hh_l0.zero_()                    # PyTorch has a second bias: zero it
    out, h_n = rnn(x)
print(out.shape, h_n.shape)                   # the state of every step, and only the last
print(f"max difference, 32 bit: {np.abs(out[0].numpy() - H).max():.2e}")
print("h_n is the last state of out:", bool(torch.equal(h_n[0, 0], out[0, -1])))

# Same weights, but in 64 bit like NumPy.
rnn64 = nn.RNN(3, 4, batch_first=True, nonlinearity="tanh").double()
with torch.no_grad():
    rnn64.weight_ih_l0.copy_(torch.tensor(W_xh))
    rnn64.weight_hh_l0.copy_(torch.tensor(W_hh))
    rnn64.bias_ih_l0.zero_(); rnn64.bias_hh_l0.zero_()   # b_h is zero
    out64, _ = rnn64(torch.tensor(sequence).unsqueeze(0))
print(f"max difference, 64 bit: {np.abs(out64[0].numpy() - H).max():.2e}")

# Output:
# torch.Size([1, 5, 4]) torch.Size([1, 1, 4])
# max difference, 32 bit: 1.06e-07
# h_n is the last state of out: True
# max difference, 64 bit: 1.67e-16

What the output shows. The shapes: out is (1, 5, 4), the state of every step, and h_n is (1, 1, 4), only the last one, with (layers, batch, units) in front; the two coincide on the last step. With the weights copied and in 32 bits the maximum difference from our states is 1.06e-07, the order of magnitude of 32-bit rounding; taking the module back to 64 bits, like NumPy, it drops to 1.67e-16, that is, to double-precision noise. Our function and the library compute the same thing. Three conventions to remember: the names (weight_ih_l0 is W_xh, weight_hh_l0 is W_hh); the two biases, which PyTorch adds (bias_ih_l0 + bias_hh_l0) and which we replace with a single one, so we zero one of them; and batch_first=True, which makes the input be read as (batch, time, feature) instead of the default order (time, batch, feature).

A real task: the network learns to count

So far the computation was by hand. Let us see a complete training run on a small but non-trivial task: given a sequence of 15 bits, say whether the 1s outnumber the 0s. To succeed the network has to accumulate an approximate count along the sequence and cannot decide by looking at a single element. It is a many-to-one case: we use the last state and a linear head with a single output number, the logit (positive means "more 1s").

Here we write the gradients ourselves, even though PyTorch would compute them for us, which is a teaching advantage. The function loss_and_grads performs the backward pass through time: it starts from the derivative of the loss with respect to the last state, climbs back through the steps in reverse order, at every step goes through the tanh (1 − h²) and accumulates the contributions to the shared weights with +=, because the same weight appears at every step. The Adam optimiser is written out in full in the training loop. The loss is the binary cross-entropy computed from the logit with np.logaddexp, which avoids overflow when the logit is large. The accuracy test uses 2000 new sequences every time.

import numpy as np

rng = np.random.default_rng(0)
T_TRAIN, HIDDEN = 15, 16

def make_batch(batch, T=T_TRAIN):
    """Binary sequences. Label 1 if there are MORE ones than zeros (T odd: no ties)."""
    x = rng.integers(0, 2, size=(batch, T)).astype(float)
    y = (x.sum(axis=1) > T / 2).astype(float)
    return x, y

# Parameters: W_xh (n,1) as a vector, W_hh (n,n), b (n,), head w (n,), c scalar.
k = 1 / np.sqrt(HIDDEN)
params = {
    "Wx": rng.uniform(-k, k, HIDDEN), "Wh": rng.uniform(-k, k, (HIDDEN, HIDDEN)),
    "b": np.zeros(HIDDEN), "w": rng.uniform(-k, k, HIDDEN), "c": np.zeros(()),
}

def forward(p, x):
    """x: (batch, T). Returns logits and the list of states [h_0 .. h_T]."""
    h = np.zeros((x.shape[0], HIDDEN))
    hs = [h]
    for t in range(x.shape[1]):
        h = np.tanh(x[:, t:t + 1] * p["Wx"] + h @ p["Wh"].T + p["b"])
        hs.append(h)
    return hs[-1] @ p["w"] + p["c"], hs

def loss_and_grads(p, x, y):
    logits, hs = forward(p, x)
    prob = 1 / (1 + np.exp(-logits))
    loss = np.mean(np.logaddexp(0, logits) - y * logits)       # stable BCE with logits
    dlogit = (prob - y) / len(y)
    g = {k_: np.zeros_like(v) for k_, v in p.items()}
    g["w"], g["c"] = hs[-1].T @ dlogit, dlogit.sum()
    dh = dlogit[:, None] * p["w"]                               # dL/dh_T
    for t in range(x.shape[1], 0, -1):                          # backwards in time
        da = dh * (1 - hs[t] ** 2)                              # through the tanh
        g["Wx"] += (da * x[:, t - 1:t]).sum(axis=0)
        g["Wh"] += da.T @ hs[t - 1]
        g["b"] += da.sum(axis=0)
        dh = da @ p["Wh"]                                       # towards h_{t-1}
    return loss, g

def accuracy(p, T, n=2000):
    x, y = make_batch(n, T)
    return ((forward(p, x)[0] > 0) == (y == 1)).mean()

# Adam, written out
m = {k_: np.zeros_like(v) for k_, v in params.items()}
s = {k_: np.zeros_like(v) for k_, v in params.items()}
lr, b1, b2, eps = 1e-2, 0.9, 0.999, 1e-8
for step in range(1, 801):
    x, y = make_batch(64)
    loss, g = loss_and_grads(params, x, y)
    for k_ in params:
        m[k_] = b1 * m[k_] + (1 - b1) * g[k_]
        s[k_] = b2 * s[k_] + (1 - b2) * g[k_] ** 2
        params[k_] = params[k_] - lr * (m[k_] / (1 - b1 ** step)) / (np.sqrt(s[k_] / (1 - b2 ** step)) + eps)
    if step % 200 == 0:
        print(f"step {step:3d}  loss {loss:.3f}  test accuracy {accuracy(params, T_TRAIN):.3f}")

# Same weights, longer sequences, no retraining
for T in (15, 31, 101):
    print(f"T={T:3d} accuracy {accuracy(params, T):.3f}")

# Output:
# step 200  loss 0.054  test accuracy 0.988
# step 400  loss 0.130  test accuracy 0.919
# step 600  loss 0.008  test accuracy 0.996
# step 800  loss 0.092  test accuracy 0.991
# T= 15 accuracy 0.991
# T= 31 accuracy 0.866
# T=101 accuracy 0.722

Before trusting the numbers, let us verify that the hand-written gradients are right by comparing them with finite differences. The block below is to be pasted after the definitions of accuracy and before the Adam loop (it needs the functions of the previous listing): it perturbs a few weights by ±10⁻⁶, recomputes the loss and compares the numerical slope with the analytic one. A difference close to 10⁻¹⁰ says the derivation is correct; a large difference says there is a sign or index error.

x, y = make_batch(8)
loss, g = loss_and_grads(params, x, y)
worst = 0
for name in ("Wx", "Wh", "b", "w"):
    flat = params[name].reshape(-1)
    for i in range(min(flat.size, 6)):
        old = flat[i]; flat[i] = old + 1e-6; lp, _ = loss_and_grads(params, x, y)
        flat[i] = old - 1e-6; lm, _ = loss_and_grads(params, x, y); flat[i] = old
        num = (lp - lm) / 2e-6; an = g[name].reshape(-1)[i]
        worst = max(worst, abs(num - an))
print("max |numeric - analytic| =", f"{worst:.1e}")

# Output:
# max |numeric - analytic| = 7.0e-11

With a maximum difference of about 7·10⁻¹¹ on the parameters checked (the first six of each matrix or vector, not all of them), we can read the training output as the behaviour of the network and not as the effect of a bug in the code.

Three observations, with a single seed and therefore indicative, not definitive. The network really learns: from the 50% of chance to 98.8% already at 200 steps, with 16 units and one layer.

Training, however, is not stable: at 400 steps the accuracy drops to 91.9% even though the batch loss is low, and it climbs back to 99.6% at 600; the printed loss is that of a single batch of 64 sequences, so it fluctuates by pure statistical chance, and the learning rate (0.01) is high and there is no gradient clipping. In practice you save the model with the best accuracy on a validation set (early stopping), not the last one.

Finally, generalisation to longer sequences is partial: 86.6% at 31 bits and 72.2% at 101, with the same weights and no retraining. The weights are shared, so the network can run on any length, but in training it never saw long sequences and the count accumulated in a saturating state does not extend by itself. "Accepts any length" is a property of the architecture, not a guarantee about the result.

Variants and notations you will meet

The network studied is Elman's (1990), in which the hidden state returns as input at the next step; in Jordan's (1986) it is instead the output that returns, and you meet both names in historical texts. Stacked RNNs (num_layers > 1 in libraries) make the sequence of states of the first layer the input sequence of the second: they increase the capacity for abstraction but are harder to train. The initial state h_0 = 0 can be treated as a learned parameter, useful when the sequences have a "typical beginning".

On notation, some texts write h_t = tanh(W [h_{t−1}, x_t] + b) with a single large matrix, others separate W_xh and W_hh; they are the same thing, because the large matrix is the concatenation of the two, and some use U and W in place of W_xh and W_hh, so always check which is which.

Trade-offs

• Pro: constant parameters and free length. The same RNN reads 10 or 10,000 elements.
• Con: sequential computation. It cannot be parallelised over time, so on a GPU it is slow compared with a convolution or a Transformer that processes the whole sequence at once.
• Con: memory is a bottleneck. The whole past has to fit in n numbers; the longer the sequence, the more information is overwritten.
• Con: long-term memory is learned badly. The error signal that must reach a step far back in time gets distorted along the way (Bengio, Simard and Frasconi analysed it in 1994), and it is the historical reason for LSTM and GRU.

The only advantage in the list, free length with few parameters, is also the cause of the three disadvantages: everything follows from having a single state that crosses time with the same weights. Choose the simple RNN when the sequences are short and your constraint is the number of parameters; move to a gated cell or to another architecture when the dependency you need lies far back in time.

The traps you really fall into

The first is confusing the size of the state with the length of the sequence: hidden_size is the capacity of the memory, not the number of steps, and doubling it does not guarantee that longer sequences are remembered. The second is getting the order of the axes wrong: passing (batch, time, feature) to a module that expects (time, batch, feature) raises no error, the network treats the batch as if it were time and learns badly, so always print the shapes.

The third is forgetting that the initial state restarts from zero at every call: that is fine if every example is independent, it is a mistake if the batches are consecutive pieces of one long text and you want to preserve continuity. The fourth concerns sequences of different lengths in the same batch: if you pad with zeros and then use the last state, you are reading the state after a run of zeros and not after the last real element; you solve it with sequence packing (pack_padded_sequence in PyTorch) or by choosing the right index for each one.

Checked with four calculations

Try to answer without rereading. An RNN with input of size 10 and state of size 20 has 20×10 = 200 weights in W_xh and 20×20 = 400 in W_hh, so 600 weights plus 20 biases. If you double the sequence length from 50 to 100 the number of parameters does not change: what changes is the computation time, about double because the steps are sequential, and the memory needed for the backward pass. If you have to classify whole sentences, you pass the linear head the last state, which has seen the whole sentence; with padding, the last real state of each one. If you want a check on your gradients, use finite differences as in the previous block.

What you can do now

You can write the forward pass of an RNN in NumPy, count its parameters and explain why they do not depend on the length of the sequence, follow two steps by hand with real matrices and recognise the four ways of using it. You can also train one on a real task with hand-written gradients, check them with finite differences and read an unstable result without mistaking it for a bug. The next step is to understand why that gradient vanishes or explodes on long sequences: you will find it in Backpropagation through time. If you want to revise the basics used here, gradient descent, backpropagation step by step and NumPy and broadcasting are the prerequisites.

Previous lesson in the course: Transfer learning: reusing a pretrained CNN. Other lessons in the series that link to this one: RNN on a Time Series: Windows, Baselines and Multi-Step Forecasting, GRU: The Lean Version of Gated Memory, Tokenisation: How Text Becomes Numbers, and Why the Model Cannot Count Letters.