59 - LSTM: The Gated Cell, Equation by Equation

The Long Short-Term Memory (LSTM, Hochreiter and Schmidhuber, 1997) is an RNN with a structure designed for a single purpose: to get the error signal back through time without it dying out. It achieves this by separating two things that the simple RNN mixes together, the memory (the cell state c) and what the network shows to the outside (the hidden state h), and by controlling the flows with three multiplicative gates: forget, write, read. The key idea in one line: the cell state is updated by addition, not by a complete rework at every step.

The words you need

A gate is a vector of numbers between 0 and 1 that is multiplied element by element with another vector: multiplying by 1 lets everything through, by 0 blocks everything, by 0.3 lets 30% through. It is an adjustable tap, one for each slot of the memory, and it is built with the sigmoid σ(z) = 1 / (1 + e^(−z)), which turns any number into a value between 0 and 1 and is read as "what percentage do I let through". The element-by-element product ⊙ (Hadamard) is [a, b] ⊙ [c, d] = [a·c, b·d] and must not be confused with the row-by-column product between matrices; the concatenation [a, b] glues two vectors into a longer one.

The cell state c is the real memory of the LSTM, a vector as long as the hidden state. The difference from the hidden state h is that c is what the cell owns, while h is what the cell shows to the rest of the network and to the next step.

The analogy: a conveyor belt with three workers

Imagine a conveyor belt that runs through the whole sequence, from left to right. On the belt travels the memory c, and at every step three workers intervene. The "forget" worker (forget gate f) decides, for each slot of the belt, how much to keep: 1 means I leave everything, 0 means I empty it. The "write" worker (input gate i) decides how much of the new candidate information g to put on the belt. The "read" worker (output gate o) decides how much of the content of the belt to show to the outside as h, without removing it from the belt.

The simple RNN, by comparison, was a notebook to be rewritten completely on every page. Here the belt runs on almost unchanged until someone touches it.

The equations, one by one

Let x_t be the input (size d), h_{t-1} and c_{t-1} the previous states (size n) and [h_{t-1}, x_t] their concatenation. σ is the sigmoid (values in (0,1), perfect for "how much") and ⊙ is the element-by-element product.

f_t = σ( W_f · [h_{t-1}, x_t] + b_f )        forget gate
i_t = σ( W_i · [h_{t-1}, x_t] + b_i )        input gate
g_t = tanh( W_g · [h_{t-1}, x_t] + b_g )     candidate content (in (-1,1))
o_t = σ( W_o · [h_{t-1}, x_t] + b_o )        output gate

c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t              memory update
h_t = o_t ⊙ tanh( c_t )                       state shown to the outside

Note where the difference from the RNN lies. In the simple RNN, the state goes through a matrix and a tanh at every step. Here c_t is a gated weighted average between the old c and the new g, with no matrix in the middle. The matrix is there only inside the computation of the gates.

Parameters: each of the four transformations has a matrix of n × (n + d) and a bias of n, so in total 4·n·(n + d + 1) parameters (PyTorch keeps two bias vectors, bias_ih and bias_hh, which are added, for a total of 4·n·(n + d + 2)). That is four times a simple RNN of the same size: the first cost to take into account.

How to read the diagram. The thick green line at the top is the cell state: it enters on the left as c(t−1) and leaves on the right as c(t). It goes through only two operations: a product (×, the forget gate that erases a fraction of it) and a sum (+, which adds the new content). At the bottom, the horizontal line represents the vector [h(t−1), x(t)], that is, the old output and the new input, which feeds four small networks: three sigmoids (f, i, o, the gates) and a tanh (g, the candidate content). The gate i and the candidate g are multiplied together and the result is added to the belt. Finally a copy of the belt goes through a tanh and is multiplied by o: it is the output h(t). Note what is missing: on the belt there is no weight matrix. The gradient flowing along the green line meets only products by numbers close to 1 and sums.

Why the gradient survives

Let us derive the derivative of the cell state with respect to the one at the previous step. From the update line:

c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t
∂c_t/∂c_{t-1} = diag(f_t)  +  (terms that pass through h_{t-1} inside the gates)

The dominant term is a diagonal matrix with the values of the forget gate. There is no W_hh multiplied in and no tanh derivative squashing. If the network learns to keep f ≈ 1 for a memory slot, the gradient through that slot is preserved for many steps. The product along the sequence becomes:

∂c_T/∂c_t ≈ Π_{j=t+1..T} diag(f_j)

With f = 0.99 at every step, after 100 steps the factor is 0.99¹⁰⁰ ≈ 0.37: the signal is reduced but alive. With f = 0.9 it is 0.9¹⁰⁰ ≈ 2.7·10⁻⁵: dead. The difference between the two situations is small in the parameters and enormous in the result, and that is why the initialisation of the forget gate bias matters.

Hochreiter and Schmidhuber called this mechanism the constant error carousel. A useful historical clarification: the original 1997 LSTM did not have the forget gate, it was as if f = 1 always held. The forget gate was added by Gers, Schmidhuber and Cummins in 2000, because without it the memory grows without limit on long sequences; the cell used today in PyTorch and Keras is this version.

One LSTM step with the numbers in hand

Before writing the complete cell, let us look at what its heart does, the update c = f·c_prev + i·g, with a single memory slot and the gates given directly. On the belt there is the value 2.0, and we try three scenarios.

import numpy as np

def lstm_update(c_prev, f, i, g, o):
    """One LSTM update with the gate values given directly (a single memory slot)."""
    c = f * c_prev + i * g
    h = o * np.tanh(c)
    return c, h

c_prev = 2.0                      # the belt currently holds the value 2.0
cases = {
    "keep everything, write nothing":   dict(f=1.0, i=0.0, g=0.8, o=1.0),
    "keep 90%, write a bit of 0.8":     dict(f=0.9, i=0.5, g=0.8, o=0.7),
    "erase, then write 0.8":            dict(f=0.0, i=1.0, g=0.8, o=1.0),
}
for name, gates in cases.items():
    c, h = lstm_update(c_prev, **gates)
    print(f"{name:34s} c={c:.3f}  h={h:.3f}")

keep everything, write nothing     c=2.000  h=0.964
keep 90%, write a bit of 0.8       c=2.200  h=0.683
erase, then write 0.8              c=0.800  h=0.664

First case: f = 1 and i = 0. The memory does not change by a hair (2.000), regardless of how long the interval is. This is the situation that gives the gradient a free path. Second case: it keeps 90% (1.8) and writes half of the 0.8 candidate (0.4): 1.8 + 0.4 = 2.2. Third case: f = 0 empties the belt, i = 1 writes the whole candidate: 0.8 remains, with no trace of the earlier 2.0. Look at h too: in the first case it is 0.964 = tanh(2.0); in the second it is 0.7·tanh(2.2) = 0.683, because the output gate shows only 70% of it, while the memory c stays 2.2. It is the separation between "what I remember" and "what I say".

Code by hand: the cell in NumPy

Let us implement the cell with a single large matrix for the four gates, as the libraries do, and initialise the forget gate bias to 1. The reason (recommended for example by Jozefowicz, Zaremba and Sutskever in 2015) is to start from a cell that, by default, remembers.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

class LSTMCell:
    def __init__(self, d, n, seed=0):
        rng = np.random.default_rng(seed)
        self.n = n
        # One big matrix for the 4 gates, acting on the concatenation [h_prev, x_t]
        self.W = rng.normal(0, 0.3, (4 * n, n + d))
        self.b = np.zeros(4 * n)
        self.b[n:2 * n] = 1.0              # forget-gate bias = 1: start by "remembering"

    def step(self, x, h, c):
        n = self.n
        z = self.W @ np.concatenate([h, x]) + self.b
        i = sigmoid(z[0:n])                # input gate: how much new content to write
        f = sigmoid(z[n:2 * n])            # forget gate: how much old memory to keep
        g = np.tanh(z[2 * n:3 * n])        # candidate content
        o = sigmoid(z[3 * n:4 * n])        # output gate: how much memory to expose
        c_new = f * c + i * g              # the memory update: additive, not a squashing
        h_new = o * np.tanh(c_new)
        return h_new, c_new

cell = LSTMCell(d=3, n=4)
rng = np.random.default_rng(1)
seq = rng.normal(size=(6, 3))
h, c = np.zeros(4), np.zeros(4)
for x in seq:
    h, c = cell.step(x, h, c)
print(h.round(3), c.round(3))

Key points of the code. self.W = rng.normal(0, 0.3, (4*n, n+d)) is a single large matrix that contains the weights of the four transformations stacked (i, f, g, o): computing everything with a single matrix-vector product is far more efficient than doing four, and it is how the libraries do it. z = self.W @ np.concatenate([h, x]) + self.b applies that matrix to [h, x]; then z[0:n], z[n:2*n], … are the pieces destined for the individual gates. The line self.b[n:2*n] = 1.0 sets the forget gate bias to 1: σ(1) ≈ 0.73, so at the start the cell tends to remember rather than forget. c_new = f * c + i * g is the additive update; h_new = o * np.tanh(c_new) is the output. The final loop reads six elements while keeping the two memories h and c.

Then with the library: the same function, copying the weights

To be sure the equations are the right ones, we copy the weights of our cell into nn.LSTM. Beware of the order in which PyTorch stacks the gates: input, forget, cell (g), output. Our concatenated vector [h, x] must be split in two: weight_hh acts on h, weight_ih on x.

import torch
import torch.nn as nn

lstm = nn.LSTM(input_size=3, hidden_size=4, batch_first=True)
n = 4
# PyTorch stacks the gates in the order: input, forget, cell(g), output.
# Our matrix is [i; f; g; o] acting on [h, x]; PyTorch splits it into W_ih (acts on x) and W_hh (acts on h).
with torch.no_grad():
    lstm.weight_hh_l0.copy_(torch.tensor(cell.W[:, :n], dtype=torch.float32))
    lstm.weight_ih_l0.copy_(torch.tensor(cell.W[:, n:], dtype=torch.float32))
    lstm.bias_ih_l0.copy_(torch.tensor(cell.b, dtype=torch.float32))
    lstm.bias_hh_l0.zero_()

out, (h_n, c_n) = lstm(torch.tensor(seq, dtype=torch.float32).unsqueeze(0))
print(np.abs(h_n[0, 0].detach().numpy() - h).max(), np.abs(c_n[0, 0].detach().numpy() - c).max())
print(sum(p.numel() for p in lstm.parameters()))     # 4*n*(n+d) + 2*4*n = 4*4*7 + 32 = 144

The block creates nn.LSTM(3, 4) and copies the weights into it. cell.W[:, :n] is the part of the matrix that multiplies h (the first n columns, because we concatenated [h, x]) and goes into weight_hh_l0; cell.W[:, n:] multiplies x and goes into weight_ih_l0. The forget gate bias (b[n:2n] = 1) is copied into bias_ih_l0, and bias_hh_l0 is zeroed so as not to count it twice. At the end h_n and c_n are compared with ours: both coincide. The parameter count closes the circle: 4·4·(4+3) = 112 weights plus 2·4·4 = 32 biases = 144.

A maximum difference of the order of 3·10⁻⁸ on h and on c (float32 precision against float64), and 144 parameters, that is 4·4·7 + 2·4·4: our implementation and the library coincide, and the count confirms the formula.

Measuring the gradient flow

Let us test the argument about the gradient. We measure how much the first input of a sequence can still influence the last state: the norm of ∂(Σ h_T)/∂x_0, for the RNN, the LSTM with PyTorch's standard initialisation and the LSTM with the forget gate bias set to 1.

def grad_to_first_input(module, T, seed=0, forget_bias=None):
    """Norm of d(sum h_T)/d x_0: how much the first input can still influence the last state."""
    torch.manual_seed(seed)
    m = module()
    if forget_bias is not None:                      # LSTM bias layout: [i | f | g | o]
        with torch.no_grad():
            m.bias_ih_l0[m.hidden_size:2 * m.hidden_size] = forget_bias
    x = torch.randn(1, T, 8, requires_grad=True)
    out, _ = m(x)
    out[0, -1].sum().backward()
    return x.grad[0, 0].norm().item()

for T in (10, 50, 100):
    r = grad_to_first_input(lambda: nn.RNN(8, 32, batch_first=True), T)
    l0 = grad_to_first_input(lambda: nn.LSTM(8, 32, batch_first=True), T)
    l1 = grad_to_first_input(lambda: nn.LSTM(8, 32, batch_first=True), T, forget_bias=1.0)
    print(f"T={T:3d}  RNN {r:.1e}   LSTM {l0:.1e}   LSTM(forget bias=1) {l1:.1e}")

The function builds a module, optionally sets the forget gate bias (in PyTorch's layout the bias vector is [i | f | g | o], so the slice [hidden : 2*hidden] is exactly f), then generates a random sequence of length T with requires_grad=True and asks PyTorch for the gradient of the sum of the last state with respect to x. x.grad[0, 0] is the gradient with respect to the first input: it measures how much that first element can still influence the end. We use .norm() to reduce the vector to a number. Everything happens without training: we measure the structure, not what the network has learned.

T= 10  RNN 2.9e-03   LSTM 2.2e-03   LSTM(forget bias=1) 3.0e-02
T= 50  RNN 2.9e-14   LSTM 1.6e-11   LSTM(forget bias=1) 9.8e-06
T=100  RNN 0.0e+00   LSTM 1.1e-21   LSTM(forget bias=1) 1.6e-09

The result does not say "the LSTM solves everything", and it must be read with three cautions. At the standard initialisation the LSTM too loses gradient with distance, only more slowly than the RNN. With the forget gate bias at 1 the long-distance signal stays much higher than with the standard LSTM, small in absolute terms but more reachable by the optimiser. Finally this is a single-seed experiment on an untrained network: it says that the structure opens a road, not that training walks along it, and we will check it on a memory task in the next article.

Gate by gate: why each one exists

The LSTM has three gates and a candidate. It is fair to ask whether all of them are needed. Here is the reason each one was introduced, reasoning about what would happen without it.

The candidate g (tanh): proposing new content

It is the "draft" of what the cell could write: a transformation of the input and of the previous state, in (−1, 1). Without it there would be nothing to add to the memory. The choice of the tanh (and not the sigmoid) is important: the candidate must be able to be negative as well as positive, otherwise the memory could only grow.

The input gate i: protecting the memory from noise

If every input wrote into the memory at full strength, every irrelevant element (in the addition task, all the unmarked values) would overwrite the precious content. The input gate says "this step is noise, write 0". Hochreiter and Schmidhuber motivated it with the problem of input weight conflict: the same weight must sometimes make the memory be written and sometimes not, and without a gate that separates the two situations training is forced into a compromise.

The output gate o: protecting the rest of the network from the memory

The memory c may hold a value that is needed only many steps later. The output gate makes it possible not to show it (h ≈ 0) until the time comes, so the other layers do not have to keep handling an irrelevant signal. It is the counterpart, on the output side, of what the input gate does on the input side (output weight conflict).

The forget gate f: being able to empty the belt

In the 1997 version the memory could only accumulate. On long or continuous sequences, with no possibility of zeroing it, the values of c grow without limit and the output tanh saturates (Gers, Schmidhuber and Cummins, 2000). The forget gate makes it possible to "reset" a slot when a new context begins (a new sentence, a new document). It costs one more transformation, and it is the gate with the most influence on long-term behaviour.

A field test: switching off one gate at a time

To see whether the gates really "are needed", we write the LSTM by hand in PyTorch with the option of forcing a gate to 1 (that is, eliminating it: f = 1 means a memory that never forgets, i = 1 that always writes, o = 1 that always shows). We repeat the addition problem at T = 30. A cell written in Python is much slower than nn.LSTM, but it is the only way to inspect inside.

Two methodological choices before the code. The first: every variant is trained on 5 seeds (0-4), and for each one we report the mean, minimum and maximum error. Training starts from random weights and draws random batches, so two runs of the same network give different errors: with a single run per variant, the difference between variants gets mixed up with the noise between runs. The second: torch.set_num_threads(2), fixed on purpose for reproducibility. The number of threads changes the order of floating-point sums, and floating-point sums are not associative: with a different thread count the same seeds produce different numbers. The program below was run twice with identical output; on another machine, or with another thread count, the exact figures may change, and the conclusions must hold anyway.

import numpy as np
import torch
import torch.nn as nn

torch.set_num_threads(2)    # reproducibility: float sums depend on the thread count

class AblatableLSTM(nn.Module):
    """LSTM written by hand so that any single gate can be switched off (forced to 1)."""
    def __init__(self, d, n, drop=None):
        super().__init__()
        self.n, self.drop = n, drop              # drop in {None, "forget", "input", "output"}
        self.lin = nn.Linear(d + n, 4 * n)
        with torch.no_grad():
            self.lin.bias[n:2 * n] = 1.0         # forget bias = 1 for everybody
    def forward(self, x):
        B, T, _ = x.shape
        h = x.new_zeros(B, self.n); c = x.new_zeros(B, self.n)
        for t in range(T):
            z = self.lin(torch.cat([h, x[:, t]], dim=-1))
            i, f, g, o = z.chunk(4, dim=-1)
            i, f, o, g = torch.sigmoid(i), torch.sigmoid(f), torch.sigmoid(o), torch.tanh(g)
            if self.drop == "forget": f = torch.ones_like(f)     # never forgets
            if self.drop == "input":  i = torch.ones_like(i)     # always writes
            if self.drop == "output": o = torch.ones_like(o)     # always shows
            c = f * c + i * g
            h = o * torch.tanh(c)
        return h                                  # final hidden state

class AddingNet(nn.Module):
    def __init__(self, drop=None, n=32):
        super().__init__()
        self.cell = AblatableLSTM(2, n, drop)
        self.head = nn.Linear(n, 1)
    def forward(self, x):
        return self.head(self.cell(x)).squeeze(-1)

def adding_batch(batch, T):
    values = torch.rand(batch, T)
    markers = torch.zeros(batch, T)
    for b in range(batch):
        markers[b, torch.randperm(T)[:2]] = 1.0
    return torch.stack([values, markers], dim=-1), (values * markers).sum(dim=1)

def reset_batch(batch, T=30, p_reset=0.15):
    """Running sum that must be CLEARED at every reset marker.
    Target: sum of the values that come strictly after the last reset (scaled by 1/5)."""
    values = torch.rand(batch, T)
    resets = (torch.rand(batch, T) < p_reset).float()
    resets[:, 0] = 1.0                                   # at least one reset at the start
    y = torch.zeros(batch)
    for b in range(batch):
        last = int(resets[b].nonzero().max())
        y[b] = values[b, last + 1:].sum()
    return torch.stack([values, resets], dim=-1), y / 5.0

def run(make_batch, drop, steps=2500, seed=0):
    torch.manual_seed(seed)
    net = AddingNet(drop)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for _ in range(steps):
        x, y = make_batch(64)
        loss = nn.functional.mse_loss(net(x), y)
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(net.parameters(), 1.0)
        opt.step()
    with torch.no_grad():
        x, y = make_batch(1000)
        return nn.functional.mse_loss(net(x), y).item()

def table(make_batch, seeds=range(5)):
    for drop in (None, "forget", "input", "output"):
        errs = np.array([run(make_batch, drop, seed=s) for s in seeds])
        print(f"gate removed: {str(drop):7s} mean {errs.mean():.4f}  min {errs.min():.4f}  max {errs.max():.4f}  "
              f"per seed {np.round(errs, 4).tolist()}")

table(lambda batch: adding_batch(batch, 30))
print("baseline (always predict the mean): 0.1667")

gate removed: None    mean 0.0004  min 0.0002  max 0.0010  per seed [0.0002, 0.001, 0.0003, 0.0002, 0.0002]
gate removed: forget  mean 0.0006  min 0.0003  max 0.0008  per seed [0.0007, 0.0008, 0.0004, 0.0003, 0.0006]
gate removed: input   mean 0.0015  min 0.0010  max 0.0022  per seed [0.001, 0.001, 0.002, 0.0014, 0.0022]
gate removed: output  mean 0.0005  min 0.0001  max 0.0009  per seed [0.0004, 0.0004, 0.0009, 0.0001, 0.0005]
baseline (always predict the mean): 0.1667

Explanation. AblatableLSTM computes the four transformations with a single nn.Linear(d+n, 4n) and separates them with chunk(4); then, depending on drop, it replaces one gate with a tensor of ones. The forward returns only the final hidden state and AddingNet puts a linear head on top of it. run trains one network for 2500 steps with a given seed; table repeats it over the 5 seeds and prints mean, minimum, maximum and the five values. What the numbers say: all four variants solve the task on every seed (the worst error is 0.0022, against the 0.1667 baseline of always predicting the mean). Among the variants, though, almost nothing separates. Forget and output have ranges (0.0003-0.0008 and 0.0001-0.0009) that lie inside that of the full cell (0.0002-0.0010): they cannot be told apart. The input gate has the highest mean error (0.0015 against 0.0004) and none of its seeds goes below 0.0010, but one seed of the full cell reaches exactly 0.0010: the ranges touch. That is a hint that removing the input gate costs something, not a clear difference, and with five seeds I do not claim it.

Why the experiment does not tell the variants apart. The task is easy: every error is a hundred to a thousand times below the baseline, and the differences are thousandths. The network is oversized: 32 units to add two numbers is plenty, and a generous network finds alternative routes when a gate is taken away. And addition never requires forgetting, so a forget gate fixed at 1 works (it is the 1997 LSTM), while the output gate is redundant with a linear head that already knows how to choose what to read. To see a difference we would need a harder task (much longer sequences), a smaller network, fewer training steps or many more seeds; I did not try longer sequences here, and I cannot say what they would show.

To make the forget gate matter we need a task that asks for emptying the memory. We build one: a running sum with reset. The network receives random values and a reset marker; the target is the sum of the values after the last reset. The classes and the table function of the previous block are reused.

def reset_batch(batch, T=30, p_reset=0.15):
    """Running sum that must be CLEARED at every reset marker.
    Target: sum of the values that come strictly after the last reset (scaled by 1/5)."""
    values = torch.rand(batch, T)
    resets = (torch.rand(batch, T) < p_reset).float()
    resets[:, 0] = 1.0                                   # at least one reset at the start
    y = torch.zeros(batch)
    for b in range(batch):
        last = int(resets[b].nonzero().max())
        y[b] = values[b, last + 1:].sum()
    return torch.stack([values, resets], dim=-1), y / 5.0

table(lambda batch: reset_batch(batch, 30))
torch.manual_seed(0)
print(f"variance of the target = {reset_batch(2000)[1].var().item():.4f}")

gate removed: None    mean 0.0004  min 0.0001  max 0.0007  per seed [0.0005, 0.0001, 0.0007, 0.0007, 0.0002]
gate removed: forget  mean 0.0014  min 0.0009  max 0.0021  per seed [0.0013, 0.0021, 0.0013, 0.0009, 0.0012]
gate removed: input   mean 0.0003  min 0.0001  max 0.0004  per seed [0.0003, 0.0002, 0.0001, 0.0003, 0.0004]
gate removed: output  mean 0.0003  min 0.0001  max 0.0006  per seed [0.0006, 0.0002, 0.0001, 0.0002, 0.0003]
variance of the target = 0.3687

Here the picture is different, and it is clear. Without the forget gate every seed lies between 0.0009 and 0.0021, while all the other variants stay below 0.0007 (full cell 0.0001-0.0007, no input gate 0.0001-0.0004, no output gate 0.0001-0.0006): the ranges are disjoint, and the mean error without forget (0.0014) is higher than that of the other three (0.0003-0.0004). This difference holds on all five seeds, and it is consistent with the idea that the forget gate is needed for zeroing. Without the input or the output gate, instead, the means (0.0003) are just below that of the full cell (0.0004), but the ranges overlap: I cannot say that removing those gates helps or is harmless, only that this experiment does not tell them apart. And even the forget effect is small in absolute terms (the variance of the target is 0.3687): a cell with 32 units finds a way to compensate anyway, for example by writing values that cancel the previous content through the candidate.

The lesson is twofold. On content: the gates give a measurable but modest advantage on small, easy tasks, and the only effect the data support clearly is that of the forget gate on the task that asks for forgetting; the picture is consistent with the systematic study by Greff and colleagues (2017, "LSTM: A Search Space Odyssey"), who compare many variants of the cell and conclude that the forget gate and the output activation function are the most critical components, while none of the proposed variants improves significantly on the standard LSTM.

On method: a single run is not enough, because training is random and the difference between two variants has to be compared with the variation between seeds of the same variant; and the number of threads changes floating-point results, so it must be fixed and declared. In the first draft of this article one seed and the default thread count gave a ranking ("the input gate costs more", "without forget the error triples") that reversed with 2 threads: it was noise, not a result.

Trade-offs

• Long memory at a price. Four times the parameters and about four times the computation per step compared with the simple RNN.
• Still sequential. Step t waits for step t−1: no parallelisation in time.
• Capacity limited to n numbers. An LSTM has no addressing: the whole past stays compressed in c.
• More hyperparameters that matter: state size, forget gate bias, dropout between layers.

After the list of compromises, the assessment that counts is this: the LSTM buys long-distance memory by paying in parameters and in computation time, and it does not change the sequential nature of the RNN. If your dependency is short, the simple RNN or the GRU cost less; if it is long, the LSTM is the first tool to try before moving to heavier architectures.

Real traps

The first is the wrong gate order when copying weights between frameworks (in PyTorch: i, f, g, o): the code raises no error, it just produces wrong results. The second is forgetting that there are two states: nn.LSTM returns (h_n, c_n) and to resume a sequence you need both, because passing only h empties the memory. The third is using h as if it were the memory: h is what the output gate decides to show and can be almost zero while c keeps information. The fourth is thinking the LSTM eliminates clipping: it can still explode, and clipping remains a good habit.

Checked with four calculations

An LSTM with input 10 and state 20 has, with PyTorch's two bias vectors, 4·20·(20+10+2) = 2560 parameters. If the forget gate is always 1 and the input gate always 0 you get c_t = c_{t−1}: the cell state stays constant forever, a frozen memory. h can be close to zero while c contains information because h = o ⊙ tanh(c), and with the output gate closed h is small even if c is large. With f = 0.98 constant, a piece of written information is halved when 0.98^k = 0.5, that is, k = ln 0.5 / ln 0.98 ≈ 34 steps: a "half-life" of about 34 steps.

A complete LSTM step, with every number in view

Let us take a single step of a cell with one unit, so that every gate is a number. Previous hidden state h = 0.5, previous memory c = 2.0, input x = 1.0. The weights (each applied to h and to x, plus a bias) are: forget (0.4, 0.6, bias 1.0), input (0.3, −0.2, 0), candidate (0.5, 0.8, 0), output (0.1, 0.7, 0).

1. The four linear combinations (weight on h × 0.5 + weight on x × 1.0 + bias):

z_f = 0.4·0.5 + 0.6·1.0 + 1.0 = 1.80
z_i = 0.3·0.5 − 0.2·1.0 + 0   = −0.05
z_g = 0.5·0.5 + 0.8·1.0 + 0   = 1.05
z_o = 0.1·0.5 + 0.7·1.0 + 0   = 0.75

2. The activations. The three gates go through the sigmoid, the candidate through the tanh:

f = σ(1.80)  = 0.858      (I keep 85.8% of the old memory)
i = σ(−0.05) = 0.488      (I write less than half)
g = tanh(1.05) = 0.782    (candidate content)
o = σ(0.75)  = 0.679      (I show 67.9%)

3. The memory update. c_new = f·c + i·g = 0.858·2.0 + 0.488·0.782 = 1.716 + 0.381 = 2.097. The old memory (2.0) has been reduced to 1.716 and a new contribution of 0.381 has been added to it.

4. The output. h_new = o · tanh(c_new) = 0.679 · tanh(2.097) = 0.679 · 0.970 = 0.659.

Note three things. The memory has changed little (from 2.0 to 2.097) because f is high and i is low: the cell is in "preserve" mode. The output 0.659 is much smaller than the memory (2.097): it is the result of two compressions, the tanh and the gate o, and it shows that h and c are two different numbers. And if the contribution i·g had been negative it would have subtracted from the memory: the candidate in (−1, 1) allows both adding and removing.

Frequently asked questions

Why do the gates use the sigmoid and not the tanh? Because a gate is a percentage (how much to let through) and must lie between 0 and 1. The tanh would give negative values, which would flip the sign of the signal instead of attenuating it.

Why does the candidate use the tanh? Because the content to be written must be able to be both positive and negative, and bounded in (−1, 1) so as not to make the memory explode at every step.

Does the LSTM really "remember" forever? No. With f = 0.98 constant, the information has a half-life of about 34 steps (0.98^34 ≈ 0.5). To remember longer f must get closer to 1, and must stay there: that is why the initial bias matters.

Why two states (h and c) and not one? Because "what I remember" and "what I say" are different tasks: the memory can hold an important value that is not needed for now (output gate closed). The GRU gives up this separation to save parameters (see the GRU).

How many units should I use? Start from 32-128 and measure: we saw that 8 are too few for the addition problem and 32 are enough; 128 bring no advantage. More units = more parameters (quadratic growth with the state size) and more risk of overfitting.

Can an LSTM be trained on a GPU? Yes, but the computation in time stays sequential: the GPU speeds up the matrix products inside each step and the parallelism over the batch, not the steps themselves. That is why attention models, which process the whole sequence together, are faster on long sequences.

What you can do now

You can write the six equations of the LSTM cell, explain why the additive update of the cell state gives the gradient a path with a factor close to 1, and follow a complete step with every number in view. You can implement the cell in NumPy, copy its weights into nn.LSTM respecting the gate order and measure the gradient flow, knowing that the structure opens the road but does not walk along it by itself. The next step is to put it to the test on memory tasks in LSTM put to the test: the adding problem; to start again from the basics, BPTT and gradient clipping.

In the series, the previous lesson is RNN on a Time Series: Windows, Baselines and Multi-Step Forecasting.