62 - Bidirectional RNNs: Reading the Sequence Both Ways

A bidirectional RNN (Schuster and Paliwal, 1997) is two independent RNNs over the same sequence: one reads it from left to right, the other from right to left, and at each position their states are glued together. This way, for every element, the representation contains a summary of what comes before and one of what comes after. It is a powerful tool for every task in which you have the whole sequence available, and a trap for those in which you do not. This lesson shows both with numbers.

The words of this lesson

The task we will use is sequence labelling: assigning a label to every element of a sequence, such as the part of speech of each word, instead of a single label for the whole sequence. For each position we will talk about context: the left context is what comes before, the right context what comes after. The networks produce a logit, the raw output before the transformation into a probability (a positive logit means a probability above 50%), and learn by minimising the binary cross-entropy, which compares the predicted probability with the 0/1 label and is small when the network is confident and right, huge when it is confident and wrong. The number we report at the end is the accuracy, the fraction of correct predictions.

The problem: an RNN is blind to the future

When an RNN processes position t, its state h_t depends only on x_1 … x_t. If you want to label the word "Washington" in "I saw Washington win the election" as a person or a place, what follows ("win the election") helps enormously, but a unidirectional RNN has not yet read it when it gets to "Washington".

The analogy: proofreading a text with two pens

A proofreader reads the text from the beginning to the end and notes what they expect for each word based on what they have already read. A second proofreader reads it from the end to the beginning and notes the same based on what comes after. In the end, for each word you have two notes. Neither of them has seen the whole context, but together they have.

The mathematics

In the drawing, the two rows of cells do not exchange information with each other: they are two independent RNNs. The blue row (forward) at each position has already seen the elements on the left; the green row (backward) those on the right. Only at the top, in the yellow box, are the two summaries glued together, and from there on the network (a linear head, another layer) can use both.

Two independent (not shared) sets of weights. The forward state is that of the usual RNN; the backward one runs through the sequence in reverse:

→h_t = f( →W_x x_t + →W_h →h_{t-1} + →b )        t = 1, 2, …, T
←h_t = f( ←W_x x_t + ←W_h ←h_{t+1} + ←b )        t = T, T−1, …, 1
h_t  = [ →h_t ; ←h_t ]                            concatenation: dimension 2n

The cost is exactly double the parameters and double the computation. The output dimension doubles: the classification head must accept 2n inputs.

Code by hand

The subtle part is the alignment: the backward pass produces the states in reverse order, so they must be flipped before concatenating. A common mistake is to forget it, and the two sides end up misaligned by one position.

import numpy as np

rng = np.random.default_rng(0)
d, n, T = 3, 4, 5

def make_rnn_params():
    return dict(Wx=rng.normal(0, .5, (n, d)), Wh=rng.normal(0, .5, (n, n)), b=np.zeros(n))

def run_rnn(p, seq):
    h, out = np.zeros(n), []
    for x in seq:
        h = np.tanh(p["Wx"] @ x + p["Wh"] @ h + p["b"])
        out.append(h)
    return np.array(out)                                  # (T, n)

fwd, bwd = make_rnn_params(), make_rnn_params()           # two INDEPENDENT sets of weights
seq = rng.normal(size=(T, d))

H_fwd = run_rnn(fwd, seq)                                 # reads left -> right
H_bwd = run_rnn(bwd, seq[::-1])[::-1]                     # reads right -> left, then re-aligned in time
H = np.concatenate([H_fwd, H_bwd], axis=1)                # (T, 2n): at each t, past AND future summary
print(H.shape)                                            # (5, 8)

Points to note. make_rnn_params returns a dictionary of weights; we call it twice to get two different sets (fwd and bwd). run_rnn is the RNN we already know. The key line is run_rnn(bwd, seq[::-1])[::-1]: seq[::-1] reverses the sequence, the RNN reads it "from left to right" of this reversed copy (that is, from right to left of the original), and the last [::-1] realigns the states to the original times, so that row t of H_bwd describes position t. np.concatenate(..., axis=1) glues the two states: shape (5, 8).

Then with the library

In PyTorch it is a flag, bidirectional=True. The weights of the other direction have the suffix _reverse. Let us copy them and check that the two coincide.

import torch
import torch.nn as nn

rnn = nn.RNN(d, n, batch_first=True, bidirectional=True)
with torch.no_grad():
    for suffix, p in [("", fwd), ("_reverse", bwd)]:
        getattr(rnn, f"weight_ih_l0{suffix}").copy_(torch.tensor(p["Wx"], dtype=torch.float32))
        getattr(rnn, f"weight_hh_l0{suffix}").copy_(torch.tensor(p["Wh"], dtype=torch.float32))
        getattr(rnn, f"bias_ih_l0{suffix}").copy_(torch.tensor(p["b"], dtype=torch.float32))
        getattr(rnn, f"bias_hh_l0{suffix}").zero_()

out, h_n = rnn(torch.tensor(seq, dtype=torch.float32).unsqueeze(0))
print(out.shape, h_n.shape)                      # (1, 5, 8) and (2, 1, 4): [forward, backward]
print(np.abs(out[0].detach().numpy() - H).max()) # ~1e-7 (float32 against float64)
# h_n[0] is the forward state AFTER the last element; h_n[1] is the backward state after reading the FIRST element.

Two things to remember about the shape. out has dimension 2n on the last axis, forward first, backward after. h_n has num_layers × 2 as its first axis (one row per direction): h_n[0] is the forward state after the last element, h_n[1] is the backward state after reading the first. To classify the whole sequence you concatenate precisely these two, not the last position of out: out[:, -1] contains the backward state after reading only the last element, poor information.

Experiment: a label that depends on both sides

Let us build a tailor-made task. Random digits from 0 to 9; for each position the label is 1 if x_{t−1} + x_{t+1} > 9 (at the borders the missing neighbour is 0). The label depends on the neighbours on the right and on the left. The positive class covers 45% of the cases (45 pairs out of 100 have a sum ≥ 10), so always guessing "0" gives about 55% accuracy.

def make_batch(batch, T=12):
    x = torch.randint(0, 10, (batch, T)).float()
    left = torch.cat([torch.zeros(batch, 1), x[:, :-1]], dim=1)     # x_{t-1}, 0 at the border
    right = torch.cat([x[:, 1:], torch.zeros(batch, 1)], dim=1)     # x_{t+1}, 0 at the border
    y = ((left + right) > 9).float()      # label depends on the neighbour on BOTH sides
    return (x / 9).unsqueeze(-1), y

class Tagger(nn.Module):
    def __init__(self, bidirectional):
        super().__init__()
        self.rec = nn.GRU(1, 32, batch_first=True, bidirectional=bidirectional)
        self.head = nn.Linear(32 * (2 if bidirectional else 1), 1)
    def forward(self, x):
        out, _ = self.rec(x)
        return self.head(out).squeeze(-1)                            # one logit per time step

def accuracy_after_training(bidirectional, steps=1500, seed=0):
    torch.manual_seed(seed)
    net = Tagger(bidirectional)
    opt = torch.optim.Adam(net.parameters(), lr=5e-3)
    for _ in range(steps):
        x, y = make_batch(64)
        loss = nn.functional.binary_cross_entropy_with_logits(net(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        x, y = make_batch(2000)
        return ((net(x) > 0).float() == y).float().mean().item()

print("unidirectional", round(accuracy_after_training(False), 3))
print("bidirectional ", round(accuracy_after_training(True), 3))

The generator make_batch creates random digits x; left and right are the same sequence shifted by one place on either side (with zero at the border): they are the neighbours x_{t−1} and x_{t+1} of each position. The label y is 1 when their sum exceeds 9. The network's input is only x / 9 (the digits rescaled to [0, 1]): the neighbours are not given to the network as separate columns, it must derive them from the sequence. Tagger has a GRU, unidirectional or bidirectional depending on the flag; the linear head receives 32 or 64 features (double if bidirectional) and produces one logit for every position (out contains all the steps, not just the last). The loss is binary_cross_entropy_with_logits, which applies the sigmoid internally in a numerically stable way; the final accuracy counts how many times logit > 0 matches the label.

unidirectional 0.791
bidirectional  1.0

The unidirectional GRU reaches 79.1%: it uses the left neighbour (which it has read) and has to guess the right one. The bidirectional one reaches 100%. One seed and a synthetic task, but the mechanism is clear: without the right side, the information needed to answer is not in the network.

When it is forbidden: future leakage

Here is the most serious trap of the whole lesson. A bidirectional network uses the future to compute the representation of every instant. If your problem is to predict something that in the real world you do not know yet, that future will not exist.

• Time-series forecasting: using a bidirectional network to predict x_{t+1} means that the input contains, through the reverse direction, the very value you want to predict. On the test data you will get beautiful errors and in production nothing will work.
• Text generation: a model that predicts the next word cannot look at the words that follow.
• Streaming recognition (live captions, online anomaly detection): you must answer before the rest arrives; at most you accept a small latency and a limited window.

Where it is fine: offline sequence labelling (parts of speech, entities), classification of whole documents, translation encoders (the whole source sentence is read), and all the cases in which the sequence is complete at the moment of the decision. ELMo (Peters and colleagues, 2018), for example, used LSTMs in both directions to produce word representations.

Seeing the leakage with your own eyes

The costliest trap of bidirectionality is not theoretical. Let us build a realistic mistake: a forecasting pipeline in which, by oversight, the window given to the network also includes the value to be predicted (it happens, for example, when a slicing index is wrong or the window is built with an inclusive end instead of an exclusive one). A bidirectional network, reading the sequence from the right too, sees that value immediately. Let us measure what the mistake promises compared with the correct pipeline.

# A leaky pipeline: the window handed to a bidirectional GRU accidentally CONTAINS the value to predict.
import numpy as np
import torch
import torch.nn as nn

rng = np.random.default_rng(0)
t = np.arange(2000)
series = np.sin(0.1 * t) + 0.5 * np.sin(0.31 * t) + rng.normal(0, 0.1, t.size)
split = int(0.8 * len(series))
mu, sigma = series[:split].mean(), series[:split].std()
norm = (series - mu) / sigma

def make(x, size=30, leaky=False):
    """Honest: input = x[i:i+size], target = x[i+size].  Leaky: input also includes x[i+size]."""
    n = len(x) - size
    if leaky:
        X = np.stack([x[i:i + size + 1] for i in range(n)])
    else:
        X = np.stack([x[i:i + size] for i in range(n)])
    return (torch.tensor(X[..., None], dtype=torch.float32),
            torch.tensor(x[size:size + n], dtype=torch.float32))

class BiGRUForecaster(nn.Module):
    def __init__(self, hidden=32):
        super().__init__()
        self.rnn = nn.GRU(1, hidden, batch_first=True, bidirectional=True)
        self.head = nn.Linear(2 * hidden, 1)
    def forward(self, x):
        _, h_n = self.rnn(x)                              # h_n: (2, batch, hidden) = [forward, backward]
        return self.head(torch.cat([h_n[0], h_n[1]], dim=-1)).squeeze(-1)

def experiment(leaky, epochs=10, seed=0):
    torch.manual_seed(seed)
    Xa, ya = make(norm[:split], leaky=leaky)
    Xb, yb = make(norm[split:], leaky=leaky)
    net = BiGRUForecaster()
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for _ in range(epochs):
        perm = torch.randperm(len(Xa))
        for i in range(0, len(Xa), 64):
            idx = perm[i:i + 64]
            loss = nn.functional.mse_loss(net(Xa[idx]), ya[idx])
            opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        return (net(Xb) - yb).abs().mean().item()

print("honest window :", round(experiment(False), 3))
print("leaky window  :", round(experiment(True), 3))
persistence = np.abs(norm[split + 30:] - norm[split + 29:-1]).mean()   # "tomorrow = today"
print("persistence baseline:", round(float(persistence), 3))

The program trains the same network twice with the same seed, once with honest windows and once with contaminated ones, and prints the mean absolute error on the final stretch of the series, never seen in training (normalised data, so the error is in units of standard deviation). The three lines are, in order, honest window, contaminated window and persistence baseline:

honest window : 0.117
leaky window  : 0.065
persistence baseline: 0.2

make(x, size, leaky) builds the windows. With leaky=False the window goes from i to i+size−1 and the target is x[i+size]; with leaky=True the window extends up to i+size inclusive, that is it contains the target as its last element. BiGRUForecaster concatenates the forward and backward states (h_n[0] and h_n[1]) and passes them to a linear head. The persistence baseline ("tomorrow will be the same as today") is computed in one line and serves as a reference point: if a model does not beat it, it has learned nothing useful.

With the honest window the mean absolute error is 0.117, with the contaminated one 0.065: about 44% less, an "improvement" that would be celebrated in any presentation and that is completely false. In production, when you have to predict x[i+size], that value does not exist. The error does not go to zero (the network, with only 10 epochs, does not copy the last value perfectly), and this makes the bug more insidious: a result that is "good but not absurd" sets off no alarm.

The countermeasures are concrete. Write an automatic test that checks that, for every example, no element of the input has an index greater than or equal to that of the target. Always compare with the persistence baseline. Be suspicious of any model that beats the baseline by a wide margin without a clear reason.

A remark on the comparison: the honest bidirectional network (0.117) beats persistence (0.2), but this should not be read as an advantage of bidirectionality. A window of 30 past values contains no "future" with respect to the target, and the backward reading offers only a second summary of the same window. The right comparison would be with a unidirectional GRU at equal parameters and epochs, which we have not measured here.

Table of shapes (to keep close)

nn.GRU / nn.LSTM / nn.RNN(input, H, num_layers=L, bidirectional=True, batch_first=True)

input x    : (B, T, input)
out        : (B, T, 2H)           forward in [..., :H], backward in [..., H:]
h_n        : (2L, B, H)           order: layer0-fwd, layer0-bwd, layer1-fwd, layer1-bwd, ...
                                   last layer: h_n[-2] (fwd) and h_n[-1] (bwd)
parameters : double a unidirectional network with the same H
next layer : receives 2H as its input dimension

The most misunderstood line is the one about h_n: the layers are stacked in order, and for each layer the forward direction comes first and then the backward one. To classify a sequence with a multi-layer network, the two states to concatenate are h_n[-2] and h_n[-1], not h_n[0] and h_n[1] (which belong to the first layer). With a single layer the two choices coincide, and that is why the mistake goes unnoticed until you add the second.

For the parameters the rule is "double". A GRU has three blocks of weights (update gate, reset gate and candidate), each with a matrix on the input, one on the state and two bias vectors, so for one direction 3·H·(H + input + 2). For nn.GRU(10, 16, bidirectional=True) that gives 2·3·16·(16+10+2) = 2688, and sum(p.numel() for p in net.parameters()) returns exactly 2688 (measured; with nn.GRU(20, 64, bidirectional=True) the count gives 33,024, and that too coincides). The output has 2·16 = 32 features per position.

A case where bidirectionality is indispensable: brackets

The experiment on neighbouring digits showed that the right context is needed. Here is a more "structured" example, resembling real problems (code analysis, languages with nested constructs): a random string of open brackets ( and closed ones ). For each symbol, the label says whether that bracket finds its partner, according to the stack rule: every ( "opens" and closes with the first free ) that follows; a ) with no open ( stays unmatched. For example, in the string )())((()(()()((( the labels are 0110001101111000: 1 = it has a partner, 0 = it does not.

For closing brackets the answer depends only on the past (is there a free open bracket?). For opening brackets it depends on the future (will a free closing one arrive?). A unidirectional network should therefore do well on the former and badly on the latter.

import torch
import torch.nn as nn

L = 16
def make_brackets(batch, seed=None):
    """Random strings of '(' and ')'. Label 1 = this bracket gets matched (stack matching), 0 = it stays unmatched.
    Token ids: 0 = '(', 1 = ')'."""
    g = torch.Generator().manual_seed(seed) if seed is not None else None
    x = torch.randint(0, 2, (batch, L), generator=g)
    y = torch.zeros(batch, L)
    for b in range(batch):
        stack = []
        for t in range(L):
            if x[b, t] == 0:
                stack.append(t)
            elif stack:                                  # a ')' that finds an open '(' closes it
                y[b, stack.pop()] = 1.0; y[b, t] = 1.0
    return x, y

class BracketTagger(nn.Module):
    def __init__(self, bidirectional, hidden=64):
        super().__init__()
        self.emb = nn.Embedding(2, 8)
        self.rnn = nn.GRU(8, hidden, batch_first=True, bidirectional=bidirectional)
        self.head = nn.Linear(hidden * (2 if bidirectional else 1), 1)
    def forward(self, x):
        out, _ = self.rnn(self.emb(x))
        return self.head(out).squeeze(-1)

def run_brackets(bidirectional, steps=2500, seed=0):
    torch.manual_seed(seed)
    net = BracketTagger(bidirectional)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for _ in range(steps):
        x, y = make_brackets(64)
        loss = nn.functional.binary_cross_entropy_with_logits(net(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
    x, y = make_brackets(3000, seed=99)
    with torch.no_grad():
        pred = (net(x) > 0).float()
    opens, closes = (x == 0), (x == 1)
    acc = lambda m: ((pred == y) & m).sum().item() / m.sum().item()
    return acc(opens), acc(closes), (pred == y).float().mean().item()

x, y = make_brackets(1, seed=5)
print("example:", "".join("()"[i] for i in x[0].tolist()), "\nlabels :", "".join(str(int(v)) for v in y[0].tolist()))
for bi in (False, True):
    o, c, a = run_brackets(bi)
    print(f"{'bidirectional' if bi else 'unidirectional':15s} accuracy on '(' tokens {o:.3f}   on ')' tokens {c:.3f}   overall {a:.3f}")

example: )())((()(()()(((
labels : 0110001101111000
unidirectional  accuracy on '(' tokens 0.713   on ')' tokens 0.995   overall 0.854
bidirectional   accuracy on '(' tokens 1.000   on ')' tokens 1.000   overall 1.000

Explaining the code. make_brackets generates random strings of length 16 (id 0 = (, 1 = )) and computes the labels with a stack: every ( is pushed onto the stack; when a ) arrives and the stack is not empty, the ( on top and the current ) receive label 1. BracketTagger is an embedding of 8 numbers, a 64-unit GRU (unidirectional or bidirectional) and a linear head that produces one logit for every position. Training is standard (BCE, Adam, 2500 steps) and the evaluation, on 3000 new strings with a fixed seed, separates the accuracy on ( from that on ).

The result is sharp and matches the analysis. The unidirectional network is almost perfect on ) (99.5%), because for them the past is enough (it only has to remember "how many free open brackets there are"), but on ( it barely reaches 71.3%: it cannot know whether the closing one appears further on. The bidirectional one is perfect on both (100%): the two directions, together, contain what is needed. The 71.3% is not chance (which would be ~50% or the frequency of the majority class): the unidirectional network has learned the statistic "a ( at the beginning of the string is more likely to be closed than one at the end", which is the part of the information available to it. This is the example to remember to understand what it means to need the right context: not a bit more accuracy, but a class of problems solved or not solved.

Trade-offs and traps

Bidirectionality is paid for in three ways. The first is cost: parameters and time double, and the backward pass cannot start until you have the whole sequence, so you also have latency. The second is memory: in training the states of both passes are needed. The third is a limit, not a cost: bidirectional does not mean that each direction remembers better. If the dependencies are far apart you still need an LSTM or a GRU, and indeed the two ideas are combined in the commonly used BiLSTM and BiGRU.

The real traps are four, and the first dominates the others. Using a bidirectional network on a forecasting problem produces excellent results offline and useless ones in production: the check question is "at the moment of the real prediction will I already have this data?". In the hand-written implementation the second trap is forgetting to flip the states of the backward pass, and the two sides stay out of phase. The third is reading out[:, -1] to classify a sequence: the backward half has seen only one element, and the right representation is in h_n. The fourth concerns padding: the backward pass starts from the zeros appended at the end if you do not use pack_padded_sequence, so its initial state is already contaminated, more than in the unidirectional case.

A worked example by hand: three steps in two directions

To get a hands-on feel for the alignment, we use an RNN with a single unit with W_x = 1, W_h = 0.5 and a sequence of three numbers x = (1, 0, −1). The states are h_t = tanh(x_t + 0.5·h_{t−1}), with h_0 = 0.

Forward pass (reads 1, then 0, then −1):

→h_1 = tanh(1 + 0.5·0)       = tanh(1)       =  0.7616
→h_2 = tanh(0 + 0.5·0.7616)  = tanh(0.3808)  =  0.3634
→h_3 = tanh(−1 + 0.5·0.3634) = tanh(−0.8183) = −0.6741

Backward pass (reads −1, then 0, then 1, with equal weights for simplicity; in the real network they are different weights):

←h_3 = tanh(−1 + 0)          = −0.7616
←h_2 = tanh(0 + 0.5·(−0.7616)) = tanh(−0.3808) = −0.3634
←h_1 = tanh(1 + 0.5·(−0.3634)) = tanh(0.8183)  =  0.6741

Concatenation position by position ([→h_t ; ←h_t]). Now, for each position t, we take the forward and the backward state and put them side by side. The delicate point is that the backward states were computed in the order 3, 2, 1: here they are already put back in position order, as the second [::-1] of the code would do.

position 1:  [  0.7616,  0.6741 ]
position 2:  [  0.3634, −0.3634 ]
position 3:  [ −0.6741, −0.7616 ]

Notice two things. At position 1, the forward state has seen only "1", while the backward one has seen "1, then 0, then −1" (the whole sequence from the end): the concatenated representation contains both the past (empty) and the future. And at position 3 it is the opposite. This is the alignment that in the code is done with the second [::-1]: without flipping, the value 0.6741 would end up at position 3 instead of 1.

Frequently asked questions

Is a bidirectional network "smarter" than a unidirectional one? It has more information available (the future), not more reasoning capacity. Where the future is not needed, it adds cost (double parameters) and no benefit.

Which state do I use to classify a whole sentence? Concatenate the forward state after the last word and the backward one after the first: h_n[-2] and h_n[-1] with several layers. It is what "has read everything" in both directions.

Can you do online forecasting with a bidirectional network? Yes, but only on an already complete window and treating the prediction as "filling a gap in the middle" (for example reconstructing a missing value), not as "predicting the future". For the real future, only unidirectional.

And in modern models? The attention models used to understand text look in both directions by construction (with a mask that decides what they may see). The principle is the same: the permitted direction depends on the task, not on the architecture.

How do I test that there is no leakage in my code? With the unique-index test: build a series in which every value is different (x[i] = i) and automatically check that the target does not appear among the input elements. It must be done before training.

Does padding hurt more with a bidirectional network? Yes: the backward pass starts from the padding zeros, so its real initial state is already contaminated. pack_padded_sequence solves the problem for both directions.

What you can do now

You can build a bidirectional RNN by hand, with two sets of weights and the backward states realigned, and reproduce it with bidirectional=True reading out and h_n correctly. You have seen the difference the right context makes (accuracy from 79.1% to 100% on the neighbouring-digits task, from 71.3% to 100% on opening brackets) and you can recognise when it is forbidden: if the whole sequence does not exist at the moment of the decision, the bidirectional network gives you beautiful and false numbers. The next step is to compare RNN, LSTM and GRU in a controlled experiment, in the next lesson of the series.

In the series, the previous lesson is GRU: The Lean Version of Gated Memory; the next one is RNN, LSTM and GRU Compared: A Controlled Experiment Over Several Seeds.