65 - Hybrid Architectures: A CNN for Space and an LSTM for Time

A hybrid architecture combines blocks designed for different purposes, so that each does what it does best. In this module the first example is the most natural one: a convolutional network (CNN) that looks at each image and extracts "what is there and where", followed by a recurrent network (LSTM) that reads the sequence of these descriptions in order and derives "how it changes over time". It is the basic structure for classifying a short video, recognising a gesture, or telling whether an object is getting closer or moving away. In this lesson you build it by hand, understand why neither a CNN alone nor an LSTM alone is enough, and check with an experiment that removing the recurrence loses information in a way you can demonstrate.

The words of this lesson

A greyscale image is a grid of numbers (height × width) with a single channel; a colour image has three channels (red, green, blue). A frame is an image that is part of a video, and a clip is a sequence of frames. A convolution uses a small grid of weights, the kernel (for example 3×3), which is slid over the image: at each position you multiply the weights by the pixels under them and add up the products. The result is a new grid, the feature map, which "lights up" where the image contains the pattern the kernel is looking for, such as an edge or a corner.

Pooling reduces the resolution of a map by keeping, for instance, the maximum in each 2×2 block (max pooling): the representation becomes smaller and less sensitive to shifts of one pixel. An encoder is the part of a network that turns a large, raw input, such as an image, into a compact vector summarising its content. The five-axis tensor (B, T, C, H, W) is the typical shape of a batch of clips: batch (how many clips), time (how many frames), channels, height, width. Finally, time-distributed means applying the same network to every frame of a clip, with the same weights, and collecting the results into a sequence: it is what weight sharing over time was for the RNN.

The analogy: a photo analyst and a narrator

Imagine having to work out where a car is going from a sequence of six photographs. You hire two people. The first, the analyst, looks at one photo at a time and writes a short description on a slip of paper: "car in the middle, near the traffic light". The analyst uses the same method for every photo. The second, the narrator, does not see the photos: they receive the slips in order and read them one after the other, keeping the story so far in mind, to conclude "the car is heading right". The analyst is the CNN, the narrator is the LSTM, the slips are the vectors. If you shuffle the slips before handing them to the narrator, or compute the "average" of the descriptions, the story disappears: what is left is a list of where the car has been, not the direction it was moving in.

Convolution, with real numbers

Before assembling the parts, let us see what the block that looks at space does. Take a 4×4 image with a vertical edge (dark on the left, bright on the right) and a 2×2 kernel that responds exactly to "dark on the left, bright on the right". We slide the kernel by hand and then with the library.

import numpy as np
import torch
import torch.nn.functional as F

image = np.array([[0, 0, 1, 1],
                  [0, 0, 1, 1],
                  [0, 0, 1, 1],
                  [0, 0, 1, 1]], dtype=float)       # dark on the left, bright on the right: a vertical edge
kernel = np.array([[-1, 1],
                   [-1, 1]], dtype=float)           # responds to "left dark, right bright"

# By hand: slide the kernel, multiply element by element, sum
out = np.zeros((3, 3))
for i in range(3):
    for j in range(3):
        patch = image[i:i + 2, j:j + 2]
        out[i, j] = (patch * kernel).sum()
print(out)

# The library: same numbers (note: deep-learning "convolution" is really cross-correlation, no kernel flip)
lib = F.conv2d(torch.tensor(image).view(1, 1, 4, 4), torch.tensor(kernel).view(1, 1, 2, 2))
print(np.abs(lib.view(3, 3).numpy() - out).max())

[[0. 2. 0.]
 [0. 2. 0.]
 [0. 2. 0.]]
0.0

Explanation. The double loop takes, for each position (i, j), the 2×2 window of the image (patch), multiplies it element by element with the kernel and sums. Where the window is entirely black (columns 0-1) or entirely white (columns 2-3) the result is 0: on the left, (−1)·0 + 1·0 + (−1)·0 + 1·0 = 0. In the middle column, where the window contains two dark pixels on the left and two bright ones on the right, you get −0 + 1 − 0 + 1 = 2. The resulting map is 2 exactly at the edge: the kernel is an "edge detector". A CNN learns the kernel values from the data instead of having an engineer choose them, and stacks many layers of them. The second block of code shows that F.conv2d gives the same numbers (difference 0.0). Note: libraries call "convolution" what mathematics calls cross-correlation (the kernel is not flipped); for learning this is irrelevant, because the network learns whatever kernel it needs anyway.

Why both parts are needed

• A CNN alone sees one frame at a time. It can tell you "there is a square at (4, 9)", not "it is moving to the right": direction is a property of the difference between frames, which no single frame contains.
• An LSTM alone on the pixels would receive at each step a vector of H × W numbers (256 for a 16×16 image, 4096 for a 64×64 one), and would have to learn from scratch even what a convolution gives for free: that neighbouring pixels are correlated, and that the same pattern can appear anywhere. Too many parameters and no useful bias.
• An MLP on the whole flattened clip works on tiny problems, but the number of weights grows with frames × height × width (we measure it shortly), and it is limited to clips of exactly the length seen in training.

The three items are three different ways of going wrong: the CNN alone loses time, the LSTM on pixels throws away geometry, the flat MLP pays for every frame with new weights. The hybrid solves the problem by splitting the work: the CNN compresses each frame (256 pixels → 32 numbers), and the LSTM, now with small and informative inputs, concentrates on time.

How to read the diagram, with tensor shapes. The batch of clips has shape (B, T, 1, H, W). The CNN is designed to receive images, that is four-axis tensors (N, C, H, W): the time-distributed trick consists in "folding" time into the batch, with clip.reshape(B*T, C, H, W), that is treating each frame as an independent image. The CNN produces (B·T, F), a vector of F numbers per frame, and a second reshape(B, T, F) puts time back in its place. The LSTM thus receives (B, T, F), which is exactly what it expects with batch_first=True. The folding is correct because the two axes, batch and time, are adjacent and in that order: if instead you swapped the order with a permute without putting it back, you would mix frames from different clips without any shape error.

Code: the synthetic data

To get a result we can check, we build a task where the answer depends only on the order of the frames. A clip consists of 6 frames of 16×16; in each there is a lit 3×3 square. From one frame to the next the square moves by one pixel in one of four directions (right, left, down, up); the starting position is random. The label is the direction.

import torch
import torch.nn as nn

SIZE, T, SQ = 16, 6, 3                                   # 16x16 frames, 6 frames per clip, 3x3 square
DIRECTIONS = [(0, 1), (0, -1), (1, 0), (-1, 0)]           # (dy, dx): right, left, down, up

def make_clips(batch, seed=None):
    """Each clip: a 3x3 square moves one pixel per frame in one of 4 directions. Label = direction."""
    g = torch.Generator().manual_seed(seed) if seed is not None else None
    labels = torch.randint(0, 4, (batch,), generator=g)
    clips = torch.zeros(batch, T, 1, SIZE, SIZE)
    for b in range(batch):
        dy, dx = DIRECTIONS[labels[b]]
        # choose a start so that the square stays inside the frame for all T frames
        y0 = int(torch.randint(1, SIZE - SQ - 1 - T, (1,), generator=g)) + (T if dy < 0 else 0)
        x0 = int(torch.randint(1, SIZE - SQ - 1 - T, (1,), generator=g)) + (T if dx < 0 else 0)
        if dy == 0: y0 = int(torch.randint(1, SIZE - SQ, (1,), generator=g))
        if dx == 0: x0 = int(torch.randint(1, SIZE - SQ, (1,), generator=g))
        for t in range(T):
            y, x = y0 + dy * t, x0 + dx * t
            clips[b, t, 0, y:y + SQ, x:x + SQ] = 1.0
    return clips, labels

x, y = make_clips(4, seed=0)
print(x.shape, y.tolist())          # torch.Size([4, 6, 1, 16, 16])  one integer per clip

The make_clips function generates a batch on the fly. It draws the labels at random, and for each clip it picks a starting position far enough from the edges to allow six moves without leaving the frame (if the direction is "up" or "left", the starting position is shifted by T pixels to leave room for the movement towards the edge). Then it lights up the square in each frame. The seed argument makes it possible to generate a fixed test set, identical for all models: this is essential for a fair comparison. The last line prints the shape (4, 6, 1, 16, 16) and four labels.

Code: three architectures compared

We compare three ways of tackling the same task, all trained in the same way (600 steps, Adam, learning rate 0.003, batch 32).

class FrameEncoder(nn.Module):
    """A small CNN: one 16x16 frame -> a vector of `dim` numbers (what is where in the frame)."""
    def __init__(self, dim=32):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),     # 16 -> 8
            nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),    # 8 -> 4
        )
        self.fc = nn.Linear(16 * 4 * 4, dim)
    def forward(self, frames):                       # frames: (N, 1, 16, 16)
        return self.fc(self.conv(frames).flatten(1))

class CNNPool(nn.Module):
    """Baseline A: encode every frame, AVERAGE over time, classify. The order of frames is lost."""
    def __init__(self, dim=32):
        super().__init__()
        self.enc, self.head = FrameEncoder(dim), nn.Linear(dim, 4)
    def forward(self, clip):                         # clip: (B, T, 1, H, W)
        B, T_, C, H, W = clip.shape
        feats = self.enc(clip.reshape(B * T_, C, H, W)).reshape(B, T_, -1)   # "time-distributed" CNN
        return self.head(torch.relu(feats).mean(dim=1))

class CNNLSTM(nn.Module):
    """Hybrid: the CNN encodes each frame, an LSTM reads the sequence of vectors in order."""
    def __init__(self, dim=32, hidden=32):
        super().__init__()
        self.enc = FrameEncoder(dim)
        self.lstm = nn.LSTM(dim, hidden, batch_first=True)
        self.head = nn.Linear(hidden, 4)
    def forward(self, clip):
        B, T_, C, H, W = clip.shape
        feats = self.enc(clip.reshape(B * T_, C, H, W)).reshape(B, T_, -1)
        _, (h_n, _) = self.lstm(torch.relu(feats))
        return self.head(h_n[-1])

class FlatMLP(nn.Module):
    """Baseline B: flatten the whole clip (6*16*16 = 1536 numbers) into one big MLP."""
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(T * SIZE * SIZE, 64), nn.ReLU(), nn.Linear(64, 4))
    def forward(self, clip):
        return self.net(clip.flatten(1))

def fit(model, steps=600, seed=0):
    torch.manual_seed(seed)
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    for _ in range(steps):
        x, y = make_clips(32)
        loss = nn.functional.cross_entropy(model(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        x, y = make_clips(1000, seed=123)                # a FIXED test set
        return (model(x).argmax(-1) == y).float().mean().item()

for name, model in [("CNN + mean over time", CNNPool()), ("CNN + LSTM", CNNLSTM()), ("flat MLP", FlatMLP())]:
    n_par = sum(p.numel() for p in model.parameters())
    print(f"{name:22s} params {n_par:6d}  test accuracy {fit(model):.3f}")

Explanation of the three networks. FrameEncoder is a small CNN: two 3×3 convolutions with ReLU (the ReLU, max(0, x), is the activation that zeroes out negative values) and max pooling that take the frame from 16×16 down to 4×4 with 16 channels, then a linear layer that reduces it to 32 numbers. CNNPool is baseline A: it encodes every frame and then takes the mean over time before classifying; it is the approach of someone who says "I aggregate the information from the frames" without thinking about order. CNNLSTM is the hybrid: the same CNN, then an LSTM that reads the six vectors in order, and the classification uses the last hidden state h_n[-1]. FlatMLP is baseline B: it flattens the whole clip into 1536 numbers and uses a dense network. The fit function is the usual loop; the test set is fixed (seed=123) and has 1000 clips.

CNN + mean over time   params   9604  test accuracy 0.378
CNN + LSTM             params  18052  test accuracy 1.000
flat MLP               params  98628  test accuracy 1.000

What the numbers say

The mean over time cannot cope. Accuracy is about 0.37: 0.378 in the run reported above, 0.372 and 0.355 in two other runs of the same code with the same seed (this model is not reproducible to the third decimal, whereas CNN + LSTM and the flat MLP gave 1.000 in every run). With 2000 steps instead of 600 and three seeds you get 0.349, 0.372 and 0.389: longer training does not help. It is not a training defect: it is a limit of the information.

A clip going right visits the positions x₀, x₀+1, …, x₀+5; a clip going left that starts from x₀+5 visits the same positions in reverse order. The mean over time of the vectors does not tell one ordering from the other, so right and left (and likewise up and down) produce the same mean when the positions visited coincide. In our generator the starting positions for "left" are shifted by T pixels, so the distributions do not coincide exactly, and the ceiling of about 0.5 (you guess the axis, you toss a coin for the direction) is an approximation of the theoretical limit, not a measured value. The model at about 0.37 is nonetheless well below it: it has not learned even the axis properly.

CNN + LSTM solves it with 18,052 parameters and accuracy 1.000. The LSTM sees the vectors in order and can build the information "the position is increasing".

The flat MLP solves it too, but with 98,628 parameters, about five times as many. On a tiny task with fixed-length clips, brute force works. The problem is what happens when the task grows:

def params_flat_mlp(frames, side, hidden=64, classes=4):
    return frames * side * side * hidden + hidden + hidden * classes + classes

def params_cnn_lstm(frames, side, dim=32, hidden=32, classes=4):
    conv = (1 * 8 * 3 * 3 + 8) + (8 * 16 * 3 * 3 + 16)       # two 3x3 convolutions
    fc = 16 * (side // 4) ** 2 * dim + dim                      # after two 2x2 poolings the map is side/4
    lstm = 4 * hidden * (dim + hidden + 2)
    head = hidden * classes + classes
    return conv + fc + lstm + head                              # NOT dependent on the number of frames

for frames, side in ((6, 16), (12, 32), (24, 64)):
    print(f"{frames:2d} frames of {side}x{side}:  flat MLP {params_flat_mlp(frames, side):9,d}   CNN+LSTM {params_cnn_lstm(frames, side):8,d}")

 6 frames of 16x16:  flat MLP    98,628   CNN+LSTM   18,052
12 frames of 32x32:  flat MLP   786,756   CNN+LSTM   42,628
24 frames of 64x64:  flat MLP 6,291,780   CNN+LSTM  140,932

The two functions count the parameters with the formula. In the MLP the first layer has frames × side² × hidden weights: doubling the frames and the side (from 6×16 to 12×32) multiplies the weights by eight, and at 24 frames of 64×64 you reach 6.3 million weights for a single layer of 64 units. In the CNN+LSTM the weights of the convolutions and of the LSTM depend neither on the number of frames nor on the frame size; only the linear layer that reduces the final map depends on the size, and it grows with the square of the side divided by 4 (not with the frames). At 24 frames of 64×64 the CNN+LSTM has 140,932 parameters against 6,291,780: forty-four times fewer. Moreover a CNN+LSTM accepts clips of any length without changing a weight, whereas the MLP does not.

Alternatives to the CNN + LSTM pair

• 3D convolutions (nn.Conv3d): the kernel slides over time as well and picks up small movements directly. More parameters per kernel and a fixed time window, but no recurrence.
• ConvLSTM (next lesson): instead of compressing each frame into a vector before the LSTM, it keeps a state that is still a spatial map and uses convolutions in the gates. It preserves geometry in the memory.
• Pretrained CNN as encoder: in place of the small CNN you can use a network already trained on millions of images (and perhaps freeze it): the LSTM learns only the temporal part. Useful with little data.
• Attention over time: in place of the LSTM, an attention model over the frame vectors. Outside the scope of this series.

The four alternatives move the cost to different places. 3D convolution and ConvLSTM merge space and time in a single operator, the pretrained CNN moves the cost to training already done by others, attention changes the whole way the sequence is read. That is why the CNN + LSTM pair remains the didactic starting point: each piece can be isolated, measured and replaced.

Trade-offs

• Compression versus detail. Reducing each frame to a single vector is cheap but throws away fine geometry: if you need to know exactly where, a vector of 32 numbers may not be enough. This is the historical reason for the ConvLSTM.
• Cost. The CNN is applied to B × T images: memory and time grow with the number of frames. With real videos this is often the main constraint.
• A single encoder for all frames is efficient (shared weights), but assumes that every frame should be described in the same way.

The three compromises should be read together: one vector per frame lowers the cost but sacrifices fine position, and the cost of the CNN grows anyway with the number of frames, because the weights are shared but the computations are not. If your problem has few frames and you need to know where, compression is the weak point; if it has many frames, the weak point is the bill for B × T images.

Real traps

The most insidious trap is the wrong reshape: clip.reshape(B*T, C, H, W) is correct only if the clip has shape (B, T, C, H, W). If by mistake it has shape (T, B, ...), the code runs all the same and the network mixes frames from different clips; check with a test that takes a known clip and verifies that the reconstructed frames are the same. The second concerns BatchNorm with the frames folded into the batch: the statistics are computed over B × T frames that are correlated with one another, not over independent images, and this can harm training (the technique is described in the paper by Ioffe and Szegedy cited in the sources).

The third is choosing a task that does not require time: if the answer can be read from a single frame, every architecture works and you have tested nothing about the hybrid, whereas our task is built so that the answer lies in the order. The fourth is concluding too much from 100%: a synthetic task this simple is solved by the MLP as well as by the hybrid, so it says that the hybrid can learn order and costs much less as the problem grows, not that it will be better on real videos. The last is using a different test set for each model: differences in accuracy can be noise, and the test set is fixed with a seed, as here.

Questions to check yourself (with answers)

Why can the mean over time of the vectors not tell right from left? The mean is invariant to the order of the frames; right and left produce the same frames in reverse order, so (for the same positions visited) the same mean.

What shape does the tensor have after reshape(B*T, C, H, W) if B = 8, T = 6, C = 1, H = W = 16? (48, 1, 16, 16).

A convolution with a 3×3 kernel on a 1-channel image with 8 output filters: how many parameters? 8 · (1·3·3) + 8 = 80, that is the weights plus one bias per filter.

If you double the number of frames, do the parameters of the CNN+LSTM change? No. The computation time and the memory change, not the parameters.

A worked example by hand: the shapes and parameters of our model

Let us follow a batch of B = 8 clips of T = 6 frames of 16×16 through the CNNLSTM, computing the shapes and counting the parameters.

input                        (8, 6, 1, 16, 16)
reshape(B·T, C, H, W)        (48, 1, 16, 16)       48 independent images
Conv2d(1→8, 3×3, pad 1)      (48, 8, 16, 16)       parameters 1·8·9 + 8      =    80
MaxPool2d(2)                 (48, 8, 8, 8)
Conv2d(8→16, 3×3, pad 1)     (48, 16, 8, 8)       parameters 8·16·9 + 16    = 1,168
MaxPool2d(2)                 (48, 16, 4, 4)
flatten                      (48, 256)             16·4·4 = 256
Linear(256→32)               (48, 32)              parameters 256·32 + 32    = 8,224
reshape(B, T, F)             (8, 6, 32)
LSTM(32→32)                  h_n: (1, 8, 32)       parameters 4·32·(32+32+2) = 8,448
Linear(32→4)                 (8, 4)                parameters 32·4 + 4       =   132

The sum is 80 + 1,168 + 8,224 + 8,448 + 132 = 18,052, exactly the number printed by the code. This hand count is useful in two ways: it checks that you have understood the shapes (if the count does not add up, something in the code is not what you think) and it predicts the cost: the 48 passes through the CNN are the real computational cost, not the LSTM.

A rule for the spatial dimensions: with a 3×3 kernel, padding = 1 and stride 1 the size does not change; with a MaxPool2d(2) it is halved (rounding down). In general, for a convolution the output is ⌊(N + 2·pad − kernel) / stride⌋ + 1: with N = 16, pad = 0, kernel = 3 you get 14.

Frequently asked questions

Conv3d or CNN + LSTM? Conv3d is simpler if the movements are short and the length is fixed; CNN + LSTM handles variable lengths and long dependencies over time. We have not compared them with a measurement, so we cannot say which wins in general.

Can I use a pretrained CNN? Yes, and it is common: you replace FrameEncoder with a network already trained on many images (freezing it or tuning it with a small learning rate), and train only the temporal part. It matters above all when you have few videos.

Which frame rate should I choose? The one that makes the movement you care about visible: if the phenomenon changes in 0.5 s, sampling at 1 Hz skips it. On the other hand, frames that are too close together give a lot of redundancy and little signal.

What if the videos have different lengths? The LSTM accepts them; the discussion of padding (pack/unpack) from lesson 60 applies. If instead you sample them to a fixed length (for example 16 frames per clip), you need to decide how: uniformly over the whole duration or a window.

Why the last hidden state and not the mean of the states? The last state has read the whole clip; the mean would repeat the mistake of CNNPool, losing order (although here it would be a mean of states that have already encoded the order, so less serious). It is a design choice and you can try it.

Is the 100% result too good? The task is deliberately simple (a single square, no background, clear directions), and that is why an MLP solves it. The point is not the difficulty but the parameter count and the demonstration of order; on real videos you expect much lower performance, which has to be measured.

Exercises with solutions

Before moving on, try to solve these exercises with pencil and paper; the solutions (checked with the code) are right below each one.

Exercise 1. A 28×28 image, 5×5 convolution without padding with stride 1: output size? And with stride 2? Solution: 28 − 5 + 1 = 24; with stride 2: ⌊(28 − 5)/2⌋ + 1 = 12.

Exercise 2. Parameters of nn.Conv2d(3, 16, 3) (with bias)? Solution: 3·16·9 + 16 = 448.

Exercise 3. A flat MLP on 10 frames of 28×28 with 128 hidden units, first layer only (with bias): how many parameters? Solution: 10·784·128 + 128 = 1,003,648, more than a million for a single layer.

Exercise 4. Why can the mean over time not tell right from left? Solution: because the frames of a rightward clip and of a leftward one are the same in reverse order, and the mean does not depend on order.

Takeaways

A hybrid architecture divides the work: the CNN answers "what and where in this frame", the LSTM "how it changes over time". The technical trick is to fold time into the batch to apply the same CNN to all frames and then put it back. The experiment showed three measured things: the mean over time cannot exceed about 0.5 because it loses order (about 0.37 obtained), the CNN+LSTM solves the task (1.000) with 18 thousand parameters, and the flat MLP solves it but with five times the parameters, a factor that rises to about forty-four at 24 frames of 64×64. In the next lesson a more elegant variant: bringing the convolutions inside the recurrent cell.

In the series, the previous lesson is Generating Text Character by Character with an LSTM; the next one is The ConvLSTM Cell: A Memory That Keeps Geometry.