67 - ConvLSTM in Practice: Predicting the Next Frame of Moving Digits

In the previous lesson you built the ConvLSTM cell and measured its properties. Now we put it to work on the task it was invented for: predicting the next frame of a video. To keep everything reproducible on a laptop we use a simplified version of a classic benchmark, "Moving MNIST": a handwritten digit bounces around inside a canvas, and the network, after seeing the frames up to time t, must draw frame t + 1. We will work out what "learning" means in this setting by comparing against a baseline that is harder to beat than it looks (copying the last frame), we will measure what happens with speeds the network has never seen and when it has to predict several steps ahead by feeding on its own predictions, and we will see why the images it produces come out blurry.

Every figure in this lesson comes from a run of the code you find in the text, repeated twice with identical output (torch and numpy seeds fixed, 2 CPU threads, no GPU). The digits are not those of MNIST: downloading it requires torchvision and a connection, so we use the 1797 8×8 digits of sklearn.datasets.load_digits, which already ship inside scikit-learn. They are small and noisy, but enough to show every phenomenon, and the numbers cannot be compared with those published on MNIST. With a different seed the values will change by a few percentage points.

The words of this lesson

Next-frame prediction is the task: given a movie up to instant t, produce the image of instant t + 1. It is self-supervised learning, because the label is the next frame, which is already in the video and needs no human annotation. Teacher forcing means giving the network, during training, the true frame at time t and not its own previous prediction: it makes training stable and parallelisable over sequences. The rollout (or autoregressive prediction) is the opposite use: to predict several steps ahead you feed the network its own prediction as the input of the next step, something it never saw in training, and the gap between the two regimes is called exposure bias, because errors accumulate without the network having learned to correct them.

The "copy the last frame" baseline (persistence) predicts that frame t + 1 equals frame t: like persistence in time series, it is simple and often hard to beat when the motion is slow. The pixel MSE (mean squared error) is the mean of the squared differences between predicted and true image, and it is the loss we will use. Moving MNIST is the name of a benchmark introduced by Srivastava and colleagues (2015) for unsupervised learning of video representations with LSTMs, in which MNIST digits move on a canvas; the version here is a variant of our own and much smaller (a single 8×8 digit on a 16×16 canvas), not the original dataset.

The analogy: guessing where the ball will be

You are watching a tennis match and someone covers your eyes for an instant: where will the ball be? If you know nothing, you answer "where it was a moment ago": that is the "copy the last frame" baseline, and if the ball moves slowly, you are off by a little. If instead you have watched enough frames to understand direction and speed, you can say "a bit more to the right and down". That "a bit more" is what the network must learn: the difference between "copy" and "predict" lies entirely in the motion. And there is a second lesson: if you are not sure whether the ball will bounce left or right, the answer that minimises the average error is a ball that is "half left and half right", that is, blurry. We will see this happen.

Code: the data generator

We do not download a ready-made video dataset: we generate one. This has an important advantage for learning: we control the speed, the position and the set of digits used, and we can create an unlimited number of new examples (no risk of memorisation). The 1797 digits of load_digits have values from 0 to 16, which we divide by 16 to bring them into [0, 1], and we split them into 1257 digits for training and 540 for testing (stratified train_test_split, seed 0): the test digits never appear in the training videos, so the network cannot memorise their shapes.

The digit moves at constant speed and bounces off the edges. The bounce is computed with a closed formula: the "free" position is p₀ + v·t; take it modulo the period 2·L (out and back over a stretch of length L) and, if it exceeds L, reflect it (2L − p). An example with numbers: the digit is 8×8 on a 16×16 canvas, so L = 16 − 8 = 8 pixels of travel; start at 3, speed 3. The positions in the first seven instants are 3, 6, 7, 4, 1, 2, 5. At the third instant the free position would be 9, beyond the edge, and it is reflected to 16 − 9 = 7; from there the digit comes back.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

torch.manual_seed(0)
torch.set_num_threads(2)

CANVAS, DIGIT = 16, 8                                     # 16x16 frames, digits stay 8x8

X, y = load_digits(return_X_y=True)                       # 1797 digits, 8x8, values 0..16
X = (X / 16.0).reshape(-1, DIGIT, DIGIT).astype(np.float32)
train_digits, test_digits = train_test_split(X, test_size=0.3, stratify=y, random_state=0)
print("digits:", train_digits.shape[0], "train /", test_digits.shape[0], "test")

def bounce(p0, v, t, limit):
    """Position of a point moving with speed v from p0, reflected inside [0, limit]."""
    p = p0 + v * t
    period = 2 * limit
    p = np.mod(p, period)
    return np.where(p > limit, period - p, p)

def make_batch(digits, batch, T, max_speed=2, rng=None, speeds=None):
    rng = rng or np.random.default_rng()
    idx = rng.integers(0, len(digits), batch)
    limit = CANVAS - DIGIT
    x0 = rng.integers(0, limit + 1, (batch, 2)).astype(float)
    if speeds is None:
        v = rng.integers(-max_speed, max_speed + 1, (batch, 2)).astype(float)
    else:                                                  # force the speed components to come from `speeds`
        v = rng.choice(speeds, (batch, 2)).astype(float)
    v[(v == 0).all(axis=1)] = 1.0                          # never a still digit
    frames = np.zeros((batch, T, 1, CANVAS, CANVAS), dtype=np.float32)
    for t in range(T):
        pos = bounce(x0, v, t, limit).round().astype(int)  # (batch, 2) -> (row, col)
        for b in range(batch):
            r, c = pos[b]
            frames[b, t, 0, r:r + DIGIT, c:c + DIGIT] = digits[idx[b]]
    return torch.from_numpy(frames)

clip = make_batch(train_digits, 4, 10, rng=np.random.default_rng(0))
print(clip.shape)
print("bounce start 3, speed 3, limit 8:", bounce(3, 3, np.arange(7), 8).tolist())
print("bounce start 5, speed 4, limit 10:", bounce(5, 4, np.arange(4), 10).tolist())

# digits: 1257 train / 540 test
# torch.Size([4, 10, 1, 16, 16])
# bounce start 3, speed 3, limit 8: [3, 6, 7, 4, 1, 2, 5]
# bounce start 5, speed 4, limit 10: [5, 9, 7, 3]

Explanation. load_digits returns the digits as vectors of 64 values; the reshape brings them back to 8×8. bounce is the reflection formula, vectorised with NumPy (np.mod, np.where). make_batch randomly picks batch digits, a starting position and a speed for each component (row and column) from −max_speed to +max_speed; if the speed happens to be (0, 0) it changes it to (1, 1) to avoid still digits. Then, for each instant, it computes the position with bounce and "pastes" the digit onto the canvas. The speeds argument is used later to force the speeds to a chosen set. The output confirms the shape, four clips of ten 16×16 greyscale frames, and the two bounce examples, the one in the text and the one in the check at the end of the lesson. The generator seed is explicit (default_rng(0)): without it, two runs would give different videos and different digits.

Code: the model

The model stacks two ConvLSTM cells (the one from the previous lesson, rewritten here in compact form) and adds a 1×1 convolution that turns the state of the last layer into a single image, with a sigmoid to keep the pixels in [0, 1]. On a canvas this small 16 channels per layer are enough, where with 32×32 images we would have chosen more. The forget-gate bias starts at 1, as in the LSTM lessons, so that the memory is not wiped out at the start.

class ConvLSTMCell(nn.Module):
    def __init__(self, in_ch, hidden_ch, kernel=3):
        super().__init__()
        self.hidden_ch = hidden_ch
        self.conv = nn.Conv2d(in_ch + hidden_ch, 4 * hidden_ch, kernel, padding=kernel // 2)
        with torch.no_grad():
            self.conv.bias[hidden_ch:2 * hidden_ch] = 1.0
    def forward(self, x, state):
        h, c = state
        i, f, g, o = torch.chunk(self.conv(torch.cat([x, h], dim=1)), 4, dim=1)
        c = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)
        h = torch.sigmoid(o) * torch.tanh(c)
        return h, c

class NextFrameNet(nn.Module):
    """Two stacked ConvLSTM layers + a 1x1 convolution that turns the state into a predicted frame."""
    def __init__(self, hidden=16, kernel=3):
        super().__init__()
        self.cells = nn.ModuleList([ConvLSTMCell(1, hidden, kernel), ConvLSTMCell(hidden, hidden, kernel)])
        self.head = nn.Conv2d(hidden, 1, 1)
        self.hidden = hidden

    def init_state(self, B, H, W):
        z = lambda: torch.zeros(B, self.hidden, H, W)
        return [(z(), z()) for _ in self.cells]

    def step(self, frame, states):
        x = frame
        new = []
        for cell, st in zip(self.cells, states):
            h, c = cell(x, st)
            new.append((h, c)); x = h
        return torch.sigmoid(self.head(x)), new               # predicted NEXT frame, pixels in [0, 1]

    def forward(self, clip):                                  # clip: (B, T, 1, H, W) -> predictions for frames 1..T-1
        B, T, _, H, W = clip.shape
        states = self.init_state(B, H, W)
        preds = []
        for t in range(T - 1):
            p, states = self.step(clip[:, t], states)         # sees frames 0..t (the TRUE ones), predicts t+1
            preds.append(p)
        return torch.stack(preds, dim=1)                      # (B, T-1, 1, H, W)

How to read it. ConvLSTMCell is the same cell as before. NextFrameNet keeps two of them in a ModuleList: the first receives the frame (1 channel), the second receives the states of the first (16 channels). init_state creates the zero initial states with the right shape (B, 16, H, W) for each layer. step is the heart: it passes one frame through the two layers, updating the states, and returns the prediction of the next frame and the new states. forward is training with teacher forcing: for t from 0 to T − 2 it passes the true frame clip[:, t] to step and collects the prediction of frame t + 1. The parameters can be counted by hand: the first layer has a 3×3 convolution from 17 channels (1 + 16) to 64 (4 × 16), that is 17·64·9 + 64 = 9856; the second from 32 to 64, that is 32·64·9 + 64 = 18,496; the 1×1 head has 16 + 1 = 17. Total 28,369, few: the ConvLSTM shares its weights across space.

Training and baseline

Before training we write down the numbers to beat on a fixed validation set: the MSE of the "copy the last frame" baseline, F.mse_loss(val[:, :-1], val[:, 1:]), that is the error you get by predicting every frame equal to the previous one, and that of an even more naive second baseline, "all black". The validation set uses 64 clips made with the test digits and a generator with seed 123; the training batch is fresh at every step, with its own generator at seed 0.

model = NextFrameNet()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
print("parameters:", sum(p.numel() for p in model.parameters()))

rng = np.random.default_rng(0)
with torch.no_grad():
    val = make_batch(test_digits, 64, 10, rng=np.random.default_rng(123))
    copy_mse = F.mse_loss(val[:, :-1], val[:, 1:]).item()
    zero_mse = F.mse_loss(torch.zeros_like(val[:, 1:]), val[:, 1:]).item()
print(f"copy-last-frame MSE: {copy_mse:.5f}   all-black MSE: {zero_mse:.5f}")

STEPS = 1200
for step in range(1, STEPS + 1):
    clip = make_batch(train_digits, 16, 10, rng=rng)
    pred = model(clip)
    loss = F.mse_loss(pred, clip[:, 1:])
    opt.zero_grad(); loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
    if step % 200 == 0:
        with torch.no_grad():
            v = F.mse_loss(model(val), val[:, 1:]).item()
        print(f"step {step:4d}  train {loss.item():.5f}  val {v:.5f}  ratio {v / copy_mse:.2f}")

# parameters: 28369
# copy-last-frame MSE: 0.06409   all-black MSE: 0.05928
# step  200  train 0.05087  val 0.05462  ratio 0.85
# step  400  train 0.05618  val 0.05268  ratio 0.82
# step  600  train 0.03449  val 0.03602  ratio 0.56
# step  800  train 0.02932  val 0.03171  ratio 0.49
# step 1000  train 0.02845  val 0.03061  ratio 0.48
# step 1200  train 0.02782  val 0.02880  ratio 0.45

Reading the loop: at every step a fresh batch of 16 clips of 10 frames is generated, the network predicts frames 1…9 from the true frames 0…8, the MSE against the true frames is computed, then backward, gradient clipping (BPTT lesson) and step. With 1200 steps the experiment finishes in a few minutes on CPU; with a free machine and more steps the curve would keep falling, so the final value is not a limit of the model. The train column is the loss of the single batch at that step, noisy; the val column is on the fixed set, and it is the one to read.

What the numbers say. The surprising fact comes before training: the "copy" baseline is 0.06409 and it is worse than predicting an all-black image (0.05928). The reason: a digit on a black background that moves leaves two errors, the pixels you predicted lit that no longer are and those you predicted dark that have lit up, while the black canvas gets only the second kind wrong. With small digits on an empty background, then, beating the copy is not enough: you should compare with black too.

Between steps 200 and 400 the network sits at 0.0546 and 0.0527, just below black: it has learned where the brightness is, on average, not the motion. Then, between steps 400 and 600, the error drops to 0.0360: this is the moment it has "latched onto" the dynamics. From there it falls slowly down to 0.02880 at 1200 steps, that is 45% of the "copy" baseline and 49% of black. It recalls the plateau of the LSTM lesson: the model spends a long stretch in which the only strategy available is to paint a grey smudge, and then it discovers the structure.

At which speeds does it work? What happens to unseen ones

In training the speeds are components in {−2, …, +2}. We evaluate the model separately on three sets: small speeds (seen), speeds up to 2 (seen), and ±3 speeds never seen. With a travel of only 8 pixels, ±3 makes the digit bounce already at the third instant, one more difficult case.

@torch.no_grad()
def one_step_mse(model, digits, speeds, n=64, seed=7):
    clip = make_batch(digits, n, 10, rng=np.random.default_rng(seed), speeds=speeds)
    return F.mse_loss(model(clip), clip[:, 1:]).item(), F.mse_loss(clip[:, :-1], clip[:, 1:]).item()

for label, speeds in [("speeds 0,+-1 (seen)", [-1, 0, 1]), ("speeds 0,+-2 (seen)", [-2, 0, 2]), ("speeds +-3 (NEVER seen)", [-3, 3])]:
    m, c = one_step_mse(model, test_digits, speeds)
    print(f"{label:26s} model {m:.5f}   copy-last {c:.5f}   ratio {m / c:.2f}")

# speeds 0,+-1 (seen)        model 0.02276   copy-last 0.04590   ratio 0.50
# speeds 0,+-2 (seen)        model 0.03164   copy-last 0.06805   ratio 0.46
# speeds +-3 (NEVER seen)    model 0.06369   copy-last 0.09437   ratio 0.67

Explanation of the code: one_step_mse generates clips with speeds forced from the speeds set and returns the MSE of the model and that of the baseline on the same batch (fixed seed, hence comparable). The model/copy ratio is the useful measure: the lower it is, the better the network beats the copy. With seen speeds the ratio is 0.50 and 0.46: the model more than halves the error (the two figures are too close, with a single seed, to say which speed is easier). With ±3 speeds, never seen, it rises to 0.67: the model still beats the copy, but much less. Note that the baseline itself gets worse (0.0944) because faster motion makes the copy more wrong.

Two causes are mixed up in this result: the network never saw speed 3 in training, and the receptive field is limited (previous lesson: two layers with 3×3 kernels, a radius of the order of 2 pixels per step, plus the memory). To separate them you would have to retrain including speed 3 and see whether the ratio goes back towards 0.46; that was not done, so the degradation must not be attributed to a single cause. The practical rule is clear anyway: evaluate the model on the speeds it will meet in production, and check that they are within the training range.

Predicting several steps: the rollout

So far the network has always received true frames. In real use, to predict five steps ahead, we give it five true frames and then let it carry on alone, feeding it its own prediction at every step. The rollout function does exactly this, and we compare it with the "repeat the last real frame" baseline (frame 4) held still over the whole horizon.

@torch.no_grad()
def rollout(model, clip, n_context=5, n_future=5):
    """Read n_context TRUE frames, then keep predicting by feeding the model its own predictions."""
    B, T, _, H, W = clip.shape
    states = model.init_state(B, H, W)
    for t in range(n_context):                         # warm-up on real frames
        pred, states = model.step(clip[:, t], states)
    outs = [pred]                                      # prediction of frame n_context
    for _ in range(n_future - 1):
        pred, states = model.step(pred, states)        # feed the prediction back in
        outs.append(pred)
    return torch.stack(outs, dim=1)

clip = make_batch(test_digits, 64, 10, rng=np.random.default_rng(7))
pred = rollout(model, clip)
for k in range(5):
    target = clip[:, 5 + k]
    print(f"horizon {k + 1}: model {F.mse_loss(pred[:, k], target).item():.5f}   copy-last-real-frame {F.mse_loss(clip[:, 4], target).item():.5f}")

# horizon 1: model 0.02894   copy-last-real-frame 0.05987
# horizon 2: model 0.04102   copy-last-real-frame 0.08599
# horizon 3: model 0.04465   copy-last-real-frame 0.09307
# horizon 4: model 0.04578   copy-last-real-frame 0.09980
# horizon 5: model 0.04837   copy-last-real-frame 0.09889

The function first passes n_context = 5 true frames to step to "warm up" the states; then, for each future step, it calls step again with the previous prediction (pred) in place of a true frame. Result: the model's error grows from 0.0289 at horizon 1 to 0.0484 at horizon 5 (about 1.7 times), but the copy's error grows too, from 0.0599 to 0.0989, because frame 4 recedes in time. The ratio between the two stays around 0.5 (0.48 at horizon 1, 0.49 at horizon 5): in this configuration the error does not explode and the advantage over the copy is not eaten away. The first-step prediction (0.0289) matches the one-step validation MSE (0.0288), as expected.

This contradicts an easy reading of the phenomenon, the one in which exposure bias would always show up as a collapse in accuracy. Here the network, trained only on true inputs, holds up through five rollout steps without degrading relative to the baseline, probably because the videos are simple (one digit, constant speed) and because from an already blurred prediction it can still extract enough information about the motion. The sign of the problem appears elsewhere, in the images, as we will see in the next section. Our experiment does not separate exposure bias from the plain fact that predicting far ahead is harder; to isolate it you would have to compare, on the same target frame, the prediction with true inputs and the one with generated inputs. The known countermeasures, which we do not apply here, are to train also with inputs generated by the model in part of the steps (scheduled sampling), or to use a loss over several steps ahead.

Why the images come out blurry

We measure sharpness: the mean squared difference between neighbouring pixels (a low value means a "soft" image, a high one means crisp edges). We compare the true frames with the predictions.

def sharpness(img):
    """Mean squared difference between neighbouring pixels: low value = blurry picture."""
    dx = (img[..., :, 1:] - img[..., :, :-1]).pow(2).mean()
    dy = (img[..., 1:, :] - img[..., :-1, :]).pow(2).mean()
    return (dx + dy).item()

with torch.no_grad():
    one_step = model(clip)                            # (B, 9, 1, H, W): predictions of frames 1..9
print("sharpness of real frames       :", round(sharpness(clip[:, 1:]), 4))
print("sharpness of 1-step predictions:", round(sharpness(one_step), 4))
print("sharpness at rollout horizon 1 :", round(sharpness(pred[:, 0]), 4))
print("sharpness at rollout horizon 5 :", round(sharpness(pred[:, 4]), 4))
print("max pixel value: real", round(clip[:, 5:].max().item(), 2), " rollout h5", round(pred[:, 4].max().item(), 2))

# sharpness of real frames       : 0.0777
# sharpness of 1-step predictions: 0.012
# sharpness at rollout horizon 1 : 0.013
# sharpness at rollout horizon 5 : 0.006
# max pixel value: real 1.0  rollout h5 0.58

The sharpness function sums the squared difference between horizontally and vertically adjacent pixels. The true frames have sharpness 0.0777, the one-step predictions 0.012: about six times blurrier. Here the accumulation of errors is visible: at horizon 5 the sharpness falls to 0.006, half that of horizon 1, and the brightest pixel reaches only 0.58 against 1.0 in the real frames. Relative to the copy the network does not lose ground, but it draws an ever paler smudge.

The explanation is mathematical and deserves an example. Suppose that, for a given pixel, the network knows only that in 50% of cases the digit will be there (pixel = 1) and in 50% it will not (pixel = 0). Which value minimises the expected squared error? Call the prediction v: the expected error is 0.5·(1 − v)² + 0.5·(0 − v)², which has its minimum at v = 0.5. The network, that is, does not choose between the two possible futures: it draws their average, a grey pixel. Repeated over all the uncertain pixels, the result is a soft image. It is not a bug in the code, it is a property of the MSE loss when the future is ambiguous; here the ambiguity is real, because the starting position and the speed are not known with certainty from the first frames. Remedies that exist (not tried here): more "perceptual" losses, adversarial losses (GANs), or probabilistic models that sample one specific future; we will see one, the VAE, in the following lessons.

Trade-offs

• One strong baseline is good, two are better. The 0.45 ratio against the copy is an interpretable result; a "low" MSE without a baseline would not be. With small digits on a black canvas the all-black image already has an MSE of 0.059, lower than that of the copy.
• Capacity versus memory. More channels (16 here) improve quality but multiply memory and time; we chose small so we could train on CPU.
• Teacher forcing versus rollout. The former stabilises and speeds up training; the latter is what you need in use. The distance between the two (exposure bias) is a real cost, which here showed up in the sharpness more than in the MSE.
• MSE versus sharpness. Optimising the pixel-by-pixel error favours average, blurry images.

The four items have a common root: the choice that is most convenient for training (a simple loss, true frames as input, a small model) is the one that pushes the cost onto real use, where long horizons and sharp images are needed. The ratio against the baseline is the tool that keeps you from missing it, provided the baseline is the right one.

Real pitfalls

The first pitfall is not having the "copy" baseline and trusting only one: with slow motion and an empty background the MSE of a mediocre network looks excellent, and here the copy loses even against black, so always write both numbers before training. The second is evaluating only with teacher forcing: the one-step MSE does not tell you what the images look like five steps out (here the error rises 1.7 times and the sharpness halves), and if your use is predicting five steps you must measure the five-step rollout. The third is testing only on the training speed distribution: a change of speed in production degrades the model (from a ratio of 0.46-0.50 to 0.67 with ±3 speeds), so build stress tests with different speeds, positions and noise.

The fourth is judging from plots: blurry images can "look" plausible, and it is better to measure numerically (here the sharpness). The fifth is mistaking the plateau for a failure: up to step 400 the model seemed not to learn (ratio 0.82-0.85), and training must be viewed on a suitable scale, comparing with the baselines. The sixth is information leakage between train and test: the test digits must differ from the training ones (here 30% of load_digits is held out before generating the videos), otherwise the model can memorise the shapes. The last is relying on a single seed: small differences, like 0.46 against 0.50, are not significant.

Questions to test yourself (with answers)

Why does training use true frames as input at every step? It is teacher forcing: it stabilises and lets you compute all the predictions together; the price is exposure bias in use.

If a pixel is 1 in 30% of the possible futures and 0 in 70%, which prediction minimises the MSE? The mean, 0.3: the expected error 0.3(1 − v)² + 0.7v² has its minimum at v = 0.3.

The model has MSE 0.0288, the copy 0.0641. What is the ratio and what does it mean? 0.0288 / 0.0641 ≈ 0.45: the model makes 45% of the copy's error.

With the bounce formula, starting from 5 with speed 4 and limit 10, where is the digit at times 0, 1, 2, 3? Free positions 5, 9, 13, 17; period 20; reflected: 5, 9, 20−13 = 7, 20−17 = 3 (this is the second line of the generator's output).

A worked example by hand: the copy, the model and the blur on four pixels

Let us do the arithmetic on a "frame" of just four pixels. At time t the digit occupies the pixels [0, 1, 1, 0]; at time t + 1 it has moved one pixel to the right: [0, 0, 1, 1].

The "copy" baseline. Predicting [0, 1, 1, 0] for the next frame gives differences [0, 1, 0, 1], hence MSE (0 + 1 + 0 + 1) / 4 = 0.5. The all-black frame [0, 0, 0, 0] gets only the two lit pixels wrong: MSE (0 + 0 + 1 + 1) / 4 = 0.5, equal to the copy. If instead the shift is two pixels and the digit does not overlap its old position ([1, 1, 0, 0] against [0, 0, 1, 1]), the copy gets all four pixels wrong (MSE 1) and black only two (MSE 0.5): this is why, in the experiment, the copy loses against black. A model that has understood the motion (one pixel to the right) predicts [0, 0, 1, 1] and gets MSE 0. Our network reduced the mean MSE from 0.0641 to 0.0288 (45%): it did not reach zero because real digits are more complex and the movements vary.

The blur. Suppose the model does not know whether the digit will go right ([0, 0, 1, 1]) or left ([1, 1, 0, 0]), with equal probability. What should it predict, pixel by pixel? For the first pixel the true value is 0 or 1 with probability ½: predicting p gives an expected error ½(1 − p)² + ½p². For p = 0.5 it is 0.25 in every case; for p = 1 it is 0.5 on average (0 if right, 1 if wrong); for p = 0 likewise. The prediction that minimises the expected error is therefore the mean, [0.5, 0.5, 0.5, 0.5]: a uniform grey frame, which resembles neither of the two possible futures. With the MSE the model "hedges its bets", and the result is a blurry image. To get sharp images you have to change the question: not "what is the average of the futures?" but "sample one plausible future" (probabilistic models, VAEs, adversarial losses).

Frequently asked questions

Why evaluate with a "copy" baseline if the network is obviously better? Because "obviously" is not a measurement. With a black background and small digits a network that predicts almost everything black already has a low MSE (0.0593 here, less than the copy); only the ratio against the baselines tells you how much the network understands of the motion.

How many steps are needed? Here the model left the plateau between 400 and 600 steps and kept improving up to 1200. With more data or larger digits you need more; you look at the validation curve, not a fixed number.

Why does the network do worse with speed 3? Two mixed causes: it never saw it in training and the receptive field is limited (previous lesson). To tell them apart it would be enough to retrain including speed 3 and compare.

How can multi-step prediction be improved? Train also on its own output (scheduled sampling), use a loss over several steps, or a model that explicitly predicts the uncertainty. We did not try them.

Is the original Moving MNIST the same? No: the one by Srivastava and colleagues uses two 28×28 digits on a 64×64 canvas and a specifically generated set of sequences. Our variant (one 8×8 digit on 16×16, from load_digits) is simpler and the numbers cannot be compared with the published values.

Is a ConvLSTM really needed for a task like this? To show the ideas, yes, but a task with a single digit and constant speed could be solved with a simpler model too. The value of the example lies in showing the three difficulties (generalisation, error accumulation, ambiguity of the future) that remain in larger problems.

What to take away

A two-layer ConvLSTM with 28 thousand parameters learns in a few minutes of CPU to predict the next frame of moving digits with 45% of the error of the "copy" baseline, after an initial plateau in which it does little better than a black image. The result has three measured limits: with never-seen speeds the advantage shrinks (ratio 0.67), predicting five steps in a row the images lose half their sharpness and the brightest pixel falls to 0.58 even though the error ratio against the copy stays stable, and already at one step the images are six times blurrier than the real ones, because the MSE loss rewards the average of the possible futures. These three limits (generalisation, error accumulation, ambiguity of the future) are the open problems of every video prediction model. In the next lesson we change topic: how to get the signal through very deep networks, with Highway Networks.