58 - RNN on a Time Series: Windows, Baselines and Multi-Step Forecasting

The quickest way to understand what an RNN does and does not do is to put it in front of a time series and compare it with the most trivial alternatives. In this lesson you build a sliding-window dataset, split it the right way (in time, not at random), train an nn.RNN in PyTorch and check whether it beats a persistence baseline ("tomorrow will be the same as today"). Then you ask the model to forecast several steps ahead by feeding it its own predictions and you watch the error grow. All the numbers below come from runs of the code shown, twice with identical output (PyTorch 2.12.0 on CPU, a single seed).

The words of this article

A time series is a sequence of measurements ordered in time (hourly temperature, daily sales, heartbeat), and the classic task is forecasting: given the past, estimate the next value; the horizon is how many steps ahead you want to predict. To turn it into training examples you use a sliding window: you cut a window of k consecutive values as input and use the value right after it as the target, then you shift the window by one step to get the next example.

The data is split into training (on which the network learns), validation (with which you choose the hyperparameters) and test (which you do not touch until you want an honest grade); here we use only training and test for simplicity. Leakage is information leaking: when something enters the training that in the real world you would not have, and the test score looks excellent but is false. As a metric we use the MAE (mean absolute error), the mean of the absolute values of the errors, which is read in the same units as the data; the MSE is the mean of the squares, penalises large errors more and is convenient for training. The baseline is the most trivial model you can imagine: if your complex model does not beat it, it is not worth having. An epoch is one complete pass over the training data, a mini-batch is the subset used for each weight update, and Adam is the optimiser, that is, the rule that turns the gradient into an update, a gradient descent that automatically adjusts the step for each weight.

The analogy: forecasting the weather by looking at the last 30 hours

If I ask you the temperature in an hour and let you look at the last 30 readings, you have a reasonable idea of the trend and of the daily oscillation. If you ask me the temperature in 50 hours and I can only reason on my own successive predictions, every small error adds to the next one. The model does the same thing. And like the honest meteorologist, you must always ask yourself: "am I doing better than saying it will stay as it is now?".

The drawing shows why from the vector of a series 2000 long you get 1970 examples (with a window of 30): every one-step shift gives a new example, but neighbouring examples overlap almost completely. It is precisely this overlap that makes it dangerous to shuffle the data before splitting it: a window in the test and one in the training that differ by a single step are almost the same example.

Step 1: prepare the data (by hand, in NumPy)

Three decisions matter more than the architecture. The first is to split in time before anything else: the test set is the final part of the series, because if you shuffle the windows before splitting, almost identical windows (shifted by a single step) end up in both the training and the test and the model "remembers" the test. The second is to normalise with the statistics of the training alone: the mean and standard deviation of the test do not exist yet when you make the real forecast, and using them is another, subtler leakage. The third is to compute the baselines right away: before training anything you write down how much the trivial forecast gets wrong, because without this number an MAE of 0.15 means nothing.

import numpy as np

rng = np.random.default_rng(0)
t = np.arange(2000)
series = np.sin(0.1 * t) + 0.5 * np.sin(0.31 * t) + rng.normal(0, 0.1, t.size)

# 1) split IN TIME first: the test set is the final part, never shuffled into the training set
split = int(0.8 * len(series))
train_raw, test_raw = series[:split], series[split:]

# 2) normalise with statistics computed on the TRAIN part only
mu, sigma = train_raw.mean(), train_raw.std()
train, test = (train_raw - mu) / sigma, (test_raw - mu) / sigma

def windows(x, size):
    """X[i] = x[i : i+size], y[i] = x[i+size]  (predict the next value)."""
    X = np.stack([x[i:i + size] for i in range(len(x) - size)])
    return X[..., None], x[size:]            # X shape: (N, size, 1)

SIZE = 30
X_tr, y_tr = windows(train, SIZE)
X_te, y_te = windows(test, SIZE)

# Baselines the model must beat
persistence = np.abs(X_te[:, -1, 0] - y_te).mean()        # "tomorrow = today"
mean_pred = np.abs(0.0 - y_te).mean()                       # always the training mean (0 after normalising)
print(f"MAE persistence {persistence:.3f}  MAE mean {mean_pred:.3f}")

The code, line by line where it matters. series is the synthetic series: two sinusoids of different frequencies added together plus noise. split = int(0.8 * len(series)) fixes the cut point in time. mu, sigma are computed only on the training part and applied to both: (x − mu) / sigma brings the values around zero with deviation 1, which helps training because the tanh works well near zero. The function windows builds, for each index i, the input x[i : i+size] and the target x[i+size]; X[..., None] adds the feature axis (here just one, the value), because nn.RNN expects the shape (examples, steps, features). The two baselines are computed in one line each: persistence compares the last value of the window (X_te[:, -1, 0]) with the target.

The series is synthetic: two superimposed sinusoids plus Gaussian noise with deviation 0.1. On this series the MAE of persistence is 0.200 (in the normalised scale) and that of "always predict the mean" is 0.886. The irreducible noise sets a floor: with noise 0.1 and a standard deviation of the series around 0.8, no model can go below an MAE of about 0.1 in the normalised scale (the MAE of a Gaussian is about 0.8 times its standard deviation).

Step 2: the model with nn.RNN

The model is the one from the previous lesson: an RNN that reads the window and a linear head applied only to the last state. Gradient clipping is also used, for the reasons seen earlier.

import torch
import torch.nn as nn

torch.manual_seed(0)

class NextValueRNN(nn.Module):
    def __init__(self, hidden=32):
        super().__init__()
        self.rnn = nn.RNN(1, hidden, batch_first=True)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):                      # x: (batch, time, 1)
        out, _ = self.rnn(x)
        return self.head(out[:, -1, :]).squeeze(-1)   # use the last state only

model = NextValueRNN()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
Xt, yt = torch.tensor(X_tr, dtype=torch.float32), torch.tensor(y_tr, dtype=torch.float32)
Xv, yv = torch.tensor(X_te, dtype=torch.float32), torch.tensor(y_te, dtype=torch.float32)

for epoch in range(15):
    model.train()
    perm = torch.randperm(len(Xt))             # shuffling WINDOWS is fine once the split is done
    for i in range(0, len(Xt), 64):
        idx = perm[i:i + 64]
        loss = nn.functional.mse_loss(model(Xt[idx]), yt[idx])
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    if epoch % 5 == 4:
        model.eval()
        with torch.no_grad():
            mae = (model(Xv) - yv).abs().mean().item()
        print(f"epoch {epoch+1:2d}  test MAE {mae:.3f}")

The class NextValueRNN has two components: nn.RNN(1, hidden) reads the window and nn.Linear(hidden, 1) turns the last state into a number (the forecast). In the forward, out[:, -1, :] picks the last step of each example and .squeeze(-1) removes the useless axis. The training loop always has the same shape: model(Xt[idx]) computes the forecast, mse_loss measures the error, opt.zero_grad() clears the old gradients (PyTorch accumulates them!), loss.backward() computes the new ones with the BPTT of the previous lesson, clip_grad_norm_ limits them and opt.step() updates the weights. Every five epochs we switch to eval() mode and torch.no_grad() to measure the MAE on the test without computing gradients.

Output obtained:

epoch  5  test MAE 0.178
epoch 10  test MAE 0.171
epoch 15  test MAE 0.146

The model beats persistence (0.200) already after 5 epochs and keeps improving, but stays above the floor of about 0.1: there is still room. The important point is methodological. The result "0.146" is interesting only because we wrote "0.200" first. For a series with strong autocorrelation, persistence is often hard to beat and many complex models fail right there.

Step 3: forecasting several steps ahead

So far the model predicts one step. To predict h steps, the autoregressive approach reuses its own prediction as the last element of the window and goes round again. The downside is that errors accumulate: every wrong prediction becomes a wrong input for the next.

@torch.no_grad()
def forecast(model, seed_window, steps):
    """Feed the model its own predictions: errors accumulate step after step."""
    model.eval()
    window = seed_window.clone()                         # (1, SIZE, 1)
    preds = []
    for _ in range(steps):
        nxt = model(window)                              # (1,)
        preds.append(nxt.item())
        window = torch.cat([window[:, 1:, :], nxt.view(1, 1, 1)], dim=1)   # slide the window
    return np.array(preds)

seed = Xv[:1]
for horizon in (1, 10, 50):
    p = forecast(model, seed, horizon)
    err = np.abs(p - y_te[:horizon]).mean()
    print(f"horizon {horizon:2d}: MAE {err:.3f}")

The function starts from a real window (seed_window), asks the model for the prediction and then slides the window: it discards the oldest value (window[:, 1:, :]) and appends at the end the prediction just made (torch.cat). From the second step on the model reads, partly or entirely, its own outputs instead of the real data: that is why the error accumulates. The decorator @torch.no_grad() avoids building the gradient graph, which is useless when using the model. The final comparison uses y_te[:horizon]: the first horizon true values following the starting window.

horizon  1: MAE 0.177
horizon 10: MAE 0.178
horizon 50: MAE 0.452

At 1 and 10 steps the error is practically the same; at 50 steps it more than doubles (0.452). A single starting point and a single seed: the precise value varies, the tendency (the error grows with the horizon) does not. This is expected behaviour and not a defect of the code.

What changes if you turn one knob at a time

A good way to learn is to change one thing and look at the effect. The function retrains an RNN from scratch with the window size and the state size you choose, and returns the MAE on the test (same 15 epochs, same seed).

def fit_and_score(size, hidden, epochs=15, seed=0):
    """Rebuild windows of the given size, train a fresh RNN, return the test MAE."""
    torch.manual_seed(seed)
    Xa, ya = windows(train, size)
    Xb, yb = windows(test, size)
    Xa, ya = torch.tensor(Xa, dtype=torch.float32), torch.tensor(ya, dtype=torch.float32)
    Xb, yb = torch.tensor(Xb, dtype=torch.float32), torch.tensor(yb, dtype=torch.float32)
    net = NextValueRNN(hidden)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for _ in range(epochs):
        perm = torch.randperm(len(Xa))
        for i in range(0, len(Xa), 64):
            idx = perm[i:i + 64]
            loss = nn.functional.mse_loss(net(Xa[idx]), ya[idx])
            opt.zero_grad(); loss.backward()
            nn.utils.clip_grad_norm_(net.parameters(), 1.0)
            opt.step()
    net.eval()
    with torch.no_grad():
        return (net(Xb) - yb).abs().mean().item()

print("window size (hidden=32):", {s: round(fit_and_score(s, 32), 3) for s in (2, 5, 30, 60)})
print("hidden size (window=30):", {h: round(fit_and_score(30, h), 3) for h in (2, 8, 32, 128)})

window size (hidden=32): {2: 0.201, 5: 0.174, 30: 0.146, 60: 0.16}
hidden size (window=30): {2: 0.215, 8: 0.171, 32: 0.146, 128: 0.16}

How to read it, with caution (a single seed, differences of a few hundredths can be noise). With a window of 2 values the model sees too little of the sinusoid's cycle and does only as well as persistence (0.201 against 0.200). With 5 it improves, with 30 it reaches the best observed. Going to 60 does not help and even gets slightly worse: more steps mean a longer BPTT and a harder optimisation task, without any extra useful information, since the signal has much shorter periods. The same goes for the state: 2 numbers are too few to represent two oscillations with different phases, 8 do better, 32 is the best point, 128 brings no advantage on this simple problem. Practical rule: the right capacity is the smallest that solves the problem, and you find it by scaling in one direction until validation stops improving.

Measuring the damage of the wrong split (and a result you do not expect)

Shuffling before splitting is a mistake. Let us measure how much it hurts, comparing the two ways of splitting with the same model and the same budget. The function train_model retrains from scratch; mae measures the error. The "wrong" split assigns at random 80% of the windows to the training and 20% to the test.

def train_model(Xa, ya, epochs=15, seed=0, hidden=32):
    torch.manual_seed(seed)
    net = NextValueRNN(hidden)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    Xa = torch.tensor(Xa, dtype=torch.float32); ya = torch.tensor(ya, dtype=torch.float32)
    for _ in range(epochs):
        perm = torch.randperm(len(Xa))
        for i in range(0, len(Xa), 64):
            idx = perm[i:i + 64]
            loss = nn.functional.mse_loss(net(Xa[idx]), ya[idx])
            opt.zero_grad(); loss.backward()
            nn.utils.clip_grad_norm_(net.parameters(), 1.0)
            opt.step()
    return net.eval()

def mae(net, X, y):
    with torch.no_grad():
        return (net(torch.tensor(X, dtype=torch.float32)) - torch.tensor(y, dtype=torch.float32)).abs().mean().item()

# --- Experiment: what does the WRONG split promise us? ---
series_n = (series - series.mean()) / series.std()           # (normalised on everything: second small leak)
X_all, y_all = windows(series_n, SIZE)
rng2 = np.random.default_rng(0)
idx = rng2.permutation(len(X_all))
cut = int(0.8 * len(idx))
tr_idx, te_idx = idx[:cut], idx[cut:]                         # RANDOM split: neighbours end up on both sides

wrong = train_model(X_all[tr_idx], y_all[tr_idx])
print("random split  test MAE:", round(mae(wrong, X_all[te_idx], y_all[te_idx]), 3))

right = train_model(X_tr, y_tr)
print("temporal split test MAE:", round(mae(right, X_te, y_te), 3))

random split   test MAE: 0.145
temporal split test MAE: 0.146

Result: no visible difference (0.145 against 0.146). Why? This series is a sum of two sinusoids plus white noise. The predictable part is periodic and the network learns it from any stretch; the noise cannot be memorised by such a small network in 15 epochs. So on this series the wrong split did not give an inflated score. Do not conclude that it is harmless: the serious work lies in the opposite lesson. The damage depends on whether the series has irregular structures that nearby windows "reveal" to each other and on whether the model has the capacity to memorise them; you cannot know without having already done the right split. The temporal split costs nothing and answers the question you really care about ("what happens when the future arrives?"): always use it, without waiting to see the damage.

Series with a trend: why work on differences

An RNN with tanh produces states in (−1, 1), and its linear output has inputs within an interval learned in training. A series with an upward trend leaves that interval in the test. Let us try a sinusoid plus a linear drift, comparing the model on the raw levels and on the first differences (x_t − x_{t−1}), that is, the series of changes.

# --- Experiment: a series with a trend leaves the range seen in training ---
t2 = np.arange(2000)
trend = np.sin(0.1 * t2) + 0.004 * t2 + rng.normal(0, 0.05, t2.size)       # sine + upward drift
split2 = int(0.8 * len(trend))

def run_on(series_in, label):
    tr, te = series_in[:split2], series_in[split2:]
    m, s = tr.mean(), tr.std()
    tr, te = (tr - m) / s, (te - m) / s
    Xa, ya = windows(tr, SIZE); Xb, yb = windows(te, SIZE)
    net = train_model(Xa, ya)
    return mae(net, Xb, yb), np.abs(Xb[:, -1, 0] - yb).mean(), ya.std()

raw_mae, raw_persist, _ = run_on(trend, "raw")
diff_mae, diff_persist, _ = run_on(np.diff(trend), "diff")
print(f"raw levels : RNN {raw_mae:.3f}   persistence {raw_persist:.3f}")
print(f"differences: RNN {diff_mae:.3f}   persistence (of the differences) {diff_persist:.3f}")

raw levels : RNN 0.062   persistence 0.041
differences: RNN 0.515   persistence (of the differences) 0.966

Within each row the comparison is fair (same data, same scale), between rows it is not (different scales). On the raw levels the RNN (0.062) is beaten by persistence (0.041): the network could not follow the drift in the test, because the test values leave the interval seen in training and persistence, which merely copies the last value, is not affected. On the differences the picture flips: the series of changes has no trend, and the RNN (0.515) beats persistence (0.966) by almost a factor of two. To get back to the levels you rebuild with a cumulative sum (np.cumsum) starting from the last known value. This is why in finance and in meteorology one works on returns or differences, not on absolute prices or temperatures.

Checked with four calculations

With 5000 points and a window of 50 you get 5000 − 50 = 4950 examples, because each window consumes 50 values plus the target. If persistence has an MAE of 0.2 and the model 0.19 you probably do not have a success: the difference is small and with a single seed it can be noise, so you need more seeds and an estimate of the uncertainty. You normalise with the mean and deviation of the training alone because in production you will not know the mean of the future, and using it in the test would leak information you would not have. If you forecast 24 hours ahead with a 1-step recursive model, you measure the MAE at horizon 24 and not at 1, because errors compound and the 1-step value is too optimistic.

Walk-forward evaluation: a single test is not enough

So far we have a single cut point and therefore a single number. But a forecasting model is used over time: today with the data up to today, in a month with the data up to that day. Walk-forward evaluation (or rolling origin) simulates this: you train on the past up to an origin, test on the next block, then move the origin forward, train again and test again. You get several measurements, and their variability tells you how far you can trust them.

# Walk-forward (rolling-origin) evaluation: train on the past, test on the next block, move on.
def walk_forward(series_in, size=30, first_train=0.5, n_folds=4, epochs=15):
    n = len(series_in)
    block = int((1 - first_train) * n / n_folds)
    results = []
    for k in range(n_folds):
        end_train = int(first_train * n) + k * block
        tr = series_in[:end_train]
        te = series_in[end_train - size:end_train + block]        # overlap `size` values so the first window is complete
        m, s = tr.mean(), tr.std()
        Xa, ya = windows((tr - m) / s, size)
        Xb, yb = windows((te - m) / s, size)
        net = train_model(Xa, ya, epochs=epochs)
        persistence = np.abs(Xb[:, -1, 0] - yb).mean()
        results.append((end_train, mae(net, Xb, yb), persistence))
    return results

for end_train, model_mae, persist in walk_forward(series):
    print(f"train up to t={end_train:4d}   model MAE {model_mae:.3f}   persistence {persist:.3f}")

train up to t=1000   model MAE 0.139   persistence 0.193
train up to t=1250   model MAE 0.163   persistence 0.202
train up to t=1500   model MAE 0.152   persistence 0.197
train up to t=1750   model MAE 0.144   persistence 0.204

Explanation of the code: walk_forward divides the final part of the series into n_folds blocks. For each block it trains on everything that comes before (series_in[:end_train]), normalises with the statistics of that training, and tests on the block; the test starts size values earlier so as to have a complete window straight away (those values are already known, there is no leakage because they are inputs and not targets). The result: the model beats persistence in all four blocks (from 0.139 to 0.163 against about 0.2), and its MAE varies by about ±0.012 between blocks (at one seed). This is a much sturdier result than a single number. The cost is clear: you retrain four times.

Direct strategy against recursive strategy, measured on the same windows

There are two ways to forecast H steps: recursive (a one-step model, reused on its own predictions) and direct (a model with H outputs that predicts everything in one go). The interesting question is which does better. The previous experiment had a limit: a single starting point. Here we compare the two approaches with H = 10, evaluating the direct one on all the test windows and the recursive one on 200 starting points.

H = 10                                                    # forecast horizon (steps ahead)

def windows_multi(x, size, horizon):
    X = np.stack([x[i:i + size] for i in range(len(x) - size - horizon + 1)])
    Y = np.stack([x[i + size:i + size + horizon] for i in range(len(x) - size - horizon + 1)])
    return torch.tensor(X[..., None], dtype=torch.float32), torch.tensor(Y, dtype=torch.float32)

class DirectRNN(nn.Module):
    """One forward pass produces ALL H future values (direct strategy)."""
    def __init__(self, hidden=32, horizon=H):
        super().__init__()
        self.rnn = nn.RNN(1, hidden, batch_first=True)
        self.head = nn.Linear(hidden, horizon)
    def forward(self, x):
        out, _ = self.rnn(x)
        return self.head(out[:, -1, :])                   # (batch, H)

Xd, Yd = windows_multi(train, SIZE, H)
Xv2, Yv2 = windows_multi(test, SIZE, H)
torch.manual_seed(0)
direct = DirectRNN()
opt = torch.optim.Adam(direct.parameters(), lr=3e-3)
for epoch in range(15):
    perm = torch.randperm(len(Xd))
    for i in range(0, len(Xd), 64):
        idx = perm[i:i + 64]
        loss = nn.functional.mse_loss(direct(Xd[idx]), Yd[idx])
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(direct.parameters(), 1.0)
        opt.step()

with torch.no_grad():
    direct_err = (direct(Xv2) - Yv2).abs().mean(dim=0)    # MAE for each of the H steps, over ALL test windows

# Recursive strategy with the 1-step model from before, on 200 starting points
@torch.no_grad()
def recursive_errors(model1, X, Y, n_start=200):
    errs = torch.zeros(H)
    for j in range(n_start):
        window = X[j:j + 1].clone()
        for h in range(H):
            nxt = model1(window)
            errs[h] += (nxt - Y[j, h]).abs().item()
            window = torch.cat([window[:, 1:, :], nxt.view(1, 1, 1)], dim=1)
    return errs / n_start

rec_err = recursive_errors(model, Xv2, Yv2)
for h in (0, 4, 9):
    print(f"step {h + 1:2d}:  direct {direct_err[h]:.3f}   recursive {rec_err[h]:.3f}")

step  1:  direct 0.179   recursive 0.148
step  5:  direct 0.231   recursive 0.229
step 10:  direct 0.416   recursive 0.247

Reading. windows_multi builds, for each window, a target of H consecutive values; DirectRNN has a linear head with H outputs. The function recursive_errors accumulates the absolute error at each step h along the recursive trajectory. In this experiment (one seed, 15 epochs) the recursive one wins, and the advantage grows with the horizon: at step 10 the direct error is 0.416 against 0.247. It is a result that goes against the intuition "the recursive one accumulates errors, so it is worse". Plausible explanations, not verified here: the direct model has 10 tasks to learn with the same capacity and the same training budget, while the recursive one learns a single, simpler one, and the series is very regular (its one-step predictions are good, so the accumulation is modest). With noisier series or longer horizons the verdict may flip, and that is why in practice you try both and measure. The rule to take away is methodological: do not trust intuition about which strategy is better, measure on the horizon you care about.

Trade-offs and alternatives

• Recursive forecasting (as above): a single model, simple, but compounded errors.
• Direct forecasting: the head produces all h steps together (nn.Linear(hidden, h)). No recursive error accumulation, but the horizon is fixed and the model is larger.
• A longer window helps only if there is useful information in the distant past, and for a simple RNN a long window is also harder to train (previous lesson). Here 30 steps cover three periods of the faster sinusoid.

The three routes answer a single question, how far you want to trust your own predictions: the recursive one reuses them, the direct one does not, and the longer window changes only how much past the model sees. There is no fixed order of merit: the experiment of the previous section gave a winner on this series, and on another it might give the opposite, so the choice is made by measuring on the horizon you need.

Real traps

The most common bug is shuffling before splitting: it gives beautiful results in the lab and terrible ones in production, and the test must always be "after" the training in time. Next comes normalising on the whole series: use only the mean and deviation of the training and reuse them identically in production, saving them together with the model. Then there is having no baseline: a network that "learns" but does not beat persistence is of no use. Also measuring at one step only and declaring victory misleads: if the use case is forecasting 24 hours ahead, evaluate at 24 hours. Finally series with a trend: a series that rises forever leaves the range seen in training and an RNN with tanh does not extrapolate, so work on the differences (x_t − x_{t−1}) or on returns.

A worked example by hand, with eight numbers

To be sure we have understood how the data is built, let us do it with a series so small that it fits on one line: 2, 4, 6, 8, 10, 12, 14, 16, with a window of 3.

Step 1, windows. The first window is [2, 4, 6] and the target is the value right after, 8. Shifting by one step: [4, 6, 8] → 10, [6, 8, 10] → 12, [8, 10, 12] → 14, [10, 12, 14] → 16. From 8 values and a window of 3 you get 8 − 3 = 5 examples (the N − k rule already seen).

Step 2, split in time. If the training part is the first 6 values (2 … 12) and the test the last 2 (14, 16), no training window can contain 14 or 16: the valid training examples are [2,4,6]→8, [4,6,8]→10 and [6,8,10]→12.

Step 3, normalisation with the training statistics alone. The mean of the six training values is 7 and the standard deviation (of the population) is √((25 + 9 + 1 + 1 + 9 + 25) / 6) = √11.67 = 3.416. A value x becomes (x − 7) / 3.416. The 12 becomes 1.46; but the 14 becomes 2.05 and the 16 becomes 2.63: values outside the interval seen in training (which reaches 1.46). This small example contains in miniature the problem of the trend: a rising series leaves the interval and a network with tanh has never seen values that large.

Step 4, the metrics. If a model is wrong by +0.2, −0.5, +0.1 on three predictions, the MAE is (0.2 + 0.5 + 0.1) / 3 = 0.267 and the MSE is (0.04 + 0.25 + 0.01) / 3 = 0.100. The large error (−0.5) weighs 83% of the MSE but only 62% of the MAE: that is what "the MSE penalises large errors more" means.

Step 5, back to real units. An MAE of 0.146 is "normalised". Multiplying by the standard deviation of the training (0.795 in this series) gives 0.116 in the original units; persistence (0.200 normalised) corresponds to 0.159. If the series were a temperature in degrees, the network would be wrong on average by about 0.12 degrees against 0.16 for the trivial method. Always report the error in the units of the problem: a dimensionless number does not tell a colleague whether it is good or bad.

Frequently asked questions

Why not train on the whole series and evaluate "by eye"? Because without a test there is no way to know whether the network has learned a rule or memorised the examples. The test is your only honest estimate of how it will really go.

Isn't shuffling the training windows leakage? No, as long as the split between training and test has already happened in time. Shuffling only serves to make the mini-batches independent during training; the test is not touched.

Why use only the last state for the forecast? Because the last state has read the whole window: it is the summary the network has built. Using the intermediate states would be useful to predict every step (many-to-many), not just the next one.

Is the window of 30 "right"? No, it is a hyperparameter. In our experiment 30 worked better than 2, 5 and 60 (0.146 against 0.201, 0.174, 0.160). The useful rule is that the window must cover at least the longest period you care about (here the slow sinusoid has a period of 63 steps, the fast one 20) without being much longer.

What if I have several simultaneous series (temperature and humidity)? You put one feature per series: nn.RNN(input_size=2, …) and the input has shape (examples, steps, 2). The RNN does not change; only the third dimension of the data changes.

Can the model forecast something that has never happened? No: it learns regularities present in the data. An extraordinary event (a spike never seen) remains unpredictable; that is why walk-forward evaluation over different periods is important.

What you can do now

You can build the windowed dataset of a time series by splitting it in time and normalising it with the training statistics alone, compute the baselines before training, and train an RNN in PyTorch judging it by how much it beats persistence and not by the number itself. You can forecast several steps with the recursive and the direct strategy, measuring the error at the horizon you care about, and you can choose between levels and differences when there is a trend. The simple RNN works here because the dependency is short and smooth; when the memory needed gets longer you need the gated cell of the LSTM. For the basics, go back to Recurrent neural networks and to BPTT and gradient clipping.