64 - Generating Text Character by Character with an LSTM

A language model is a network that, given the text written so far, assigns a probability to what can come next. If the question is "what is the next character?", we speak of a character-level (char-level) model. It is the smallest possible way of getting hands-on with text generation: no word dictionaries, no tokenisation, just letters, spaces and punctuation. In this lesson you build such a model from scratch with a two-layer LSTM, train it on a corpus of technical English text, evaluate it against simple baselines with real numbers and make it write at three different "temperatures". At the end you will understand why the text that comes out is plausible in form and devoid of sense in content, and what "a language model" really means.

The words of this lesson

The vocabulary is the set of distinct symbols the model knows: in our corpus there are 141 characters (letters, digits, space, punctuation and a few non-ASCII symbols), and each has an integer, its index. The embedding is a table of learned numbers with one row per character: given the index it returns a vector (here of 64 numbers), that is a dense, adjustable representation in place of a meaningless label. For each possible next character the network produces a raw score, the logit, and the softmax turns the scores into positive probabilities that sum to 1.

The cross-entropy is the loss −log p, where p is the probability the model assigned to the character that was really there: it is small if the model was sure and right, large if it was sure and wrong. The average over the text is measured in nats (natural logarithm) or, dividing by ln 2, in bits per character. The perplexity is exp(cross-entropy in nats) and reads as "among how many equally likely characters the model is hesitating, on average": a perplexity of 5 means hesitating as if choosing at random among 5 characters.

Sampling consists in drawing the next character at random according to the model's probabilities, instead of always choosing the most probable one; the temperature regulates how random. Finally teacher forcing: in training, at every position the network is given the true previous character and not the one it would have predicted itself. We will not give it a separate piece of code, because it is already in the way we build input and target.

The analogy: the autocomplete of someone who has read a million lines

Imagine a reader who has devoured a million lines of Italian text without ever understanding its meaning, but who, from seeing so much of it, has an "ear" for the form: he knows that after "Il modell" an "o" almost certainly follows, that after a full stop and a space comes a capital letter, that "zx" is never found. Now you read him the beginning of a sentence and ask him to continue letter by letter, each time drawing a letter at random but weighted by what "sounds" right to him. You will get text that sounds Italian and means nothing. This is exactly what our model does, with one technical difference: its "ear" does not look only at the last letter but, thanks to the LSTM state, at a summary of everything it has read before.

The drawing shows the trick that makes training efficient. If the input window is c i a o ␣ m, the target is i a o ␣ m o: the same sequence shifted by one position. In a single pass the network makes six predictions, one per position, and each has its own loss. There is no need to build six separate examples. The orange arrow is the state (h, c) that passes from one position to the next: thanks to it, the prediction at the third position takes into account c and i, not just a.

The mathematics: the probability of a text as a product

The chain rule of probability says that the probability of a whole text is the product of the probabilities of each character given those before it:

P(c_1, c_2, …, c_T) = P(c_1) · P(c_2 | c_1) · P(c_3 | c_1, c_2) · … · P(c_T | c_1 … c_{T−1})

The model learns to estimate each factor P(c_t | c_1 … c_{t−1}): this is exactly what the LSTM computes, because its state h_{t−1} summarises c_1 … c_{t−1}. In practice, at every step:

e_t   = Embedding[c_t]                    (vector of 64 numbers)
h_t   = LSTM(e_t, h_{t−1}, c_{t−1})       (state of 128 numbers)
l_t   = W · h_t + b                       (141 logits, one per possible character)
p_t   = softmax(l_t)                      (141 probabilities)
loss_t = −log p_t[ c_{t+1} ]              (the true character that follows)

The total loss is the mean of the loss_t over all positions and all examples. Minimising the loss is equivalent to maximising the probability the model assigns to the true text: it is maximum-likelihood estimation.

A reference number. A model that knows nothing assigns uniform probability 1/141 to every character: its loss is ln 141 = 4.949 nats, that is 7.14 bits per character, perplexity 141. Everything the model learns lies in the space between 4.949 and zero.

The corpus: a text you can reproduce

An experiment whose numbers you cannot redo is not an experiment. That is why the corpus is not a private file: it is the text of the documentation of the language constructs that every Python installation carries with it, in the pydoc_data.topics module (a dictionary that associates to each topic of the interactive help, such as "for" or "class", its text). We concatenate it in alphabetical order of key and save it in corpus.txt. It is technical English text, not Italian, and its content depends on the Python version: the numbers in this lesson are obtained with Python 3.14.4.

from pydoc_data.topics import topics

text = "\n\n".join(topics[key] for key in sorted(topics))   # sorted keys: same text on every run
with open("corpus.txt", "w", encoding="utf-8") as f:
    f.write(text)
print(len(topics), "topics,", len(text), "characters,", len(set(text)), "distinct")

80 topics, 577618 characters, 141 distinct

The program prints 80 topics, 577,618 characters and 141 distinct symbols (letters, digits, punctuation and a few non-ASCII characters). Half a million characters is a very small corpus for a language model, and it is a deliberate limit: training stays in the range of minutes on CPU. The vocabulary is therefore 141 symbols, and the model that guesses among all of them has the loss ln 141 = 4.949 nats already computed above.

Baselines first: how hard is it really?

Before training a recurrent network it is worth knowing what much simpler models achieve. We compute two baselines on our corpus, evaluated on the final part (the 5% of validation). The unigram model assigns to each character its frequency in the training part (the "e"s are more frequent than the "z"s). The bigram model assigns to each character the probability of following the previous character, counted in the training part. In both, a small constant is added to the counts (smoothing) to avoid zero probabilities for pairs never seen.

import numpy as np
from collections import Counter

text = open("corpus.txt", encoding="utf-8").read()
cut = int(0.95 * len(text))
train_txt, val_txt = text[:cut], text[cut:]
chars = sorted(set(text))

# Baseline 0: uniform guess over the vocabulary
print("uniform  :", round(np.log(len(chars)), 3), "nats/char")

# Baseline 1: unigram frequencies estimated on the training part
counts = Counter(train_txt)
p1 = {c: (counts[c] + 1) / (len(train_txt) + len(chars)) for c in chars}      # +1 smoothing
loss1 = -np.mean([np.log(p1[c]) for c in val_txt])
print("unigram  :", round(loss1, 3))

# Baseline 2: bigram = "what usually follows the previous character", with smoothing
pair = Counter(zip(train_txt, train_txt[1:]))
prev = Counter(train_txt[:-1])
def p2(a, b):
    return (pair[(a, b)] + 0.1) / (prev[a] + 0.1 * len(chars))
loss2 = -np.mean([np.log(p2(a, b)) for a, b in zip(val_txt, val_txt[1:])])
print("bigram   :", round(loss2, 3))
print("(bits per char = nats / ln 2 ; perplexity = exp(nats))")

uniform  : 4.949 nats/char
unigram  : 3.211
bigram   : 2.349

The code uses Counter to count, computes the mean loss as −mean(log p) on the validation text and tries three levels of knowledge: none (4.949), the frequencies of single characters (3.211), the pair of characters (2.349). The progress 4.949 → 3.211 → 2.349 shows that even a single character of context removes a lot of uncertainty. An LSTM that reads up to 128 characters should do much better than 2.349. The two runs of the calculations are identical: there is nothing random in them. If that were not so, there would be a bug.

Code: preparing the data

The corpus is the same corpus.txt produced above (577,618 characters). If you want to try a text of your own, any text file will do, but the numbers in this lesson hold only for the one from pydoc_data.topics: changing file changes vocabulary, baselines and losses.

import torch
import torch.nn as nn

text = open("corpus.txt", encoding="utf-8").read()       # the file written by the corpus block above
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}               # character -> integer
itos = {i: c for c, i in stoi.items()}                   # integer -> character
data = torch.tensor([stoi[c] for c in text], dtype=torch.long)

split = int(0.95 * len(data))                            # validation = the LAST 5% of the text
train_data, val_data = data[:split], data[split:]
print(len(text), "characters,", len(chars), "distinct")

def get_batch(source, batch=32, length=128):
    """Random windows. Input: chars [i, i+length). Target: the same window shifted by ONE."""
    starts = torch.randint(0, len(source) - length - 1, (batch,))
    x = torch.stack([source[s:s + length] for s in starts])
    y = torch.stack([source[s + 1:s + length + 1] for s in starts])
    return x, y

Explanation. sorted(set(text)) extracts the distinct characters in a stable order, stoi ("string to integer") and itos are the dictionaries for the way there and back: they must be saved together with the model, otherwise the weights can no longer be used. data is the whole text as a list of indices. The split is in time, as for series: the last 5% is validation, and validation never enters training. get_batch extracts 32 random windows (64 in validation) 128 characters long: x starts at s, y starts at s+1, and that is the "shifted by one" of the drawing. The windows are taken at random rather than in order: every training step sees different parts of the text, which stabilises the gradient.

Code: the model and the training

class CharLSTM(nn.Module):
    def __init__(self, vocab, emb=64, hidden=128, layers=2, dropout=0.2):
        super().__init__()
        self.emb = nn.Embedding(vocab, emb)
        self.lstm = nn.LSTM(emb, hidden, layers, dropout=dropout, batch_first=True)
        self.head = nn.Linear(hidden, vocab)

    def forward(self, x, state=None):
        out, state = self.lstm(self.emb(x), state)       # a prediction at EVERY position
        return self.head(out), state                      # logits: (batch, length, vocab)

torch.manual_seed(0)                                       # fixed seed: same weights, same batches
model = CharLSTM(len(chars))
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
loss_fn = nn.CrossEntropyLoss()

@torch.no_grad()
def val_loss():
    model.eval()
    x, y = get_batch(val_data, batch=64)
    logits, _ = model(x)
    model.train()
    return loss_fn(logits.reshape(-1, logits.size(-1)), y.reshape(-1)).item()

for step in range(1, 1001):
    x, y = get_batch(train_data)
    logits, _ = model(x)
    loss = loss_fn(logits.reshape(-1, logits.size(-1)), y.reshape(-1))
    opt.zero_grad(); loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
    if step % 250 == 0:
        print(f"step {step:4d}  train {loss.item():.3f}  val {val_loss():.3f}")

How to read it. CharLSTM has three blocks: nn.Embedding(vocab, 64) turns each index into a vector; nn.LSTM(64, 128, 2, dropout=0.2) is the two-layer LSTM with 128 units and dropout between the layers (it randomly switches off 20% of the outputs of the first layer in training, to reduce overfitting); nn.Linear(128, vocab) produces one logit per character at every position. The forward also returns the state, the pair (h, c): it is needed in generation to continue from where we stopped. In total the model has 258,637 parameters. The CrossEntropyLoss expects logits of shape (N, classes) and targets of shape (N,): that is why reshape(-1, logits.size(-1)) flattens batch and time into a single axis. The rest of the loop is the one already seen: zero the gradients, backward, clipping, step. The val_loss function measures the loss on a validation batch with model.eval() and torch.no_grad().

step  250  train 2.046  val 1.998
step  500  train 1.654  val 1.709
step  750  train 1.635  val 1.414
step 1000  train 1.375  val 1.500

Reading (one seed; 1000 steps in 170-206 seconds on CPU, 4 threads, with the machine busy with other processes; two runs with identical output, also in the generated text further down). After only 250 steps the model is already at 2.00 on validation, below the bigram (2.349): it has learned more than the simple pair of characters. At 1000 steps the validation is 1.500 nats, that is 2.16 bits per character and a perplexity of about 4.5: it is as if the model chose, on average, among fewer than five equally likely characters instead of among 141.

The two columns should be read with caution: each number is the loss of a single batch (32 training windows, 64 validation windows), so it fluctuates from one row to the next (validation rises from 1.414 to 1.500 between step 750 and 1000 while training falls), and the differences of a few hundredths between train and validation are noise, not overfitting.

In all, the model has seen about 4.1 million characters (1000 steps × 32 × 128), that is the half-million corpus about seven times, and the curve is still falling: more steps would improve it. A larger model or one trained for longer would sooner or later begin to separate the two numbers, and that is the moment to stop or to raise the dropout.

Sampling and temperature

Now the model has to write. At each step it produces 141 logits; we want to turn them into a choice. Two extreme strategies: greedy (always take the most probable character) and sampling (draw at random according to the probabilities). Greedy is deterministic and tends to fall into repetitive cycles; pure sampling gives variety but also errors. The temperature T regulates the compromise by dividing the logits before the softmax:

p_i(T) = exp(l_i / T) / Σ_j exp(l_j / T)

With T = 1 you get the model's probabilities as they are. With T < 1 the differences between the logits are amplified: the most probable character gains mass. With T > 1 they flatten: chance takes over. Let us do it with three logits, to see it:

import numpy as np

def softmax(logits, temperature=1.0):
    z = np.array(logits) / temperature
    z = z - z.max()                       # subtract the max: same result, no overflow
    e = np.exp(z)
    return e / e.sum()

logits = [2.0, 1.0, 0.1]                  # the model's raw scores for 3 candidate characters
for temp in (0.5, 1.0, 2.0, 10.0):
    print(f"T={temp:4.1f}", softmax(logits, temp).round(3))

T= 0.5 [0.864 0.117 0.019]
T= 1.0 [0.659 0.242 0.099]
T= 2.0 [0.502 0.304 0.194]
T=10.0 [0.366 0.331 0.303]

The logits are [2.0, 1.0, 0.1]. At T = 1, the softmax gives 0.659, 0.242, 0.099: the first character is favoured but not certain. Halving the temperature, the first rises to 0.864 (almost deterministic); doubling it, it falls to 0.502; at T = 10 the three probabilities are almost equal (0.37, 0.33, 0.30), that is the model is almost indifferent. In the code z.max() is subtracted before exponentiating: it gives the same result but avoids overflow with large logits, a classic numerical-stability trick. In the limit T → 0 you get greedy, at T → ∞ the uniform choice.

Generating: the loop with state

@torch.no_grad()
def generate(prompt, length=300, temperature=1.0):
    model.eval()
    x = torch.tensor([[stoi[c] for c in prompt]])
    logits, state = model(x)                              # read the prompt, keep (h, c)
    out = list(prompt)
    for _ in range(length):
        probs = torch.softmax(logits[0, -1] / temperature, dim=-1)   # temperature rescales the logits
        nxt = torch.multinomial(probs, 1).item()                      # SAMPLE, do not take the argmax
        out.append(itos[nxt])
        logits, state = model(torch.tensor([[nxt]]), state)           # one char in, state carried on
    model.train()
    return "".join(out)

torch.manual_seed(1)                                       # fixed seed for the sampling too
for temp in (0.5, 1.0, 1.5):
    print(f"--- temperature {temp}")
    print(generate("The class ", length=200, temperature=temp))

How it works. generate first reads the prompt with a single call to the model: it returns the logits for all positions and the state (h, c) after the last character. From there the loop is always the same: take the logits of the last position (logits[0, -1]), divide by the temperature, apply the softmax, draw a character with torch.multinomial, append it to the text and pass to the model only that character together with the state. This is an important point: during generation we do not reread the whole text at every step, the summary of the past is in the state. So the cost per character is constant.

A real example of the output, run twice with the same result (prompt "The class ", 200 characters generated at each temperature, a single sampling per temperature: it is only an illustration). The text is English because the corpus is the Python documentation:

--- temperature 0.5
The class or value properation and be result be cormand contain or the methods in the context and module is defined in the or the subclass statement function of the "__class__()" is setive the subclass to the o
--- temperature 1.0
The class the farally "clocal", "rufallows conted.

   Changed in the syntaxs that continue unare numary value is identifly asues the called in lists; of the type: dicterals character).

Same if not
objects a m
--- temperature 1.5
The class one a Pythor. for
lock=-condex atngingP"F.dewcod_*?b:ca0nex~'٥ 7, C
Agf
the edlif
   bulc numeror23.

"fulgcisingslem__()" +s conz []*+]
   | Nasi'”.aln(!k)
   |2/200-1200ₙ2~!"'w<if Ds
   = op] passit

Three observations. First, at 0.5 the text is the "cleanest": most of the words exist and domain terms such as subclass, module, methods and even "__class__()" appear, but the sentences make no sense ("be result be cormand contain or the methods"), with a few invented words ("properation", "setive") and the repetition of "or" and "the". Second, at 1.0 the structure of the documentation appears (indented lines, "Changed in the syntax", lists with semicolons), but the invented words increase ("farally", "identifly asues"): the model's distribution is respected, and the corpus distribution contains these elements. Third, at 1.5 the words dissolve into sequences of random characters, symbols and even non-ASCII characters that appear rarely in the corpus: noise prevails. The right temperature depends on the use: low for predictable and coherent texts (at the risk of repetitions), high for variety (at the risk of nonsense).

What this experiment shows, and what it does not. It shows that with about 260 thousand parameters and three minutes of CPU an LSTM learns the form of text: approximate spelling, typical word length, punctuation, the look of the documentation. It does not show, and cannot, that it understands the meaning: the text has no coherence over more than a few words. There are precise limits: the training window is 128 characters, so nothing that lies farther away can be learned; the corpus is small; and the LSTM has 128 units. These limits are the reason why modern language models are much larger and use attention.

Trade-offs

• Character or word. The character level has a tiny vocabulary (here 141 symbols: no unknown words) but long sequences: to remember a word of 8 letters you have to remember 8 steps. The word level has short sequences and enormous vocabularies (tens of thousands of entries) with the problem of rare terms. Modern solutions sit in between (subwords).
• Depth and size. Two layers of 128 are a compromise; more capacity reduces the loss but requires more data to avoid overfitting.
• Window length. A longer one allows longer contexts, but costs memory and makes the gradient harder (the whole lesson on vanishing).

The three choices have the same pattern: every step towards more capacity or more context moves the cost onto memory, compute time or difficulty of training. In our experiment the window of 128 characters and the state of 128 units were chosen so that training stays in the range of minutes on CPU, not because they are optimal; with a corpus of half a million characters it would make no sense to increase capacity before having seen validation separate from training.

Real traps

The first is not saving the vocabulary with the model: without the original stoi and itos the weights are unusable, because adding a single new character shifts all the indices. The second is evaluating only with the loss on the training set: if the model has the capacity to memorise, that one always falls, and only validation says whether it is generalising, so split in time as for time series. The third is judging the model by the one sample that is written well: sampling is random, and a beautiful sample proves no more than an ugly one; the real measure is the validation loss, or the bits per character, not the impression.

The fourth is a temperature that is too low, which leads to cycles: at very small T, or with greedy, the model can repeat the same sequence indefinitely. The fifth is forgetting the state during generation: if at each character you send the model only that character without the state, the LSTM restarts from zero at every step and produces incoherent text; the code above passes state from one step to the next. The sixth is forgetting model.eval(), which leaves the dropout on and makes the text noisier. The last is confusing nats and bits: 1 nat is 1/ln 2 ≈ 1.44 bits, and comparing a number in nats with one in bits is a common mistake when reading different articles.

Questions to check yourself (with answers)

Why is the target the input shifted by one? Because at every position the task is to predict the next character; with the shift you get many trainable predictions for each pass.

A loss of 2.0 nats: how many bits per character and what perplexity? 2.0 / ln 2 ≈ 2.885 bits and exp(2.0) ≈ 7.39.

You divide the logits by T = 0.1: what happens to the distribution? The differences between logits are multiplied by 10: the distribution becomes almost a peak on the most probable character, that is almost greedy.

Why is the validation loss, and not the generated text, the metric to watch? The generated text is a random sample and its perceived quality is subjective; the validation loss is a numerical measure and comparable between models.

A worked example by hand: cross-entropy and perplexity on four characters

The model reads four positions and, for each, assigns a probability to the character that then really appeared: 0.7, 0.5, 0.9, 0.05. The losses are the negative natural logarithms: −ln 0.7 = 0.357, −ln 0.5 = 0.693, −ln 0.9 = 0.105, −ln 0.05 = 2.996. The mean is (0.357 + 0.693 + 0.105 + 2.996) / 4 = 1.038 nats, that is 1.038 / ln 2 = 1.50 bits per character, and the perplexity is e^1.038 = 2.82.

Note how a single very wrong prediction (probability 0.05 for the true character) weighs more than all the others: it contributes 2.996 out of a sum of 4.151, that is 72%. This is why the cross-entropy punishes harshly a model that is "sure and wrong", and why a language model must never assign almost-zero probability to what can happen (hence the smoothing in the unigram and bigram baselines).

Sampling better: top-k and nucleus

Temperature is not the only way to govern sampling. Two widely used variants: top-k keeps only the k most probable characters and redistributes the probability among them (it avoids the absurd characters which, added together, still had a non-zero probability); nucleus sampling or top-p (Holtzman and colleagues, 2019) keeps the smallest set of characters whose cumulative probability exceeds a threshold p (for example 0.9). With a very concentrated distribution (0.95, 0.03, 0.02) the nucleus at 0.9 keeps a single character; with a flat distribution it keeps many: it adapts to the uncertainty of the moment, unlike a fixed top-k. We have not compared them here with a measurement, so we cannot say which is better for this model.

Frequently asked questions

Why an embedding and not a one-hot vector? The one-hot (141 numbers with a single 1) is a representation without structure: every character is equidistant from the others. The embedding learns a dense vector of 64 numbers in which characters that behave similarly (the vowels, the digits) end up close. With a single matrix per layer, it is also equivalent to multiplying the one-hot by a 141×64 matrix, only much more efficient.

Dropout during generation? It must be switched off (model.eval()). Dropout serves in training to regularise; leaving it on in generation adds random noise on top of that of sampling.

Why random windows and not the text in order? Every training step sees different parts of the corpus, so the gradient is less correlated from one step to the next. The cost is that every window starts with a null state: the network never sees contexts longer than 128 characters (except in generation, where the state is carried forward).

How do I continue training the next day? Save together the weights, the optimiser state and the vocabulary (stoi, itos); reload them with load_state_dict. Without the vocabulary the weights are useless.

Is a perplexity of 4.5 good? It has to be compared with the baselines: for a uniform model it would be 141, for the unigram e^3.211 ≈ 24.8, for the bigram e^2.349 ≈ 10.5. So 4.5 is more than twice as good as the bigram, but it remains far from the perplexity of a good language model, which has much more context and data.

Can I use the same code with words instead of characters? Yes: only the vocabulary changes (thousands of words, with a symbol for the unknown ones) and the embedding size. The rest is identical, but the cost of the last linear layer grows with the size of the vocabulary.

Takeaways

A character-level language model is an LSTM that, at every position, produces a probability distribution over the next character; it is trained by minimising the cross-entropy loss with the target shifted by one, it is evaluated in bits per character or perplexity by comparing with baselines (here 4.949, 3.211 and 2.349 nats against about 1.50 for the LSTM), and it is used by sampling with a temperature. It learns the form of text, not the meaning. In the next lessons we leave the world of pure sequences to see how different networks are combined: ConvLSTM, hybrid networks, and then reinforcement learning.

In the series, the previous lesson is RNN, LSTM and GRU Compared: A Controlled Experiment Over Several Seeds; the next one is Hybrid Architectures: A CNN for Space and an LSTM for Time.