07 - Reasoning models: o1, DeepSeek-R1, RLHF

A model that "reasons" tends to answer more slowly than a normal one, because before giving the answer it generates more text. The label, however, covers three different things, and confusing them leads to wrong conclusions: how a model is steered towards what people prefer (RLHF, DPO), how it is trained on problems whose answer can be checked (reinforcement learning with rule-based rewards, for example GRPO) and how more compute is spent at answer time (sampling several chains of reasoning and voting). In this article I build each of the three in miniature, with numbers you can redo, and from the original papers I report only what I have read.

This is the seventh article of the series Modern Artificial Intelligence. If you come from the one on the Model Context Protocol or from the one on agents, note that extended reasoning is a different way of using more compute: there the extra compute goes through external tools, here through more generated text. You find the comparison with the growth of models in the article on scaling laws and emergent capabilities. Python with numpy is enough, and no model is downloaded or called.

Three different things under the same label

• Alignment to preferences: you have pairs of answers, one preferred to the other, and you move the model towards the preferred one.
• Reinforcement with a verifiable reward: the model generates several answers to the same problem, an automatic check says which are correct, and those are reinforced.
• Compute at answer time: with the model frozen, several attempts are generated and their results are combined.

The three items act at different moments (training on preferences, training on verifiable problems, use of the already trained model) and have different costs: the first two are paid once, the third at every question. In real systems they are combined, but the papers that introduced them examine them one at a time, and that is the only honest way to understand what each does. One limit declared straight away: the announcement page of o1 was not reachable from my environment, so of that family I report only what the abstract of its system card says, namely that the models are trained with large-scale reinforcement learning to reason using a chain of thought. Nothing else about o1 is claimed here.

RLHF in three steps, one sentence each

The InstructGPT work (Ouyang and colleagues, 2022) starts from an observation: making a model bigger does not by itself make it better at following what the user wants, because it can produce false, toxic or useless answers. The recipe they propose has three steps: collect demonstrations written by people and fine-tune the model with supervised learning, collect rankings of several model answers and train a reward on them, then fine-tune the model further with reinforcement learning from that human feedback. The DeepSeekMath work, describing the same scheme, adds that a penalty is usually added to the reward, per token, on the distance from the reference model, typically the model fine-tuned at the first step: it serves to prevent the model from drifting too far and exploiting the flaws of the reward.

DPO: the same preference, without a reinforcement loop

The expensive step of that recipe is the third, because it requires the model to generate text during training. The Direct Preference Optimization work (Rafailov and colleagues, 2023) shows that the reward can be reparameterised so that the problem is solved with a simple classification loss on the preference pairs, without sampling from the model during training: the authors describe it as stable, performant and lightweight, and report that in their experiments it matches or exceeds PPO-based RLHF in sentiment control, summarization and single-turn dialogue. The loss (equation 7 of the paper) is the negative logarithm of a sigmoid applied to β times the difference between two log-probability ratios, that of the preferred answer and that of the rejected one, each with respect to the reference model.

To get a feel for it I build the smallest possible policy: a single prompt, four candidate answers and four numbers (the logits) in place of a network. The reference model is fixed, answer 1 is the preferred one and answer 2 is the rejected one. With this simplification the gradient has a closed form, and I check it against finite differences.

"""DPO on a tiny policy: 4 candidate answers, a single prompt.
The "policy" is 4 logits; the loss is the one in the DPO paper (eq. 7)."""
import numpy as np


def log_softmax(z):
    z = z - z.max()
    return z - np.log(np.exp(z).sum())


THETA_REF = np.array([0.5, 0.0, 0.3, -0.5])   # reference policy, frozen
WINNER, LOSER = 1, 2                            # the preference pair


def margin(theta, beta):
    lp, lr = log_softmax(theta), log_softmax(THETA_REF)
    return beta * ((lp[WINNER] - lr[WINNER]) - (lp[LOSER] - lr[LOSER]))


def loss(theta, beta):
    return float(np.log1p(np.exp(-margin(theta, beta))))   # -log sigmoid(margin)


def gradient(theta, beta):
    s = 1.0 / (1.0 + np.exp(margin(theta, beta)))           # sigmoid(-margin)
    e = np.zeros(4)
    e[WINNER], e[LOSER] = 1.0, -1.0   # grad log pi(w) - grad log pi(l) = e_w - e_l
    return -beta * s * e


def kl(theta):
    p, q = np.exp(log_softmax(theta)), np.exp(log_softmax(THETA_REF))
    return float((p * (np.log(p) - np.log(q))).sum())


# second calculation: finite differences against the analytic gradient
rng = np.random.default_rng(0)
max_error = 0.0
for _ in range(20):
    th, beta, h = rng.normal(size=4), rng.uniform(0.05, 2.0), 1e-6
    num = np.array([(loss(th + h * np.eye(4)[i], beta) - loss(th - h * np.eye(4)[i], beta)) / (2 * h)
                    for i in range(4)])
    max_error = max(max_error, float(np.abs(num - gradient(th, beta)).max()))
print(f"analytic gradient against finite differences: maximum error {max_error:.1e}")
print(f"initial loss {loss(THETA_REF.copy(), 0.1):.4f} (expected ln 2 = {np.log(2):.4f})")

for beta in (0.1, 1.0):
    th = THETA_REF.copy()
    print(f"--- beta = {beta}")
    print("step   loss    p(win)   p(lose)  p(alt0)  p(alt3)   KL")
    for step in range(0, 201):
        if step in (0, 10, 50, 200):
            p = np.exp(log_softmax(th))
            print(f"{step:5d}  {loss(th, beta):.4f}   {p[1]:.4f}   {p[2]:.4f}   "
                  f"{p[0]:.4f}   {p[3]:.4f}  {kl(th):.3f}")
        th = th - 0.5 * gradient(th, beta)

# Output:
# analytic gradient against finite differences: maximum error 2.2e-10
# initial loss 0.6931 (expected ln 2 = 0.6931)
# --- beta = 0.1
# step   loss    p(win)   p(lose)  p(alt0)  p(alt3)   KL
#     0  0.6931   0.2172   0.2931   0.3580   0.1317  0.000
#    10  0.6687   0.2790   0.2297   0.3592   0.1321  0.015
#    50  0.5824   0.5483   0.0704   0.2787   0.1025  0.312
#   200  0.3733   0.9585   0.0005   0.0300   0.0110  1.318
# --- beta = 1.0
# step   loss    p(win)   p(lose)  p(alt0)  p(alt3)   KL
#     0  0.6931   0.2172   0.2931   0.3580   0.1317  0.000
#    10  0.0990   0.5354   0.0752   0.2847   0.1047  0.292
#    50  0.0203   0.7404   0.0205   0.1748   0.0643  0.682
#   200  0.0050   0.8569   0.0058   0.1004   0.0369  0.979

Three functions matter. margin computes the quantity inside the sigmoid, that is β times the difference between the two log-probability ratios; loss takes the negative logarithm of the sigmoid; gradient exploits the fact that, for a softmax, the gradient of the logarithm of an answer's probability is a vector with 1 in its position minus the distribution, so that in the difference between preferred and rejected the distribution cancels and only +1 and -1 remain. The beta parameter, which here is 0.1 or 1, has the role the paper assigns to it: it regulates how far the model can move away from the reference. The check with finite differences gives a maximum error of the order of 1e-10, and the initial loss coincides with ln 2, as it must when the model equals the reference.

Read the output in three points. First: with β = 1 the loss collapses in ten steps (0.099) but the probability of the preferred answer rises only to 0.54, and to 0.86 after 200 steps; with β = 0.1 the loss falls slowly (0.37 at 200 steps) but the preferred answer reaches 0.96. A low loss therefore does not mean a high probability: it measures the margin with respect to the reference, and with a large β the margin is obtained with a small shift. Second: the distance from the reference (the KL column) at 200 steps is 0.98 with β = 1 and 1.32 with β = 0.1, so in this toy the large β keeps the model closer, as expected. Third: the answers that do not appear in the pair, alt0 and alt3, change probability anyway, because the probabilities are recomputed together: the loss guarantees nothing about what you did not compare.

GRPO: reinforcement with verifiable rewards

For mathematics or programming problems there is a way to decide whether an answer is right without people: compare the final result with the known one, or run the tests. In the DeepSeek-R1 work the DeepSeek-R1-Zero model is trained exactly like this, with rule-based rewards of two kinds: an accuracy one, which checks correctness (for example the answer in a predetermined format, or a compiler on the test cases) and a format one, which asks for the reasoning to be enclosed in dedicated markers. The authors write that they do not use neural reward models for reasoning tasks, because they observed that these are exposed to reward hacking, that is the exploitation of the flaws of the reward, during large-scale reinforcement learning.

The algorithm is GRPO, introduced in DeepSeekMath (Shao and colleagues, 2024): for each question a group of answers is sampled from the current policy, and in place of a learned value function as a baseline the mean reward of the group is used. The advantage of each answer is its reward minus the group mean, divided by the standard deviation of the group. The listing computes that advantage, for each number k of correct answers in a group of 8 with reward 0 or 1, and then uses it to train a bandit: a policy that chooses among five "strategies" whose probability of success I know.

"""Group-relative advantage (GRPO) and a bandit with a verifiable reward.
The "model" is a softmax policy over 5 strategies with known success probabilities."""
import numpy as np

# 1) the advantage: (r - mean) / standard deviation, within the group
G = 8
print("k correct out of 8 -> advantage of a correct answer / of a wrong one")
for k in range(0, G + 1):
    r = np.array([1.0] * k + [0.0] * (G - k))
    sd = r.std()                           # population standard deviation
    if sd == 0:
        print(f"  k={k}: all equal, deviation 0, advantage undefined (set to 0 here)")
        continue
    a = (r - r.mean()) / sd
    print(f"  k={k}: correct {a[0]:+.3f}, wrong {a[-1]:+.3f}")

# 2) a bandit: the policy learns which strategy succeeds most often
P_SUCCESS = np.array([0.10, 0.20, 0.30, 0.50, 0.80])


def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()


def train(seed, steps=400, lr=0.3, g=8):
    rng = np.random.default_rng(seed)
    theta = np.zeros(5)
    for _ in range(steps):
        pi = softmax(theta)
        choices = rng.choice(5, size=g, p=pi)
        r = (rng.random(g) < P_SUCCESS[choices]).astype(float)
        sd = r.std()
        if sd == 0:
            continue                         # uniform group: no signal
        adv = (r - r.mean()) / sd
        grad = np.zeros(5)
        for y, a in zip(choices, adv):
            e = -pi.copy()
            e[y] += 1.0                      # grad log pi(y) = e_y - pi
            grad += a * e / g
        theta += lr * grad
    return float(softmax(theta) @ P_SUCCESS)


finals = [train(s) for s in range(20)]
print(f"initial expected reward (uniform policy): {P_SUCCESS.mean():.3f}")
print(f"after 400 updates, mean over 20 seeds: {np.mean(finals):.3f} "
      f"(min {min(finals):.3f}, max {max(finals):.3f}); best possible strategy: {P_SUCCESS.max():.2f}")

# second calculation: exact gradient ascent on the expected reward, without sampling
theta = np.zeros(5)
for _ in range(400):
    pi = softmax(theta)
    expected = pi @ P_SUCCESS
    theta += 0.3 * pi * (P_SUCCESS - expected)   # analytic gradient of E[r]
print(f"check with the exact gradient, same steps: {softmax(theta) @ P_SUCCESS:.3f}")

# Output:
# k correct out of 8 -> advantage of a correct answer / of a wrong one
#   k=0: all equal, deviation 0, advantage undefined (set to 0 here)
#   k=1: correct +2.646, wrong -0.378
#   k=2: correct +1.732, wrong -0.577
#   k=3: correct +1.291, wrong -0.775
#   k=4: correct +1.000, wrong -1.000
#   k=5: correct +0.775, wrong -1.291
#   k=6: correct +0.577, wrong -1.732
#   k=7: correct +0.378, wrong -2.646
#   k=8: all equal, deviation 0, advantage undefined (set to 0 here)
# initial expected reward (uniform policy): 0.380
# after 400 updates, mean over 20 seeds: 0.796 (min 0.794, max 0.798); best possible strategy: 0.80
# check with the exact gradient, same steps: 0.792

In the code, sd is the population standard deviation of the group: in the two papers I read I did not find it specified which standard deviation is used, so this is my own choice. The two rows at the edges show a fact that depends on the formula: when all the answers have the same reward, all right or all wrong, the deviation is zero and the advantage is undefined; my implementation skips that group, and the model learns nothing from a problem that is too easy or too hard. In the other rows, when only one answer out of eight is correct it receives +2.646 and the wrong ones -0.378 each, because a rare success is the most informative signal; with seven correct out of eight the asymmetry reverses.

The bandit starts from the uniform policy, with expected reward 0.380, and after 400 updates reaches 0.796 on average over 20 seeds, with minimum 0.794 and maximum 0.798, against a possible maximum of 0.80. As a second calculation, the exact gradient ascent on the expected reward, without sampling, gives 0.792 with the same steps and the same learning rate. The closeness of the two numbers does not prove that the methods are equivalent, because the normalisation changes the scale of the update: it says that the group signal is enough to find the best strategy in a problem where the reward is verifiable. An LLM chooses sequences of tokens and not among five strategies, and my listing has neither the clipping of the ratio between policies nor the KL penalty of the paper's formula: it is the group-baseline mechanism, not the whole of GRPO.

The same authors report, for DeepSeek-R1-Zero, that the accuracy on AIME 2024 (mean pass@1) rises during training from 15.6% to 77.9% and that the mean length of the answer grows, so much so that they write that the model learns to solve tasks with more thinking time. These are figures from its authors, on a specific benchmark and model, and I have not reproduced them: the experiment above explains the mechanism, not the result.

More compute at answer time: voting

The third way of spending compute does not change the model. Self-consistency (Wang and colleagues, 2022) replaces the greedy decoding used with chain-of-thought: it samples various reasoning paths and chooses the most consistent answer among them, marginalising over the paths. The authors report improvements on arithmetic and commonsense benchmarks, among them +17.9% on GSM8K. The intuition is that a complex problem admits many ways of reasoning that lead to the same correct answer, while errors tend to scatter.

That last sentence, however, is a hypothesis, and the listing puts it to the test: each "sample" is an answer drawn from a fixed distribution, of which answer 0 is the correct one. In the first case the correct one is the most likely (0.40 against 0.15 for each of the other four). In the second a wrong answer is more likely than the correct one (0.45 against 0.30). I compute the probability that the correct one wins the vote in two ways: exactly, by enumerating all the possible counts with a multinomial distribution (ties are split in equal parts), and by simulation with a fixed seed.

"""More samples, more compute at answer time: the majority vote.
Each "sample" is an answer drawn from a fixed distribution; there is no model."""
import itertools
import math
import numpy as np

# probability that a sample gives each answer; answer 0 is the correct one
CASES = {
    "correct is the most likely (0.40 against 0.15 each)": [0.40, 0.15, 0.15, 0.15, 0.15],
    "a wrong one is more likely (0.30 against 0.45)": [0.30, 0.45, 0.10, 0.10, 0.05],
}
NS = [1, 5, 9, 17, 33]


def exact(p, n):
    """Exact probability that answer 0 wins the vote (ties split in equal parts)."""
    k, tot = len(p), 0.0
    for counts in itertools.product(range(n + 1), repeat=k - 1):
        rest = n - sum(counts)
        if rest < 0:
            continue
        c = (rest,) + counts               # c[0] = votes for the correct answer
        prob = math.factorial(n)
        for ci, pi in zip(c, p):
            prob *= pi ** ci / math.factorial(ci)
        top = max(c)
        if c[0] == top:
            tot += prob / c.count(top)
    return tot


def simulated(p, n, trials, seed):
    rng = np.random.default_rng(seed)
    votes = rng.multinomial(n, p, size=trials)
    top = votes.max(axis=1, keepdims=True)
    winners = votes == top
    return float(((votes[:, 0] == top[:, 0]) / winners.sum(axis=1)).mean())


for name, p in CASES.items():
    print(name)
    for n in NS:
        e = exact(p, n)
        s = simulated(p, n, 40000, seed=n)
        print(f"  n={n:2d}: exact {e:.3f}  simulated {s:.3f}  (difference {abs(e - s):.3f})")

# Output:
# correct is the most likely (0.40 against 0.15 each)
#   n= 1: exact 0.400  simulated 0.402  (difference 0.002)
#   n= 5: exact 0.549  simulated 0.550  (difference 0.001)
#   n= 9: exact 0.665  simulated 0.662  (difference 0.003)
#   n=17: exact 0.808  simulated 0.805  (difference 0.003)
#   n=33: exact 0.934  simulated 0.938  (difference 0.003)
# a wrong one is more likely (0.30 against 0.45)
#   n= 1: exact 0.300  simulated 0.303  (difference 0.003)
#   n= 5: exact 0.303  simulated 0.306  (difference 0.004)
#   n= 9: exact 0.282  simulated 0.280  (difference 0.002)
#   n=17: exact 0.232  simulated 0.228  (difference 0.004)
#   n=33: exact 0.158  simulated 0.158  (difference 0.000)

The first case shows the effect of self-consistency: the probability of winning rises from 0.40 with one sample to 0.665 with 9 and to 0.934 with 33. The second case shows its reverse: with a wrong answer more likely than the correct one, voting makes things worse, because it amplifies the most frequent answer, whatever it is. It goes from 0.300 to 0.158 with 33 samples, and nothing in the vote warns you that you are reinforcing the error. The simulation and the exact calculation agree within 0.004, compatible with the statistical error of about 0.0025 that 40,000 trials have. The cost is linear: n samples mean n complete generations, therefore n times the tokens, and for this reason compute at answer time is paid at every question.

What the experiments do not prove

The three listings use policies with a few numbers in place of language models, and they demonstrate mechanisms, not results of a product: none of them says how good a real model is. The figures on the benchmarks (17.9%, 15.6% and 77.9%) are those of the respective authors, reported as such. The range of each technique depends on things that are not visible here: the quality of the preference pairs, the goodness of the automatic verifier, the real distribution of a model's answers. No article in this series has artificial intelligence itself as its thesis, and this one less than the others: it is about three methods, and it judges them for what they do in a verifiable case.

What you can do now

You can tell apart the three things hiding under "model that reasons" and say when each acts and what it costs. You can write the DPO loss and check its gradient, read a group advantage and predict when a group gives no signal, and you can test with an exact calculation whether a majority vote helps or harms. The next step is practical: if you want to run an open model on your own machine, it helps to know how much memory it needs and how more than one request at a time is served, the topic of the last article in the series.