99 - Running a Model Locally: Quantisation, Memory and Latency, Measured

A model with eight billion parameters, written at 16 bits per weight, takes up 14.96 GiB; the same model in the Q4_K_M format takes 4.58. Those are the numbers in the table that the llama.cpp documentation gives for Llama 3.1 8B (tools/quantize/README.md), and they say why quantization is the lever that decides whether a model fits in the memory you have. They also say, to anyone who reads them closely, that Q4_K_M is not worth 4 bits per weight but 4.8944, and that no size table answers the question that matters: how much precision did you lose to make it fit.

This lesson answers on the smallest piece that holds the whole question, a tensor of weights. I quantize it to 8, 4 and 2 bits, with one scale for the whole tensor or one per row, in symmetric and asymmetric form, and I measure three things: the error it introduces, the bytes it really costs, and the point where the result stops being useful. Then I go up one step, to a linear layer and a small trained network, to see how the error on the weights becomes error on the output.

Let me say right away what this lesson is not. I did not run any language model and I did not download any weights: you will not find tokens per second or GPU memory here, because they would be figures from a machine that is not yours and that I could not redo here. The figures on real models (GGUF, GPTQ, AWQ, bitsandbytes formats) come from the official documentation and from the papers, and are attributed to the source. Everything else I ran, twice, with identical output: Python 3.14.4 and NumPy 2.3.5 for the listings on tensors, PyTorch 2.12.0 on CPU, with the GPU hidden and two threads, for the last one. The machine was loaded (average load around 33 on 12 cores), and that is why the timings, where they appear, are ranges.

The ruler with few marks

Imagine having to record the length of a thousand wooden planks using a ruler with only 16 marks, instead of the 65,536 you would have at 16 bits. Every measurement has to be rounded to the nearest mark, so the maximum error is half the interval between two marks. So far it is easy. The difficulty lies in two decisions that the ruler does not make by itself. The first is how far it reaches: if the longest plank measures three metres and all the others thirty centimetres, a single three-metre ruler with 16 marks puts them 20 centimetres apart, and almost all the short planks end up on zero. The second is what to do with the exceptional plank: shorten the ruler and sacrifice it, or keep the ruler long and lose all the others.

The three levers of quantization are these. The number of marks is the precision, in bits. Giving a ruler to each group of planks, instead of one for all of them, is the per-channel or per-group scale. Shortening the ruler is clipping. In formulas, each weight w becomes an integer q = round(w / s) and is reconstructed as q · s, where s is the scale, the step between two marks. The rest of the lesson measures what each choice costs, starting from a tensor you can redo.

A tensor in hand: three bit widths, two scales, two schemes

The quantize function is mine: it does not come from any library, it only needs NumPy. It takes the tensor and four choices. bits is the precision (here 8, 4 or 2). per_channel, false by default, decides whether the scale is a single one for the whole tensor or one per row, that is per output channel: with True the tensor must be two-dimensional, because the reduction happens on axis 1. symmetric, true by default, uses the levels from -qmax to +qmax with qmax = 2^(bits-1) - 1, so zero stays exact; with the asymmetric form, scale and zero point cover the range from minimum to maximum and use all 2^bits levels. clip, 1.0 by default, is the fraction of the range that the ruler covers: below 1 the most extreme weights are cut off.

Two details of the code deserve a look before the output. At 2 symmetric bits qmax is 1, and the levels are only three (-1, 0, +1): the fourth code, which would exist at 2 bits, is not used. And the test tensor is built with a different amplitude for each row (from 0.0042 to 0.1123, a choice of mine to put the per-channel scale to the test): in a real model the heterogeneity between rows has to be measured, not assumed.

import numpy as np

def quantize(w, bits, per_channel=False, symmetric=True, clip=1.0):
    """Quantize w to `bits` bits and return (integers, scale, zero point, reconstructed weights).
    per_channel=False: one scale for the whole tensor; True: one per row (output channel)."""
    kd = {"axis": 1, "keepdims": True} if per_channel else {}
    if symmetric:
        qmax = 2 ** (bits - 1) - 1                     # 127, 7, 1
        a = np.abs(w).max(**kd) * clip
        scale = np.where(a > 0, a / qmax, 1.0)
        q = np.clip(np.round(w / scale), -qmax, qmax).astype(np.int32)
        return q, scale, 0, q * scale
    lo, hi = w.min(**kd) * clip, w.max(**kd) * clip
    scale = np.where(hi > lo, (hi - lo) / (2 ** bits - 1), 1.0)
    zero = np.round(-lo / scale)
    q = np.clip(np.round(w / scale) + zero, 0, 2 ** bits - 1).astype(np.int32)
    return q, scale, zero, (q - zero) * scale

def metrics(w, r):
    e = w - r
    mse = float(np.mean(e ** 2))
    snr = 10 * np.log10(float(np.mean(w ** 2)) / mse) if mse > 0 else float("inf")
    return mse, float(np.abs(e).max()), snr

def synthetic_weights(rows=512, cols=512, seed=2398):
    """Bell-shaped weights; every row has its own amplitude (log-normal): my own choice, to stress the per-channel scale."""
    rng = np.random.default_rng(seed)
    amplitude = 0.02 * np.exp(0.5 * rng.standard_normal((rows, 1)))
    return (rng.standard_normal((rows, cols)) * amplitude).astype(np.float32)

W = synthetic_weights()
print("shape", W.shape, "standard deviation %.4f" % W.std(), "max |w| %.4f" % np.abs(W).max())
print("row amplitude: from %.4f to %.4f" % (W.std(axis=1).min(), W.std(axis=1).max()))
print()
print("bits  scale         scheme       MSE          max error    SNR dB   levels used")
for bits in (8, 4, 2):
    for per_channel in (False, True):
        for sym in (True, False):
            q, s, z, r = quantize(W, bits, per_channel, sym)
            mse, emax, snr = metrics(W, r)
            print("%3d   %-12s  %-10s   %.3e    %.5f      %6.2f   %d" % (
                bits, "per channel" if per_channel else "per tensor",
                "symmetric" if sym else "asymmetric", mse, emax, snr, len(np.unique(q))))

# Output:
# shape (512, 512) standard deviation 0.0247 max |w| 0.3712
# row amplitude: from 0.0042 to 0.1123
#
# bits  scale         scheme       MSE          max error    SNR dB   levels used
#   8   per tensor    symmetric    7.100e-07    0.00146       29.34   166
#   8   per tensor    asymmetric   6.336e-07    0.00138       29.84   177
#   8   per channel   symmetric    3.385e-08    0.00146       42.56   255
#   8   per channel   asymmetric   2.926e-08    0.00138       43.19   256
#   4   per tensor    symmetric    1.889e-04    0.02651        5.09   13
#   4   per tensor    asymmetric   1.566e-04    0.02345        5.91   15
#   4   per channel   symmetric    1.118e-05    0.02651       17.37   15
#   4   per channel   asymmetric   8.455e-06    0.02335       18.59   16
#   2   per tensor    symmetric    6.040e-04    0.18525        0.05   3
#   2   per tensor    asymmetric   5.795e-04    0.11723        0.23   4
#   2   per channel   symmetric    4.340e-04    0.18525        1.48   3
#   2   per channel   asymmetric   2.096e-04    0.11712        4.64   4

The first four rows after the header are the eight-bit ones, and the last column already says something that will come up several times: with a single scale the tensor uses 166 levels out of 255 (symmetric), with one scale per row it uses them all. The maximum errors coincide between the two schemes (0.00146 at 8 bits, 0.02651 at 4) because they are half the step of the widest row, 0.3712 / 127 / 2 and 0.3712 / 7 / 2, and that row has the same scale in both. What changes is the error of the other rows, and the signal-to-noise ratio (SNR) sees it.

• The per-channel scale gains 13.2 dB at 8 bits (from 29.34 to 42.56), 12.3 dB at 4 bits (from 5.09 to 17.37) and 1.4 dB at 2 bits (from 0.05 to 1.48).
• The asymmetric form gains from 0.2 to 0.8 dB with a single scale, and from 0.6 to 3.2 dB with one scale per row.
• At 2 bits the tensor with a single scale and symmetric form has an SNR of 0.05 dB: the error is as large as the signal, because almost all the weights end up on zero.

How to read these three numbers: the granularity of the scale matters far more than the form. At 4 bits, going from one scale per tensor to one per row is worth 12 dB, going from symmetric to asymmetric from 0.8 to 1.2 dB. The asymmetric form begins to count where levels are few, because at 2 bits it offers four levels instead of three and per row it reaches 4.64 dB against 1.48. The cost is a second quantity to store for each scale, the zero point, and you pay for it in bytes: I discuss it in the section on real bytes.

Six decibels per bit

How much is one more bit worth? For a uniform quantizer the answer is known: the squared error drops by a factor of four, so you gain 20·log10(2) = 6.02 dB per bit. I check it on a Gaussian sample of 262,144 values, with the scale set on the maximum, for bits from 2 to 8. The listing reuses quantize and metrics from the previous block: the blocks of this lesson run in the order in which they appear, in the same session.

rng = np.random.default_rng(1)
G = rng.standard_normal((512, 512)).astype(np.float32)        # N(0,1), 262,144 values
print("bits   SNR dB   gain per added bit   MSE")
prev = None
series = []
for bits in range(2, 9):
    mse, emax, snr = metrics(G, quantize(G, bits)[3])
    series.append((bits, snr))
    print("%3d   %6.2f   %s   %.3e" % (bits, snr, "      -      " if prev is None else "%11.2f dB " % (snr - prev), mse))
    prev = snr
print("largest |x| in the sample: %.3f standard deviations" % np.abs(G).max())

# Output:
# bits   SNR dB   gain per added bit   MSE
#   2     0.39         -         9.106e-01
#   3     7.44          7.05 dB    1.797e-01
#   4    14.81          7.36 dB    3.298e-02
#   5    21.41          6.60 dB    7.213e-03
#   6    27.73          6.32 dB    1.681e-03
#   7    33.88          6.15 dB    4.084e-04
#   8    39.97          6.09 dB    1.004e-04
# largest |x| in the sample: 4.406 standard deviations

The gain per bit starts at 7.05 dB and drops to 6.09: it tends to the theoretical value of 6.02 from above. Between 7 and 8 bits the MSE goes from 4.084e-04 to 1.004e-04, a factor of 4.07. The first steps are worth more because at few bits the uniform-quantizer assumption does not hold: many values fall on zero and the maximum of the sample (4.4 standard deviations) sets a wide step. It is an explanation I derive from the shape of the numbers; I did not separate the contributions.

The rule to take away is this: every bit removed costs about six decibels, and two bits removed multiply the squared error by sixteen. That is why the move from 8 to 4 bits, which halves the bytes, brings the SNR down from 39.97 to 14.81 dB, 25.2 dB less: the format you choose in practice is not the one with the fewest bits, but the one in which the decibels that remain are enough for the task. A synthetic tensor cannot tell you what the task is; to measure it you need a task and a set of cases, like those in the lesson on evaluating generative systems.

How much a quantized weight really weighs

The name of a format states the bits of the values and says nothing about the rest: every scale that accompanies a group of weights takes up bytes too. To see it without assumptions I really pack the integers, two per byte at 4 bits and four per byte at 2, unpack them and check with an assert that they come back identical. Then I count the bytes the way a file would: packed weights plus an fp16 scale (2 bytes) for each group, for four group schemes.

def pack(q, bits):
    """Map signed integers to unsigned codes and put 8/bits of them in every byte."""
    off = 2 ** (bits - 1)
    u = (q.reshape(-1) + off).astype(np.uint8)
    per = 8 // bits
    u = u.reshape(-1, per)
    out = np.zeros(u.shape[0], dtype=np.uint8)
    for k in range(per):
        out |= (u[:, k] << (bits * k)).astype(np.uint8)
    return out

def unpack(b, bits, shape):
    off = 2 ** (bits - 1)
    per = 8 // bits
    mask = (1 << bits) - 1
    u = np.stack([(b >> (bits * k)) & mask for k in range(per)], axis=1)
    return (u.astype(np.int32) - off).reshape(shape)

n = W.size
print("weights:", n, "  float32 = %d bytes" % W.astype(np.float32).nbytes)
for bits in (8, 4, 2):
    q = quantize(W, bits, per_channel=True)[0]
    # with symmetric 2 bits the values are -1, 0, 1: code 0 (= -2) never appears
    pk = pack(q, bits)
    assert np.array_equal(unpack(pk, bits, q.shape), q)
    print("%d bits: %d packed bytes, identical reconstruction: yes" % (bits, pk.nbytes))
print()
print("bits  scale scheme            scale bytes  total bytes  bits/weight  formula b + 16/g")
for bits in (8, 4, 2):
    for name, g in (("per tensor", n), ("per channel (512)", 512), ("groups of 128", 128), ("groups of 32", 32)):
        scales = np.zeros(n // g, dtype=np.float16)         # one fp16 scale per group
        total = n * bits // 8 + scales.nbytes
        print("%3d   %-20s  %11d  %11d  %11.4f  %16.4f" % (bits, name, scales.nbytes, total, 8 * total / n, bits + 16 / g))

# Output:
# weights: 262144   float32 = 1048576 bytes
# 8 bits: 262144 packed bytes, identical reconstruction: yes
# 4 bits: 131072 packed bytes, identical reconstruction: yes
# 2 bits: 65536 packed bytes, identical reconstruction: yes
#
# bits  scale scheme            scale bytes  total bytes  bits/weight  formula b + 16/g
#   8   per tensor                      2       262146       8.0001            8.0001
#   8   per channel (512)            1024       263168       8.0312            8.0312
#   8   groups of 128                4096       266240       8.1250            8.1250
#   8   groups of 32                16384       278528       8.5000            8.5000
#   4   per tensor                      2       131074       4.0001            4.0001
#   4   per channel (512)            1024       132096       4.0312            4.0312
#   4   groups of 128                4096       135168       4.1250            4.1250
#   4   groups of 32                16384       147456       4.5000            4.5000
#   2   per tensor                      2        65538       2.0001            2.0001
#   2   per channel (512)            1024        66560       2.0312            2.0312
#   2   groups of 128                4096        69632       2.1250            2.1250
#   2   groups of 32                16384        81920       2.5000            2.5000

The packing is lossless (the three reconstruction lines confirm it) and the bytes match the formula bits + 16/g, where g is the number of weights that share a scale. With one scale for the whole tensor the overhead is invisible, 2 bytes out of 262,146. With one scale every 32 weights it becomes half a bit: at 4 bits you pay 4.5 bits per weight, and at 8 bits 8.5. If the scale were asymmetric, the zero point per group would have to be added: I did not include it in the count above, so those values are the minimum for the symmetric form.

The llama.cpp blocks, rewritten

The numbers 4.5 and 8.5 are no accident: they are the simplest formats of llama.cpp. In the source (ggml-common.h) block_q4_0 holds an fp16 scale and 16 bytes of 4-bit values, that is 18 bytes for 32 weights, and block_q8_0 an fp16 scale and 32 8-bit integers, 34 bytes; a static_assert in the file fixes the sizes. The reference quantization functions, quantize_row_q4_0_ref and quantize_row_q8_0_ref in ggml-quants.c, I read and rewrote in NumPy: for Q4_0 the scale is max / -8, where max is the signed value of largest magnitude in the block, and the code is min(15, int(x/d + 8.5)).

The NumPy version keeps two details that get lost in a version written "by eye". The sign in the scale makes the largest weight fall exactly on level -8, and lets you use all 16 codes instead of the 15 of my symmetric version. And the roundf of Q8_0 rounds halves away from zero, while np.round takes them to the even number: that is why I use sign · floor(|t| + 0.5). The comparison is on the same tensor W with groups of 32.

# llama.cpp-style blocks, rewritten from ggml-quants.c (quantize_row_q4_0_ref / q8_0_ref)
Q4_0 = np.dtype([("d", "<f2"), ("qs", "u1", 16)])     # 2 + 16 bytes per 32 weights
Q8_0 = np.dtype([("d", "<f2"), ("qs", "i1", 32)])     # 2 + 32 bytes per 32 weights
print("bytes per block of 32 weights: Q4_0 = %d, Q8_0 = %d  ->  %.2f and %.2f bits/weight" % (
    Q4_0.itemsize, Q8_0.itemsize, 8 * Q4_0.itemsize / 32, 8 * Q8_0.itemsize / 32))

def q4_0(x):
    b = x.reshape(-1, 32).astype(np.float32)
    i = np.abs(b).argmax(axis=1)
    mx = b[np.arange(len(b)), i]                       # the value WITH SIGN of largest magnitude
    d = (mx / -8).astype(np.float16)
    idv = np.where(d != 0, 1.0 / d.astype(np.float32), 0.0)[:, None]
    xi = np.minimum(15, np.floor(b * idv + 8.5)).astype(np.int32)
    return ((xi - 8) * d.astype(np.float32)[:, None]).reshape(x.shape)

def q8_0(x):
    b = x.reshape(-1, 32).astype(np.float32)
    d = (np.abs(b).max(axis=1) / 127).astype(np.float16)
    idv = np.where(d != 0, 1.0 / d.astype(np.float32), 0.0)[:, None]
    t = b * idv
    q = (np.sign(t) * np.floor(np.abs(t) + 0.5)).astype(np.int32)   # roundf: halves away from zero
    return (q * d.astype(np.float32)[:, None]).reshape(x.shape)

print()
print("scheme                                    bits/weight   SNR dB")
for name, r, bpw in (
    ("Q8_0 (blocks of 32, fp16 scale)", q8_0(W), 8.5),
    ("mine, 8-bit symmetric, groups of 32", quantize(W.reshape(-1, 32), 8, True)[3].reshape(W.shape), 8.5),
    ("Q4_0 (blocks of 32, fp16 scale)", q4_0(W), 4.5),
    ("mine, 4-bit symmetric, groups of 32", quantize(W.reshape(-1, 32), 4, True)[3].reshape(W.shape), 4.5),
    ("mine, 4-bit symmetric, per channel", quantize(W, 4, True)[3], 4.0 + 16 / 512),
    ("mine, 4-bit symmetric, per tensor", quantize(W, 4)[3], 4.0)):
    print("%-40s  %11.4f   %6.2f" % (name, bpw, metrics(W, r)[2]))

# Output:
# bytes per block of 32 weights: Q4_0 = 18, Q8_0 = 34  ->  4.50 and 8.50 bits/weight
#
# scheme                                    bits/weight   SNR dB
# Q8_0 (blocks of 32, fp16 scale)                8.5000    45.45
# mine, 8-bit symmetric, groups of 32            8.5000    45.45
# Q4_0 (blocks of 32, fp16 scale)                4.5000    21.28
# mine, 4-bit symmetric, groups of 32            4.5000    20.24
# mine, 4-bit symmetric, per channel             4.0312    17.37
# mine, 4-bit symmetric, per tensor              4.0000     5.09

Q8_0 and my symmetric 8-bit with groups of 32 give the same SNR, 45.45 dB: the reading of the layout and of the algorithm is consistent. Q4_0 gains 1.04 dB over my symmetric 4-bit (21.28 against 20.24), at equal bytes, thanks to 16 codes instead of 15. The comparison that weighs most is the last: the same 4 bits with one scale per tensor has an SNR of 5.09 dB, with one per row 17.37 and with one every 32 weights 20.24. Going from 4.0 to 4.5 bits per weight, half a bit of scales, is worth 15 dB. That is why real formats never quantize at "pure 4 bits".

The figures of real models, attributed

For the more complex formats, the k-quants (Q4_K_M, Q6_K, Q2_K), I do not redo the calculation from scratch: their structure of super-blocks with 6-bit scales and tensors treated differently is in the source, and the measurements on real models are in the documentation. The rows that follow I copied from the table in tools/quantize/README.md (Llama 3.1 8B: bits per weight and size in GiB). The listing derives the implied number of weights: if the table is consistent, seven different formats must give the same one.

# Rows copied from the table in llama.cpp's tools/quantize/README.md (Llama 3.1 8B):
# (bits per weight, size in GiB). The number of weights is derived from these two columns.
table = {"Q2_K": (3.1593, 2.95), "Q4_K_S": (4.6672, 4.36), "Q4_K_M": (4.8944, 4.58),
         "Q5_K_M": (5.7036, 5.33), "Q6_K": (6.5633, 6.14), "Q8_0": (8.5008, 7.95), "F16": (16.0005, 14.96)}
print("format    bits/weight  GiB     implied weights (billions)   error of reading the name as bits")
for name, (bpw, gib) in table.items():
    weights = gib * 2 ** 30 * 8 / bpw
    nominal = {"Q2_K": 2, "Q4_K_S": 4, "Q4_K_M": 4, "Q5_K_M": 5, "Q6_K": 6, "Q8_0": 8, "F16": 16}[name]
    print("%-8s  %11.4f  %5.2f   %.3f                        %+.1f %%" % (name, bpw, gib, weights / 1e9, 100 * (nominal - bpw) / bpw))

# Output:
# format    bits/weight  GiB     implied weights (billions)   error of reading the name as bits
# Q2_K           3.1593   2.95   8.021                        -36.7 %
# Q4_K_S         4.6672   4.36   8.025                        -14.3 %
# Q4_K_M         4.8944   4.58   8.038                        -18.3 %
# Q5_K_M         5.7036   5.33   8.027                        -12.3 %
# Q6_K           6.5633   6.14   8.036                        -8.6 %
# Q8_0           8.5008   7.95   8.033                        -5.9 %
# F16           16.0005  14.96   8.031                        -0.0 %

The seven formats imply between 8.021 and 8.038 billion weights: the table is consistent, with the dispersion due to the two decimal digits of the size. The last column is the reason this block exists. Whoever reads Q4_K_M as "four bits" underestimates the size by 18.3%; whoever reads Q2_K as "two bits" by 36.7%, because in that format the average weight is worth 3.16 bits. These are numbers from the documentation, not a measurement of mine on GGUF files: I used them to show a reading error, not to say what is inside the file in which format.

Outliers and clipping

The per-channel scale solves one problem, that of rows with different amplitudes. Another remains, a more insidious one: a few weights much larger than the others inside the same row. With the scale set on the maximum, a single anomalous value widens the step for everyone. I build the case in two stages. First a homogeneous tensor without outliers, to see whether shortening the ruler pays off even there; then the same tensor with 26 weights (0.01%) pushed to 25-40 standard deviations.

The parameter I touch is clip, already seen: the reference scale is the maximum multiplied by the value, and the weights that exceed it are saturated at the extreme level. I look for the best value on a grid, for each number of bits. In the second stage, besides the total SNR, I separate the error on the normal weights (the MSE without outliers) from the error on the outliers, because clipping moves the error from one side to the other and the total SNR alone does not say so.

rng = np.random.default_rng(7)
H = (0.02 * rng.standard_normal((512, 512))).astype(np.float32)   # a homogeneous tensor, sigma 0.02
sigma = float(H.std())
print("homogeneous tensor without outliers: reference scale = maximum * clip")
print("bits  clip   SNR dB  (the best of the group is marked with *)")
for bits in (8, 4, 2):
    res = [(clip, metrics(H, quantize(H, bits, clip=clip)[3])[2]) for clip in (1.0, 0.8, 0.6, 0.5, 0.4, 0.3, 0.2)]
    best = max(res, key=lambda t: t[1])
    print("%3d  " % bits + "   ".join("%.1f:%.2f%s" % (c, s, "*" if (c, s) == best else "") for c, s in res))
print()
Ho = H.copy().reshape(-1)
positions = rng.choice(Ho.size, 26, replace=False)                # 0.01% of the weights
Ho[positions] = rng.choice([-1, 1], 26) * rng.uniform(25, 40, 26) * sigma
Ho = Ho.reshape(H.shape)
mask = np.zeros(Ho.size, bool); mask[positions] = True; mask = mask.reshape(Ho.shape)
print("with %d outliers (25 to 40 sigma): max |w| = %.1f sigma" % (mask.sum(), np.abs(Ho).max() / sigma))
print("4-bit symmetric, per tensor")
print("clip     SNR dB   MSE without outliers   mean error on the outliers   weights at zero")
for clip in (1.0, 0.5, 0.25, 0.15, 0.10):
    r = quantize(Ho, 4, clip=clip)[3]
    e = Ho - r
    print("%-6.2f  %6.2f   %.3e              %.4f                       %5.1f %%" % (
        clip, metrics(Ho, r)[2], np.mean(e[~mask] ** 2), np.abs(e[mask]).mean(), 100 * np.mean(r == 0)))
print("(reference without outliers, clip 1: SNR %.2f dB, weights at zero %.1f %%)" % (
    metrics(H, quantize(H, 4)[3])[2], 100 * np.mean(quantize(H, 4)[3] == 0)))
print()
print("4-bit symmetric with the same outliers, more local scales (clip 1)")
for name, r in (("per channel (512 scales)", quantize(Ho, 4, True)[3]),
                ("groups of 128", quantize(Ho.reshape(-1, 128), 4, True)[3].reshape(Ho.shape)),
                ("groups of 32", quantize(Ho.reshape(-1, 32), 4, True)[3].reshape(Ho.shape))):
    print("%-24s SNR %.2f dB   weights at zero %.1f %%" % (name, metrics(Ho, r)[2], 100 * np.mean(r == 0)))
print()
print("signal power with outliers / without: %.3f" % (np.mean(Ho ** 2) / np.mean(H ** 2)))

# Output:
# homogeneous tensor without outliers: reference scale = maximum * clip
# bits  clip   SNR dB  (the best of the group is marked with *)
#   8  1.0:39.74   0.8:40.63*   0.6:29.62   0.5:22.96   0.4:17.05   0.3:11.86   0.2:7.32
#   4  1.0:14.58   0.8:16.52   0.6:18.68   0.5:18.68*   0.4:16.05   0.3:11.69   0.2:7.30
#   2  1.0:0.33   0.8:0.98   0.6:2.50   0.5:3.80   0.4:5.54   0.3:7.08*   0.2:6.39
#
# with 26 outliers (25 to 40 sigma): max |w| = 39.4 sigma
# 4-bit symmetric, per tensor
# clip     SNR dB   MSE without outliers   mean error on the outliers   weights at zero
# 1.00      0.50   3.930e-04              0.0263                        99.5 %
# 0.50      2.57   2.370e-04              0.2505                        84.1 %
# 0.25      7.07   6.605e-05              0.4475                        52.0 %
# 0.15      9.28   2.380e-05              0.5263                        32.7 %
# 0.10     10.11   1.053e-05              0.5657                        22.2 %
# (reference without outliers, clip 1: SNR 14.58 dB, weights at zero 25.4 %)
#
# 4-bit symmetric with the same outliers, more local scales (clip 1)
# per channel (512 scales) SNR 12.50 dB   weights at zero 22.1 %
# groups of 128            SNR 16.41 dB   weights at zero 17.0 %
# groups of 32             SNR 19.52 dB   weights at zero 13.5 %
#
# signal power with outliers / without: 1.105

Even without outliers shortening the ruler pays off, and it depends on the bits: at 8 bits the optimum of the grid is 0.8 (40.63 dB against 39.74, less than a decibel), at 4 bits it lies between 0.6 and 0.5 (18.68 dB against 14.58, four decibels) and at 2 bits it is 0.3 (7.08 dB against 0.33). The fewer bits there are, the more it pays to sacrifice the tails to thicken the centre, where the mass of the weights lies. This holds for bell-shaped weights; I did not verify it on heavy-tailed distributions.

With outliers the picture reverses. Without clipping, at 4 bits 99.5% of the weights go to zero (the step is 5.6 standard deviations, and below 2.8 the weight rounds to zero) and the SNR drops to 0.50 dB: 26 weights have destroyed the representation of 262,118. With clip 0.10 the SNR rises back to 10.11 dB and the weights at zero fall to 22.2%, but the outliers are saturated: the mean error on them goes from 0.0263 to 0.5657, that is they lose almost all their value. If those 26 weights matter for the output, clipping has removed exactly the ones that mattered. The trade is not decided on a tensor: it is decided on a task.

More local scales avoid having to choose. With 512 scales (one per row) the SNR with outliers is 12.50 dB, with groups of 128 it is 16.41 and with groups of 32 it is 19.52, more than the reference without outliers with a single scale (14.58). The comparison is not exact, because the outliers raise the signal power by a factor of 1.105 (last line of the output), but the order of magnitude holds: the outlier infects only its own group, and a group of 32 weights is small. The same idea lies behind the block formats, and the price is the half bits per weight of the earlier count.

From weights to the output of a layer

What does a 12% error on the weights do to the output? An explanation you often hear is that "errors cancel out", because each output sums thousands of products. I measure it on a 256×256 linear layer with independent Gaussian inputs: I compare the relative error on the weights with that on the output y = X · Aᵀ, obtained with the same quantized weights. Then I quantize the activations too, W8A8, computing the multiplication with 32-bit integers and rescaling: it is what the hardware does with integer instructions.

The second part of the listing reproduces, in miniature, the phenomenon that the LLM.int8() paper describes in large models: a few input channels with an amplitude much greater than the others, which dominate the result. I multiply four columns of X by 40 and measure how much W8A8 suffers with a single scale for the activations. The repair is the decomposition described in the abstract of the paper, rewritten by me in a few lines: the columns with a maximum above a threshold (6, my value, chosen because normal columns do not reach it) stay in floating point, and the rest goes through at 8 bits.

rng = np.random.default_rng(11)
din, dout, nx = 256, 256, 2048
A = (rng.standard_normal((dout, din)) * np.exp(0.5 * rng.standard_normal((dout, 1))) / np.sqrt(din)).astype(np.float32)
X = rng.standard_normal((nx, din)).astype(np.float32)
Y = X @ A.T
def rel(y, yq): return float(np.linalg.norm(y - yq) / np.linalg.norm(y))
print("linear layer %dx%d, %d Gaussian inputs; relative error = ||y - yq|| / ||y||" % (dout, din, nx))
print("bits  scale        error on weights   error on output   ratio")
for bits in (8, 4, 2):
    for pc in (False, True):
        r = quantize(A, bits, pc)[3]
        ew, ey = rel(A, r), rel(Y, X @ r.T)
        print("%3d   %-11s  %.5f            %.5f           %.3f" % (bits, "per channel" if pc else "per tensor", ew, ey, ey / ew))
print()
# W8A8: activations at 8 bits too, integer matmul in int32 and rescaling
def w8a8(X, A, threshold=None):
    qw, sw, _, _ = quantize(A, 8, True)                          # one scale per output channel
    if threshold is None:
        mo = np.zeros(X.shape[1], bool)
    else:
        mo = np.abs(X).max(axis=0) > threshold                   # "outlier" columns, kept in float
    qx, sx, _, _ = quantize(X[:, ~mo], 8)                        # one scale for the whole matrix
    y = (qx.astype(np.int32) @ qw[:, ~mo].astype(np.int32).T) * sx * sw.T
    if mo.any():
        y = y + X[:, mo] @ (qw[:, mo] * sw).T
    return y, int(mo.sum())
Xo = X.copy()
channels = rng.choice(din, 4, replace=False)
Xo[:, channels] *= 40                                            # 4 input channels with 40 times the amplitude
Yo = Xo @ A.T
print("normal activations:      weights-only 8 bits %.5f   W8A8 %.5f" % (rel(Y, X @ quantize(A, 8, True)[3].T), rel(Y, w8a8(X, A)[0])))
print("4 channels at 40x:       weights-only 8 bits %.5f   W8A8 %.5f" % (rel(Yo, Xo @ quantize(A, 8, True)[3].T), rel(Yo, w8a8(Xo, A)[0])))
y3, n3 = w8a8(Xo, A, threshold=6.0)
print("4 channels at 40x, threshold 6 (columns kept in float: %d):  W8A8 %.5f" % (n3, rel(Yo, y3)))

# Output:
# linear layer 256x256, 2048 Gaussian inputs; relative error = ||y - yq|| / ||y||
# bits  scale        error on weights   error on output   ratio
#   8   per tensor   0.02579            0.02571           0.997
#   8   per channel  0.00697            0.00696           0.998
#   4   per tensor   0.44769            0.44644           0.997
#   4   per channel  0.12691            0.12626           0.995
#   2   per tensor   0.99035            0.99025           1.000
#   2   per channel  0.80964            0.80732           0.997
#
# normal activations:      weights-only 8 bits 0.00696   W8A8 0.01321
# 4 channels at 40x:       weights-only 8 bits 0.00682   W8A8 0.07029
# 4 channels at 40x, threshold 6 (columns kept in float: 4):  W8A8 0.00714

The ratio between output error and weight error is between 0.995 and 1.000 in all six rows: errors do not cancel out, they transfer one to one. With independent inputs of equal amplitude each column of the weights contributes in proportion to its own square, and the relative error of the output equals that of the weights. At 4 bits with a single scale the output has a 44.6% error, at 4 bits per channel 12.6%, at 8 bits per channel 0.70%. In real models the inputs are neither independent nor of equal amplitude, and the abstract of AWQ argues precisely that not all weights count the same (protecting 1% of them reduces the error a lot, according to the authors): this calculation, therefore, is the simplest case, not a prediction.

The second half says something more practical. With normal activations W8A8 is wrong by 1.32%, against the 0.70% of weights-only at 8 bits: quantizing the activations too doubles the error. With four channels at 40 times the amplitude weights-only does not move (0.68%), while W8A8 rises to 7.03%, more than five times higher: the single scale of the activations widens for the anomalous channels and crushes the others. Keeping those four columns in floating point brings the error back to 0.71%. The cost is a special case in the kernel, and four columns out of 256 are 1.6%; the percentages of real models are reported by the paper (more than 99.9% of the values stay in 8 bits), not by this toy.

A small network, trained

A tensor says how wrong the representation is, not whether the result is useful. To get closer to a question of accuracy I train a tiny network: 32 inputs, 64 hidden neurons with ReLU, 10 classes, 2,762 parameters, on synthetic data (ten overlapping Gaussian clouds, fixed seed, 4,000 training examples and 2,000 test examples). The training is a hand-written gradient descent, 300 steps with the full batch. Then I quantize the two weight matrices (not the biases) at 8, 4, 3 and 2 bits, with one scale per tensor or per channel, without retraining, and count the right answers on the test set.

I add a second column, how many predictions coincide with those of the floating-point network, because two networks can have the same accuracy while being wrong on different examples. The last line of the output gives the standard error of the accuracy with 2,000 examples: it serves to decide which differences are noise. The domain is a toy, chosen because it trains in seconds on a CPU and because every figure can be redone; it is not a measurement on a language model.

rng = np.random.default_rng(5)
K, D, H1, NTR, NTE = 10, 32, 64, 4000, 2000
centers = rng.standard_normal((K, D)) * 1.1
def sample(n):
    y = rng.integers(0, K, n)
    return (centers[y] + rng.standard_normal((n, D)) * 1.6).astype(np.float32), y
Xtr, ytr = sample(NTR); Xte, yte = sample(NTE)
def softmax(z):
    e = np.exp(z - z.max(1, keepdims=True)); return e / e.sum(1, keepdims=True)
W1 = (rng.standard_normal((H1, D)) * np.sqrt(2 / D)).astype(np.float32); b1 = np.zeros(H1, np.float32)
W2 = (rng.standard_normal((K, H1)) * np.sqrt(2 / H1)).astype(np.float32); b2 = np.zeros(K, np.float32)
def forward(X, W1, b1, W2, b2):
    h = np.maximum(X @ W1.T + b1, 0); return h, h @ W2.T + b2
for epoch in range(300):                                           # gradient descent, full batch
    h, z = forward(Xtr, W1, b1, W2, b2)
    g = softmax(z); g[np.arange(NTR), ytr] -= 1; g /= NTR
    gW2, gb2 = g.T @ h, g.sum(0)
    gh = (g @ W2) * (h > 0)
    gW1, gb1 = gh.T @ Xtr, gh.sum(0)
    for P, G in ((W1, gW1), (b1, gb1), (W2, gW2), (b2, gb2)): P -= 0.5 * G
_, zf = forward(Xte, W1, b1, W2, b2)
pf = zf.argmax(1)
print("network %d-%d-%d, %d parameters; float32: %d / %d correct on the test set (%.1f %%)" % (
    D, H1, K, W1.size + b1.size + W2.size + b2.size, (pf == yte).sum(), NTE, 100 * (pf == yte).mean()))
print()
print("bits  scale         correct   same as float   relative error on the logits")
for bits in (8, 4, 3, 2):
    for pc in (False, True):
        q1, q2 = quantize(W1, bits, pc)[3], quantize(W2, bits, pc)[3]
        _, zq = forward(Xte, q1, b1, q2, b2)
        pq = zq.argmax(1)
        print("%3d   %-11s  %5d     %5d           %.4f" % (
            bits, "per channel" if pc else "per tensor", (pq == yte).sum(), (pq == pf).sum(),
            np.linalg.norm(zf - zq) / np.linalg.norm(zf)))
p = (pf == yte).mean()
print()
print("standard error of the accuracy with %d examples: %.2f percentage points" % (NTE, 100 * np.sqrt(p * (1 - p) / NTE)))

# Output:
# network 32-64-10, 2762 parameters; float32: 1878 / 2000 correct on the test set (93.9 %)
#
# bits  scale         correct   same as float   relative error on the logits
#   8   per tensor    1879      1999           0.0079
#   8   per channel   1878      2000           0.0046
#   4   per tensor    1870      1946           0.1483
#   4   per channel   1877      1969           0.0833
#   3   per tensor    1792      1838           0.3371
#   3   per channel   1838      1913           0.2328
#   2   per tensor     621       614           0.9595
#   2   per channel   1521      1553           0.7381
#
# standard error of the accuracy with 2000 examples: 0.54 percentage points

The floating-point network gets 1,878 out of 2,000. The standard error is 0.54 percentage points, about 11 examples: anything within a dozen answers is indistinguishable from zero. At 8 bits (1,879 and 1,878) and at 4 bits per channel (1,877) quantization does not change the accuracy, and at 4 bits per tensor it loses 8 answers (1,870), still inside the noise. The column next to it shows what accuracy hides, though: at 4 bits per channel, 31 predictions out of 2,000 differ from those of the float, and the tie in accuracy is the result of changes that balance out.

The cliff begins below 4 bits. At 3 bits per tensor the network loses 86 answers (1,792), 8 times the noise, and per channel 40 (1,838). At 2 bits per tensor it gets 621 right, 31%: a little more than three times chance (10 classes, about 10%), and with the per-channel scale 1,521, 76%. The relative error on the logits tells the same story: 0.0046 at 8 bits per channel, 0.083 at 4, 0.23 at 3, 0.74 at 2. The numbers belong to one network, one seed, one synthetic domain; the point is not the threshold, which in real models will be elsewhere, but the shape: an almost flat plateau down to 4 bits, then a drop.

PyTorch dynamic quantization

PyTorch has a ready-made quantization for nn.Linear layers: torch.ao.quantization.quantize_dynamic. The official documentation describes it as the conversion of a float model into a quantized one "dynamic (i.e. weights-only)", with dtype torch.qint8 by default, a set of module types to replace and inplace=False so as not to modify the original. I verified in this version (2.12.0) that the API exists and works, but the call emits a DeprecationWarning: the text of the warning says that the torch.ao.quantization module is deprecated and will be removed in 2.10, and points to torchao. The warning and the fact that on 2.12 the function runs coexist: it is a trap to know about before writing code that has to live long.

The listing builds a three-layer network (1,024, 1,024, 10 neurons), quantizes it, serializes the two state_dicts in memory to measure their bytes, inspects the weight of the first layer and compares the output. Then it redoes by hand the same calculation with my symmetric 8-bit weights per tensor, first with floating-point activations, then with activations quantized to 256 and to 128 levels. The last two tests arise from a suspicion: with weights-only at 8 bits I expect an output error much smaller than the one PyTorch produces.

import io, os, warnings
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import numpy as np, torch
from torch import nn
torch.set_num_threads(2)
torch.manual_seed(2398)
print("torch", torch.__version__, "- quantization engine:", torch.backends.quantized.engine)

model = nn.Sequential(nn.Linear(1024, 1024), nn.ReLU(), nn.Linear(1024, 1024), nn.ReLU(), nn.Linear(1024, 10)).eval()
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    qm = torch.ao.quantization.quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)
print("warnings raised by the call:", sorted({a.category.__name__ for a in caught}))

def serialized_bytes(m):
    buf = io.BytesIO(); torch.save(m.state_dict(), buf); return buf.getbuffer().nbytes

bf, bq = serialized_bytes(model), serialized_bytes(qm)
print("float32 : %d serialized bytes (state_dict)" % bf)
print("qint8   : %d serialized bytes (state_dict), %.2f times smaller" % (bq, bf / bq))
w, b = qm[0]._packed_params._weight_bias()
print("quantized weight:", w.dtype, "scheme", w.qscheme(), "scale %.3e" % w.q_scale(), "zero point", w.q_zero_point())

x = torch.randn(256, 1024)
with torch.no_grad():
    y, yq = model(x).numpy(), qm(x).numpy()
rel = lambda a, b: float(np.linalg.norm(a - b) / np.linalg.norm(a))
print("relative output error, PyTorch qint8 against float: %.5f" % rel(y, yq))

def weights8(W):                                 # 8-bit symmetric, one scale per tensor
    s = np.abs(W).max() / 127
    return np.clip(np.round(W / s), -127, 127) * s
def act(h, levels):                              # activations: affine per tensor, with `levels` levels
    lo, hi = min(h.min(), 0), max(h.max(), 0)
    s = (hi - lo) / (levels - 1); z = np.round(-lo / s)
    return (np.clip(np.round(h / s) + z, 0, levels - 1) - z) * s
def network(levels=None):
    h = x.numpy()
    for l in model:
        if isinstance(l, nn.Linear):
            h = (h if levels is None else act(h, levels)) @ weights8(l.weight.detach().numpy()).T + l.bias.detach().numpy()
        else:
            h = np.maximum(h, 0)
    return h
print("by hand, weights-only at 8 bits:                       %.5f" % rel(y, network()))
print("by hand, 8-bit weights and 256-level activations:      %.5f" % rel(y, network(256)))
print("by hand, 8-bit weights and 128-level activations:      %.5f" % rel(y, network(128)))

# Output:
# torch 2.12.0+cu130 - quantization engine: x86
# warnings raised by the call: ['DeprecationWarning']
# float32 : 8440469 serialized bytes (state_dict)
# qint8   : 2120255 serialized bytes (state_dict), 3.98 times smaller
# quantized weight: torch.qint8 scheme torch.per_tensor_affine scale 2.451e-04 zero point 0
# relative output error, PyTorch qint8 against float: 0.02677
# by hand, weights-only at 8 bits:                       0.00665
# by hand, 8-bit weights and 256-level activations:      0.01451
# by hand, 8-bit weights and 128-level activations:      0.02654

The size drops by 3.98 times (from 8,440,469 to 2,120,255 bytes): not exactly four, because the biases stay float32 and the file has metadata. The weight is qint8 with per-tensor scheme and zero point 0, that is symmetric, like mine. The output error is 0.02677, four times what I get by hand with weights only (0.00665). It does not match the documented "weights-only"; it does match, instead, my calculation that quantizes the activations too at 128 levels (0.02654). It is a deduction from the numbers: I did not read the kernel, and the match at 128 levels would be explained by an activation with 7 bits of precision ("reduce range"), which is a hypothesis, not a verified fact. The solid datum is the other one: the output of quantize_dynamic departs from the float by 2.7%, and not by the 0.7% that the name "weights only" would suggest.

The timing remains. The listing that follows runs the float model and the quantized one on a batch of 256, seven times per round, for three rounds, and prints the minimum and maximum of each round. It is the only block that does not reproduce the same output from one run to the next, by construction.

import os, time
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import warnings; warnings.simplefilter("ignore")
import torch
from torch import nn
torch.set_num_threads(2); torch.manual_seed(2398)
m = nn.Sequential(nn.Linear(1024, 1024), nn.ReLU(), nn.Linear(1024, 1024), nn.ReLU(), nn.Linear(1024, 10)).eval()
q = torch.ao.quantization.quantize_dynamic(m, {nn.Linear}, dtype=torch.qint8)
x = torch.randn(256, 1024)
def timings(f, n=7):
    with torch.no_grad():
        f(x); t = []
        for _ in range(n):
            a = time.perf_counter(); f(x); t.append((time.perf_counter() - a) * 1e3)
    return min(t), max(t)
for rnd in range(3):
    f32, i8 = timings(m), timings(q)
    print("round %d  float32 %5.1f - %5.1f ms   qint8 %5.1f - %5.1f ms" % (rnd + 1, *f32, *i8))

# Output (changes on every run, loaded machine):
# round 1  float32  47.1 -  85.9 ms   qint8  26.3 -  47.9 ms
# round 2  float32  31.1 -  84.9 ms   qint8   7.3 -  49.9 ms
# round 3  float32  10.1 -  16.3 ms   qint8  13.4 -  42.1 ms

Of the three rounds, two see the quantized model faster (minima 26.3 and 7.3 ms against 47.1 and 31.1) and one does not (13.4 against 10.1 ms): with the machine loaded, and at a scale of this order, it is not a speed measurement. What can be said is that the advantage, if there is one, is of the same order as the dispersion, and that a serious measurement requires a free machine, more repetitions and the comparison on the real model. The memory advantage, on the other hand, is a calculation and not a time measurement: 3.98 times fewer bytes.

From tensors to real models: schemes to adapt

What precedes refers to weights that I generated myself. For a real model the commands are different, and I did not run them here: they require weights that I did not download. I report them because they are the destination of the lesson. The first is the llama-quantize command of llama.cpp, taken from the documentation: it converts a 16-bit GGUF into the chosen format, here Q4_K_M. It is a scheme to adapt, not run: the file names are those of the official example, and the binary is obtained by compiling the project.

# scheme to adapt, NOT run here (requires a 16-bit GGUF and a compiled llama.cpp)
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M

The second is quantization at load time with bitsandbytes, exposed by Transformers through BitsAndBytesConfig. In the Hugging Face documentation, load_in_4bit=True enables 4 bits and bnb_4bit_quant_type="nf4" chooses the NF4 type, described as suited to weights initialized from a normal distribution; for inference the choice of type, they say, does not have a big impact. The data type comes from the QLoRA paper, which defines it as "information theoretically optimal for normally distributed weights". This too is a scheme to adapt, not run: model_id is a placeholder for a model of your choice, and a supported GPU is required.

# scheme to adapt, NOT run here (requires a supported GPU and the model weights)
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

nf4_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")
model_nf4 = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", quantization_config=nf4_config)

The other names you will meet each have a paper, and are described by those who wrote them. GPTQ is a "one-shot" weight quantization based on approximate second-order information: the authors state that they quantize models of 175 billion parameters in about four GPU hours, at 3 or 4 bits per weight, with negligible degradation compared with the original. AWQ, already cited, protects the "salient" channels by scaling them, and chooses them by looking at the distribution of the activations, not of the weights. LLM.int8() isolates the anomalous dimensions in a 16-bit multiplication. These are statements by the authors about their models and their benchmarks: I did not reproduce them, and I do not present them as my measurements.

What I did not measure

The boundaries of what this lesson can say have to be written down, because the numbers you have read are true and small. I do not know how much memory a real model takes while running, because besides the weights you need the key-value cache, which grows with the context and does not depend on the format of the weights, plus the compute graph. I do not know how many tokens per second it produces: it depends on the memory bandwidth of your machine and on the available kernels, and the llama.cpp documentation gives processing and generation times for each format, which I did not reproduce and do not report. I do not know how much quality is lost on a real task: the accuracy on the toy has a shape, not a threshold.

What this lesson does demonstrate holds on any model: bytes can be computed (bits + 16/g per weight), the error drops by six decibels per bit, outliers decide what the scale is worth, and the real cost is measured on the task, with a set of cases and a confidence interval, just as in the lesson on RAG from scratch a retrieval was measured. The weights you quantize in a real model are the projection matrices of attention and of the feed-forward blocks, the ones that the lesson on the Transformer from scratch builds with 27,253 parameters.

Trade-offs

• Precision against memory. Every bit removed costs about six decibels; at 4 bits with local scales the small network does not change accuracy, at 3 and 2 it loses it. Mitigation: go down one step at a time, measuring on your own task.
• Local scales against bytes. A group of 32 weights adds half a bit per weight and, in our tensor, is worth 15 dB at 4 bits compared with the single scale. Mitigation: choose the group size by looking at the formula bits + 16/g before looking at the SNR.
• Clipping against outliers. Shortening the ruler improves the centre and destroys the tails; if the tails matter, that is exactly what you lose. Mitigation: per-group scales, or keep the outliers in higher precision, as LLM.int8() does.
• Weights only against weights and activations. Quantizing the activations speeds up integer arithmetic and doubles the error in the toy, up to multiplying it by five with anomalous channels. Mitigation: start with weights only.

The four lines are four ways of saying the same thing: quantization moves the cost from one place to another, and the best place depends on what your task does not tolerate. If bytes are the constraint, you choose local scales and pay half a bit; if quality on the tails is the constraint, you leave the outliers aside. None of the four choices is wrong in itself: it becomes so when it is made without having measured.

The traps you really fall into

• Estimating the size from the format name. Symptom: "Q4 means half a byte per parameter" and a model that does not fit. Cause: scales and tensors promoted in precision bring Q4_K_M to 4.8944 bits per weight. Remedy: weights × bits per weight / 8, with the bits per weight from the documentation or from the file.
• Judging by accuracy alone. Symptom: same accuracy and different behaviour. Cause: in the test network, at 4 bits per channel, 31 predictions out of 2,000 change and compensate each other. Remedy: also look at how many answers coincide with the original model.
• Trusting the description of an API. Symptom: quantize_dynamic described as "weights only" and an output error four times larger than expected. Cause: probable quantization of the activations at runtime. Remedy: measure the output error against the float before trusting it.
• Trusting a deprecated API. Symptom: DeprecationWarning ignored. Cause: the module is announced as removed in 2.10 and is still present in 2.12. Remedy: read the warning and decide whether to migrate to torchao before writing more code on top of it.
• Cutting the outliers because the SNR goes up. Symptom: better SNR and worse output. Cause: clipping reduces the error on the centre and moves it to the tails. Remedy: measure the error on the output, not only on the weights.

What these five traps have in common is a habit: reading a number or a name as if it were the thing it describes. The bit in the name is not the bit in the file, the word "weights" does not say what is being quantized, and the SNR of the weights is not the error of the output. The defence is the same as in the whole lesson: a calculation you can redo, next to every claim.

Exercises

Frequently asked questions

Is Q4_K_M really worth 4 bits per weight?

No. In the llama.cpp documentation table for Llama 3.1 8B it is worth 4.8944 bits per weight, because every block carries its scales with it and because some tensors are saved with more bits. The name indicates the family of the format, not its size: for the size you need the bits per weight, which are read from the table or derived from the file.

Is a scale per tensor or per channel better?

Per channel, when the weights have different amplitudes from row to row. In the test tensor it gains 13.2 dB at 8 bits and 12.3 dB at 4 bits, at the cost of two bytes per row (1,024 bytes over 262,144 weights). With a single amplitude for all the rows the gain disappears: measure it on your own weights.

Is quantizing the weights enough to save memory?

It reduces that of the weights, which the bytes counted above make certain: 3.98 times fewer in the test network with qint8. By itself it does not reduce the key-value cache or the compute buffers of a running model, which this lesson did not measure.

What you can do now

You can quantize a tensor to several bit widths, with one scale per tensor, per row or per group, and you can measure the error with MSE, maximum error and SNR. You can compute the bytes of a format with bits + 16/g, verify them by packing the integers and recognise why Q4_K_M is not worth four bits. You can predict what an outlier does to a single scale, choose a clip on a grid and understand what clipping sacrifices. You can measure how the error of the weights becomes the error of the output of a layer and of a network, and how much the difference is between quantizing only the weights and the activations as well. And you know what you do not have yet: the quality of a real model on a real task, which is measured with a set of cases, and not with a tensor.

The same series also includes ML-Ops: from notebook to service (next article).

Sources

• llama.cpp, tools/quantize/README.md - the table of bits per weight and size for Llama 3.1 8B (Q4_K_M 4.8944 bits, 4.58 GiB; F16 14.96 GiB) and the llama-quantize command; consulted on 2026-11-12.
• llama.cpp, ggml/src/ggml-common.h - the block_q4_0 and block_q8_0 structures and their sizes; consulted on 2026-11-12.
• llama.cpp, ggml/src/ggml-quants.c - the quantize_row_q4_0_ref and quantize_row_q8_0_ref algorithms rewritten in NumPy; consulted on 2026-11-12.
• Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv:2208.07339) - the isolation of anomalous dimensions in a 16-bit multiplication, with more than 99.9% of the values in 8 bits; consulted on 2026-11-12.
• Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323) - one-shot quantization at 3-4 bits with second-order information; consulted on 2026-11-12.
• Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv:2306.00978) - weights do not all count the same and salient channels are chosen from the activations; consulted on 2026-11-12.
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314) - the NF4 data type for normally distributed weights; consulted on 2026-11-12.
• Hugging Face, Transformers: bitsandbytes - BitsAndBytesConfig, load_in_4bit and bnb_4bit_quant_type; consulted on 2026-11-12.
• PyTorch, torch.ao.quantization.quantize_dynamic - signature, default dtype and the "weights-only" description; consulted on 2026-11-12 (the destination page of the "stable" address redirects to 2.15).
• The figures on tensors, on the network and on PyTorch come from the listings shown, run twice with identical output (the timings: three rounds of seven runs, reported as a range). No language model was run and no weights were downloaded.

The nine external sources are primary and I opened them on 2026-11-12 and read the passages cited: in the llama.cpp README and sources the table, the structures and the two quantization functions; in the arXiv abstracts the statements attributed to the authors; in the Hugging Face and PyTorch documentation the examples and the description of quantize_dynamic. Of the figures in the papers I report only those that appear in their abstracts, attributed to the authors. The last entry declares where the measured figures come from: a measurement made in this repository is cited with the program that produces it.