Quantization: smaller numbers

Llama 2 7B has 6.74 billion parameters. Stored as 16-bit numbers, its weights take 13.5 GB, more memory than most laptop graphics cards have. Stored in 4 bits, they take about 3.4 GB and fit. And as the KV cache lesson showed, generating text one token at a time is usually limited by how fast weights can be read from memory, so smaller weights are also faster weights.

Storing numbers in fewer bits is called quantizationQuantizationStoring a model's numbers in fewer bits, for example as 8-bit or 4-bit integers with a scale, to save memory and speed up inference at some cost in accuracy.Open in glossary. It always loses something. This lesson is about how much, and why.

Number formats

Training usually stores numbers as floating pointFloating pointA way to store real numbers as a sign, an exponent that sets the scale, and a fraction that sets the precision. float32 and float16 are the common 32-bit and 16-bit IEEE 754 formats.Open in glossary: a sign, an exponent that sets the scale, and a fraction that sets the precision within that scale. More exponent bits give more range; more fraction bits give finer steps.

The last column is the precision: the next number after 1 is $1 + 2^{-10} \approx 1.001$ in float16 but $1 + 2^{-7} \approx 1.008$ in bfloat16 (largest values are rounded). float16 and bfloat16 both use 16 bits but split them differently. float16 is more precise but overflows above 65,504. bfloat16, designed at Google Brain, keeps float32’s 8 exponent bits, so it has the same range with less precision. That is why it is popular for training, where gradients and activations can span huge ranges. Newer hardware also supports 8-bit floating point formats (Micikevicius et al., 2022).

Integers have no exponent at all: 8 bits give 256 evenly spaced values, 4 bits give 16. To store a weight like 0.0137 as an integer, you need a scale that says what one step is worth.

Scales: absmax and zero-point

The simplest recipe, absmax quantizationAbsmax quantizationQuantization that sets the scale from the largest magnitude in a block, so that value maps to the largest integer. Levels are symmetric around zero.Open in glossary, uses the largest magnitude in a block of numbers to set the scale, so that it maps to the largest integer:

With $b = 8$ bits, the largest magnitude becomes $\pm 127$. The model stores the integers $q_i$ and one scale $s$, and computes with $\hat{x}_i$. Rounding to the nearest level means every error is at most half a step: $|x_i - \hat{x}_i| \le s / 2$.

Absmax puts its levels symmetrically around zero. If the values are all positive, as activations after a ReLU are, half the levels are wasted. Zero-point quantizationZero-point quantizationQuantization that maps a block's actual range, from its minimum to its maximum, onto the integers, with an integer offset marking where zero lands. Suits values that are not centered on zero.Open in glossary maps the actual range $[\min, \max]$ onto $0, \dots, 2^b - 1$ with a scale and an integer offset $z$ that marks where zero lands:

(The range is widened to include 0 if needed, so that zero is stored exactly.)

One scale, or many?

A whole weight matrix can share one scale (per tensor), or each row, or each group of, say, 64 or 128 consecutive weights can have its own. Group-wise quantizationGroup-wise quantizationGiving each group of consecutive values, such as 64 or 128 weights, its own quantization scale, so one large value only affects its own group.Open in glossary costs a little storage, one scale per group, but each group’s step size fits its own values.

This matters because of outliersOutlierA data point far from the pattern of the rest, from a rare event or a recording error. Some loss functions are pulled hard by outliers.Taught in Linear regression and lossOpen in glossary. One value far larger than the rest sets the scale for everything that shares it, and stretches the step between levels until ordinary values round to zero.

import numpy as np

def absmax(x, bits, group):
    """Quantize to signed integers with one scale per group, then dequantize."""
    qmax = 2 ** (bits - 1) - 1                    # 127 for 8 bits, 7 for 4 bits
    g = x.reshape(-1, group)
    scale = np.abs(g).max(axis=1, keepdims=True) / qmax
    q = np.round(g / scale)                       # the stored integers
    return (q * scale).reshape(-1)                # what the model computes with

rng = np.random.default_rng(0)
w = rng.normal(0, 0.02, size=128)                 # synthetic weights
typical = np.sqrt(np.mean(w ** 2))
w_out = w.copy()
w_out[45] = 0.4                                   # one outlier, 20 standard deviations out

for name, x in [("weights", w), ("with outlier", w_out)]:
    for bits, group in [(8, 128), (4, 128), (4, 16)]:
        err = absmax(x, bits, group) - x
        zeros = np.sum(absmax(x, bits, group) == 0)
        print(f"{name:12} {bits}-bit, {group:3} per scale: typical error "
              f"{np.sqrt(np.mean(err ** 2)) / typical:6.1%}, {zeros:3} rounded to zero")

# Memory for Llama 2 7B (6,738,415,616 parameters).
n = 6_738_415_616
for label, bits in [("float32", 32), ("float16", 16), ("int8", 8),
                    ("4-bit + a 16-bit scale per 128", 4 + 16 / 128)]:
    print(f"{label:31} {n * bits / 8 / 1e9:5.1f} GB")

weights      8-bit, 128 per scale: typical error   0.5%,   1 rounded to zero
weights      4-bit, 128 per scale: typical error   9.4%,  17 rounded to zero
weights      4-bit,  16 per scale: typical error   7.6%,  12 rounded to zero
with outlier 8-bit, 128 per scale: typical error   4.8%,   6 rounded to zero
with outlier 4-bit, 128 per scale: typical error  85.6%, 113 rounded to zero
with outlier 4-bit,  16 per scale: typical error  32.9%,  22 rounded to zero
float32                          27.0 GB
float16                          13.5 GB
int8                              6.7 GB
4-bit + a 16-bit scale per 128    3.5 GB

The demo’s numbers are synthetic, but the effect is the real one. In the demo, at 4 bits with a single scale, one outlier sends 113 of the other 127 weights to exactly zero, and the typical error jumps from 12% of a typical weight to 82%. Giving every 16 weights their own scale brings it back to 28%, because the outlier now spoils only its own group.

Where the outliers are

Real models do have outliers, and they matter most in activations, the vectors flowing between layers. Dettmers et al. (2022) found that in transformers of around 6.7 billion parameters and more, a few hidden dimensions carry values far larger than the rest, consistently across layers. Standard 8-bit quantization, which lets those few values set the scale, lost much of these models’ accuracy. Their method, LLM.int8(), keeps those few dimensions in 16 bits and multiplies everything else, more than 99.9% of the values, in 8 bits; they report inference for models up to 175 billion parameters without loss of performance.

Better methods choose more than just a scale. GPTQ (Frantar et al., 2022) quantizes weights one column at a time and adjusts the remaining weights to compensate for each rounding error, using a little calibration data; the authors report 3 or 4 bits per weight for models with 175 billion parameters with negligible loss in accuracy.

What it costs in memory

These are weights only (GB here means $10^9$ bytes). Running a model also needs memory for activations and the KV cache, which can be quantized too. Scales are not free: the last column stores 4.125 bits per weight. QLoRA (Dettmers et al., 2023) stores the frozen base model in a 4-bit format and trains LoRA adapters on top; its 32-bit scales for blocks of 64 weights would add 0.5 bits per weight, so it quantizes the scales themselves, saving about 0.37 bits per parameter. With these tricks the authors fine-tuned a 65-billion-parameter model on a single 48 GB GPU.

A measured case: SmolLM2-135M on this site

The live model lab runs SmolLM2-135M in your browser. Smaller 8-bit and 4-bit versions of it are published, and would cut the download to half or less. Before choosing, this site compared each published version (as ONNX exports) with the full 32-bit model on the same 24 prompts, measuring how far each version’s next-token probabilities moved, with the KL divergenceKL divergenceA measure of how much one probability distribution differs from another, in bits or nats. It is zero only when the two are identical. In RLHF a KL penalty keeps the trained model close to where it started.Open in glossary in bits, and whether it picked the same most likely token.

• The 16-bit version matched: KL 0.000 bits, and the same top token every time.
• Every 8-bit and 4-bit export distorted the distribution badly: KL 0.7 to 1.2 bits, and the top token agreed on only 13 to 16 of the 24 prompts.
• One example: after “Twinkle, twinkle, little”, the 8-bit version gives ” star” about 3% instead of 29%.

A lab about a model’s probabilities needs faithful weights, so the site’s SmolLM2 labs download the model’s original weights, which are 16-bit (bfloat16), and compute in 32-bit; their probabilities match a reference implementation in PyTorch to within a few millionths. This is one small model and a set of general-purpose exports; other methods, such as ones that use calibration data or keep sensitive layers in 16 bits, might well do better, and this site did not test them. The lesson is narrower: quantization changes a model’s outputs, by an amount you have to measure rather than assume.