Latent

Squeeze the weights

Quantize 128 weights to 8, 4, or 3 bits with absmax or zero-point scales, add an outlier, and see what rounding costs.

AdvancedExplained in Quantization: smaller numbers

Squeeze the weights

Store 128 numbers in fewer bits. Circles are the original values, bars the values the model would compute with.

Synthetic data: 128 weights drawn from a normal distribution with standard deviation 0.02, a typical size for a trained layer; the outlier sets one of them to 0.4. Positive activations are absolute values of normal samples. "Typical" means the root mean square of the values without the outlier.

Values
Stored as
Values per scale
Method
Typical error (of a typical value)12%
Largest error (of a typical value)20%
Rounded to exactly zero20 of 128
Bits per value, with scales4.125
Llama 2 7B stored this way3.5 GB

Challenges

0 of 4 complete

  1. With one outlier among the weights, find a 4-bit setting that rounds more than half of the other weights to exactly zero. (not yet)
  2. Keep the outlier and 4-bit absmax, and bring the typical error under 30% of a typical weight by changing only how many values share a scale. (not yet)
  3. On positive activations at 4 bits, find the change that cuts the error by more than half without changing the group size. (not yet)
  4. Compare the two 16-bit float formats on the weights and select the one that stores them more precisely. (not yet)

Try this

  • At 4 bits with one scale, switch between Weights and Weights + one outlier and count the weights that round to zero.
  • Keep the outlier and give every 16 values their own scale.
  • On Positive activations, compare Absmax and Zero-point.
  • Compare float16 and bfloat16 on the weights.
Absmax quantization with and without groupsPython with NumPy
import numpy as np

def absmax(x, bits, group):
    """Quantize to signed integers with one scale per group, then dequantize."""
    qmax = 2 ** (bits - 1) - 1                    # 127 for 8 bits, 7 for 4 bits
    g = x.reshape(-1, group)
    scale = np.abs(g).max(axis=1, keepdims=True) / qmax
    q = np.round(g / scale)                       # the stored integers
    return (q * scale).reshape(-1)                # what the model computes with

rng = np.random.default_rng(0)
w = rng.normal(0, 0.02, size=128)                 # synthetic weights
typical = np.sqrt(np.mean(w ** 2))
w_out = w.copy()
w_out[45] = 0.4                                   # one outlier, 20 standard deviations out

for name, x in [("weights", w), ("with outlier", w_out)]:
    for bits, group in [(8, 128), (4, 128), (4, 16)]:
        err = absmax(x, bits, group) - x
        zeros = np.sum(absmax(x, bits, group) == 0)
        print(f"{name:12} {bits}-bit, {group:3} per scale: typical error "
              f"{np.sqrt(np.mean(err ** 2)) / typical:6.1%}, {zeros:3} rounded to zero")

# Memory for Llama 2 7B (6,738,415,616 parameters).
n = 6_738_415_616
for label, bits in [("float32", 32), ("float16", 16), ("int8", 8),
                    ("4-bit + a 16-bit scale per 128", 4 + 16 / 128)]:
    print(f"{label:31} {n * bits / 8 / 1e9:5.1f} GB")
weights      8-bit, 128 per scale: typical error   0.5%,   1 rounded to zero
weights      4-bit, 128 per scale: typical error   9.4%,  17 rounded to zero
weights      4-bit,  16 per scale: typical error   7.6%,  12 rounded to zero
with outlier 8-bit, 128 per scale: typical error   4.8%,   6 rounded to zero
with outlier 4-bit, 128 per scale: typical error  85.6%, 113 rounded to zero
with outlier 4-bit,  16 per scale: typical error  32.9%,  22 rounded to zero
float32                          27.0 GB
float16                          13.5 GB
int8                              6.7 GB
4-bit + a 16-bit scale per 128    3.5 GB

Try "embedding", "softmax", "overfitting", or "backpropagation".