How low can the rank go?

LoRA learns a change to a weight matrix as B times A. Choose the rank and see how much of a target change it can capture.

Target change

B A, rank 2 (best)

What it misses

negative
positive (paler is closer to zero)

B is 32 x 2 and A is 2 x 32: 64 + 64 = 128 numbers, against 1,024 for the full 32 x 32 change.

Singular values of the target, largest first kept by rank 2 dropped

Target update

Low-rank updateSpread outRandom noise

Rank r2

Numbers LoRA trains here128 of 1,024

Missed at best13.3%

Best possible at this rank86.7% captured

Same rank on a 4,096 x 4,096 matrix16,384 numbers, 0.10% of the full matrix

Challenges

0 of 3 complete

On the low-rank update, find the smallest rank that captures at least 99% of it. (not yet)
Find the smallest rank that captures 90% of the random noise. (not yet)
Train B and A from B = 0 until they miss at most one percentage point more of the change than the best that rank allows. (not yet)

Try this

On Low-rank update, raise the rank one step at a time and watch where the error stops falling.
Compare Spread out and Random noise at the same rank.
Press Train B and A and watch the share missed fall from 100% to the best possible line.

Ranks, parameter counts, and a zero startPython with NumPy

import numpy as np

rng = np.random.default_rng(0)
d = 32

# A target change built like the demo's: four strong directions plus faint noise.
U, _ = np.linalg.qr(rng.normal(size=(d, d)))
V, _ = np.linalg.qr(rng.normal(size=(d, d)))
s = np.full(d, 0.15)
s[:4] = [8, 5, 3, 2]
delta = U @ np.diag(s) @ V.T

sv = np.linalg.svd(delta, compute_uv=False)
for r in (1, 2, 4, 8):
    kept = (sv[:r] ** 2).sum() / (sv ** 2).sum()  # the best rank-r share
    print(f"rank {r}: B and A hold {r * (d + d):4d} numbers (full: {d * d}), "
          f"best possible captures {kept:.1%}")

# LoRA starts exactly at the base model, because B starts at zero.
W = rng.normal(size=(d, d))
r, alpha = 4, 8
A = rng.normal(0, 1 / np.sqrt(d), size=(r, d))
B = np.zeros((d, r))
x = rng.normal(size=d)
print("starts at the base model:", np.allclose(W @ x + (alpha / r) * B @ (A @ x), W @ x))

# Real shapes: GPT-3 175B adapts W_q and W_v (12,288 x 12,288) in 96 layers.
gpt3 = 96 * 2 * 4 * (12288 + 12288)
print(f"GPT-3, r = 4 on W_q and W_v: {gpt3:,} numbers, {2 * gpt3 / 1e6:.1f} MB at 16 bits")

rank 1: B and A hold   64 numbers (full: 1024), best possible captures 62.4%
rank 2: B and A hold  128 numbers (full: 1024), best possible captures 86.7%
rank 4: B and A hold  256 numbers (full: 1024), best possible captures 99.4%
rank 8: B and A hold  512 numbers (full: 1024), best possible captures 99.5%
starts at the base model: True
GPT-3, r = 4 on W_q and W_v: 18,874,368 numbers, 37.7 MB at 16 bits