Will it fit?
Pick a model and how many bits each of its numbers gets. See the download, the memory it needs, and where it can run.
Phone
Fits8 GB memory, 60 GB/s
Needs 4% of its memory, leaving room for the system and your apps.
Writes at most 223 tokens a second: faster than you read.
Laptop
Fits16 GB memory, 100 GB/s
Needs 2% of its memory, leaving room for the system and your apps.
Writes at most 372 tokens a second: faster than you read.
Data center
1 GPU80 GB and 3,350 GB/s per GPU, 8 per server
One GPU holds it.
Writes at most over 1,000 tokens a second: faster than you read.
Challenges
0 of 3 complete
- Get Llama 2 7B to fit comfortably on the phone, using at most half of its memory. (not yet)
- With 16-bit numbers, find the biggest model the laptop could still run faster than you read (at least 2 billion parameters). (not yet)
- Fit Llama 3.1 405B into a single 8-GPU server. (not yet)
Try this
- Start with SmolLM2-135M at 16-bit and predict its download before you look: 135 million parameters times 2 bytes. Then compare with what this site really downloads.
- Choose Llama 2 7B and step from 32-bit down to 4-bit. Watch the phone and the laptop change.
- At 16-bit, drag Parameters up slowly and watch the laptop’s best-case speed fall.
- Choose Llama 3.1 405B and count the GPUs at each precision.
The same formula against the 15 models this site runs in your browser: the formula column is parameters times bits each, divided by 8 to get bytes. The real downloads are a little larger: they include small files such as the tokenizer, 8-bit versions keep a few small parts at 32-bit, and 4-bit files also store a scale for each small block of weights. Janus-Pro-1B stores its parts at different precisions, so its formula adds them up part by part, and the calculator above leaves it out.
| Model | Parameters (millions) | Bits each | Formula | Real download |
|---|---|---|---|---|
| DeiT-Tiny Names what is in a photo | 5.9 | 32 | 23.6 MB | 24 MB |
| YOLOS-Tiny Finds and boxes objects in a photo | 6.5 | 32 | 26.0 MB | 27 MB |
| all-MiniLM-L6-v2 Turns sentences into numbers | 22.7 | 8 | 22.7 MB | 24 MB |
| Depth Anything V2 Small Estimates how far away things are | 24.8 | 16 | 49.6 MB | 50 MB |
| ViTPose+ small Finds a body's keypoints | 33.1 | 16 | 66.3 MB | 67 MB |
| SAM 2.1 tiny Outlines the object you click | 39.0 | 16 | 77.9 MB | 78 MB |
| BiRefNet lite Cuts the subject out of a picture | 44.3 | 16 | 88.7 MB | 99 MB |
| Supertonic 2 Reads text aloud | 66.0 | 32 | 264 MB | 263 MB |
| Whisper base.en Turns speech into text | 72.6 | 8 | 72.6 MB | 80 MB |
| Audio Spectrogram Transformer Names the sounds in a clip | 86.6 | 8 | 86.6 MB | 91 MB |
| SmolLM2-135M Writes text | 134.5 | 16 | 269 MB | 272 MB |
| SmolLM2-135M-Instruct Chats | 134.5 | 16 | 269 MB | 272 MB |
| CLIP ViT-B/32 Matches photos and words | 151.3 | 16 | 303 MB | 306 MB |
| OWL-ViT B/32 Finds anything you name in a photo | 153.2 | 16 | 306 MB | 311 MB |
| Janus-Pro-1B Makes pictures from text | 1,530.7 | 4 to 32 | 1.02 GB | 1105 MB |
| Janus-Pro-1B, reading pictures Answers questions about a photo, sharing two files with the row above | 2,172.4 | 4 to 16 | 1.51 GB | 1628 MB |
The calculator in a few linesPython with the standard library
# Download, memory, and best-case writing speed on a laptop, from a parameter count.
GB = 1e9 # file sizes: a billion bytes
LAPTOP_MEMORY = 16 * 2**30 # a "16 GB" laptop holds 16 x 2^30 bytes
LAPTOP_SPEED = 100 * GB # bytes its memory moves per second (Apple M2)
def on_laptop(params, bits):
download = params * bits / 8 # bytes: 2 per parameter at 16-bit, half at 4-bit
memory = 1.2 * download # plus about 20% working space
share = memory / LAPTOP_MEMORY
if share > 1:
return download, memory, "too big"
fit = "fits" if share <= 0.5 else "tight"
speed = LAPTOP_SPEED / download # each token reads every number once
return download, memory, f"{fit}, at most {speed:.0f} tokens a second"
models = [("SmolLM2-135M", 134_515_008), ("Llama 2 7B", 6_738_415_616),
("Llama 3 70B", 70_553_706_496)]
for name, params in models:
for bits in (16, 4):
download, memory, verdict = on_laptop(params, bits)
print(f"{name:12} {bits:2}-bit: download {download / GB:6.2f} GB, "
f"memory {memory / GB:6.2f} GB: {verdict}")SmolLM2-135M 16-bit: download 0.27 GB, memory 0.32 GB: fits, at most 372 tokens a second
SmolLM2-135M 4-bit: download 0.07 GB, memory 0.08 GB: fits, at most 1487 tokens a second
Llama 2 7B 16-bit: download 13.48 GB, memory 16.17 GB: tight, at most 7 tokens a second
Llama 2 7B 4-bit: download 3.37 GB, memory 4.04 GB: fits, at most 30 tokens a second
Llama 3 70B 16-bit: download 141.11 GB, memory 169.33 GB: too big
Llama 3 70B 4-bit: download 35.28 GB, memory 42.33 GB: too big