Compute and scaling laws

Spend a compute budget

Pick a model size and a number of training tokens. The curves show the fitted loss of every other way to spend the same compute.

Replication fit (Besiroglu et al. 2024)
Original fit as printed (Hoffmann et al. 2022)
Your model

Best use of this budget, replication fit: 73.0B parameters on 1.34T tokens, about 18 tokens per parameter
Best use of this budget, original fit: 32.5B parameters on 3.02T tokens, about 93 tokens per parameter

Parameters, N70.0B

Training tokens, D1.40T

Published training runs

Training compute, 6ND5.9 x 10²³ FLOP

Tokens per parameter20

Fitted loss, replication fit1.974

Challenges

0 of 3 complete

Using no more training compute than GPT-3, reach a fitted loss at least 0.01 lower than GPT-3's. (not yet)
Match Llama 3 8B's fitted loss or beat it, with less than a third of its training compute. (not yet)
With a budget at least 100 times smaller than Chinchilla's, put your model within 10% of the replication fit's best size. (not yet)

Loss values are a curve fitted to one family of training runs, in nats per token on that paper's data. They show the shape of the tradeoff, not the loss any real model would reach on your data.

Try this

Press each published training run and see where it sits relative to the minimum for its own budget.
Hover the curves to read off the loss of every other split of the same compute.
Notice how far apart the two fits put the best model size, and read the lesson for why.

6ND and the best split of GPT-3's budgetPython with NumPy

import numpy as np

# Replication fit (Besiroglu et al. 2024): L(N, D) = E + A / N^alpha + B / D^beta
E, A, B, alpha, beta = 1.8172, 482.01, 2085.43, 0.3478, 0.3658

def loss(N, D):
    return E + A / N**alpha + B / D**beta

N, D = 174.6e9, 300e9                    # GPT-3: parameters, training tokens
C = 6 * N * D                            # training compute, 6ND
print(f"GPT-3: {C:.3g} FLOP, fitted loss {loss(N, D):.3f}")

# Try many model sizes that spend exactly the same compute.
sizes = np.logspace(9, 12, 3001)
losses = loss(sizes, C / (6 * sizes))
best = sizes[np.argmin(losses)]
print(f"best split: {best:.3g} parameters on {C / (6 * best):.3g} tokens")
print(f"fitted loss {losses.min():.3f}, {C / (6 * best) / best:.0f} tokens per parameter")

GPT-3: 3.14e+23 FLOP, fitted loss 2.009
best split: 5.3e+10 parameters on 9.89e+11 tokens
fitted loss 1.992, 19 tokens per parameter