From scores to a chosen token
The bars show the probability of each candidate after temperature, top-k, and top-p. Then one token is drawn at random.
The capital of France is?
| Token | Probability | p | Draws |
|---|---|---|---|
| ␣Paris | 77.9% | ||
| ␣a | 7.1% | ||
| ␣the | 5.2% | ||
| ␣located | 2.6% | ||
| ␣known | 1.7% | ||
| ␣home | 1.4% | ||
| ␣one | 1.2% | ||
| ␣not | 0.9% | ||
| ␣also | 0.7% | ||
| ␣called | 0.6% | ||
| ␣beautiful | 0.5% | ||
| ␣Lyon | 0.3% |
after your settings at temperature 1, before any cuts
Tokens still in play12 of 12
Uncertainty (entropy)1.39 bits
Most likely␣Paris 77.9%
Challenges
0 of 4 complete
- On Fact at temperature 1 with top-k off, find the highest top-p that still keeps exactly one token. (not yet)
- Set top-p to 0.8 with top-k off and temperature 1. Find the prompt that keeps the most tokens in play. (not yet)
- On Fact with top-k and top-p off, raise the temperature until Paris holds less than half the probability. (not yet)
- Push the uncertainty readout above 3.5 bits. (not yet)
Try this
- Compare the Fact and Story prompts at temperature 1. Which has the higher entropy, and why does that match how many good continuations each prompt has?
- Set temperature to 0.3 and draw 100 times. Then try 1.8. Watch how the counts follow the bars.
- With top-p at 0.8, switch between prompts and count how many tokens survive each time.
Temperature and top-p in a few linesPython with NumPy
import numpy as np
# The demo's scores for "The capital of France is" (tokens without their leading spaces).
tokens = ["Paris", "a", "the", "located", "known", "home",
"one", "not", "also", "called", "beautiful", "Lyon"]
logits = np.array([6.0, 3.6, 3.3, 2.6, 2.2, 2.0, 1.8, 1.5, 1.3, 1.1, 0.9, 0.4])
def softmax(z, temperature=1.0):
z = z / temperature
e = np.exp(z - z.max()) # subtracting the max avoids overflow
return e / e.sum()
def top_p(probs, p):
order = np.argsort(-probs, kind="stable") # most likely first
total = np.cumsum(probs[order])
n = np.argmax(total >= p) + 1 # smallest set whose total reaches p
kept = np.zeros_like(probs)
kept[order[:n]] = probs[order[:n]]
return kept / kept.sum() # renormalize
for t in (0.5, 1.0, 1.6):
print(f"temperature {t}: P(Paris) = {softmax(logits, t)[0]:.3f}")
probs = softmax(logits)
for p in (0.75, 0.8, 0.9):
kept = top_p(probs, p)
names = [tokens[i] for i in np.flatnonzero(kept)]
print(f"top-p {p}: keeps {names}")temperature 0.5: P(Paris) = 0.985
temperature 1.0: P(Paris) = 0.779
temperature 1.6: P(Paris) = 0.498
top-p 0.75: keeps ['Paris']
top-p 0.8: keeps ['Paris', 'a']
top-p 0.9: keeps ['Paris', 'a', 'the']