Build a tokenizer with byte-pair encoding
Start from single characters. Each step merges the most frequent neighboring pair into a new token.
Every character is its own token. The vocabulary is the 11 distinct characters in the text.
low␣lower␣lowest␣slow␣slower␣slowest␣new␣newer␣newest␣wide␣wider␣widest␣old␣older␣oldest
Vocabulary size11
Tokens in the text88
Characters per token1.00
Challenges
0 of 3 complete
- Find the fewest merges that cut the token count to half the number of characters. (not yet)
- Choose Your text and make a word of five or more letters into a single token. (not yet)
- Keep merging until the vocabulary has more entries than the text has tokens. (not yet)
Try this
- Predict the first three merges for Word family before pressing Next merge.
- Press Run all and read the chart: early merges save the most tokens.
- Type your own text and repeat one word several times. It becomes a single token.
Byte-pair encoding in a few lines of codePython with the standard library
import re
from collections import Counter
text = "low lower lowest slow slower slowest"
# Split into words, keeping each word's leading space, and count each word.
words = Counter(re.findall(r"\s*\S+", text))
# Every word starts as a list of single characters.
pieces = {w: list(w) for w in words}
def count_tokens():
return sum(len(pieces[w]) * n for w, n in words.items())
print(f"start: {count_tokens()} tokens")
for step in range(1, 6):
# Count every neighboring pair inside every word.
pairs = Counter()
for w, n in words.items():
for pair in zip(pieces[w], pieces[w][1:]):
pairs[pair] += n
# Most frequent pair; ties go to the pair that sorts first.
best = min(pairs, key=lambda p: (-pairs[p], p))
if pairs[best] < 2:
break
# Replace that pair with one new symbol everywhere.
for w in pieces:
out, i = [], 0
while i < len(pieces[w]):
if pieces[w][i:i + 2] == list(best):
out.append(best[0] + best[1])
i += 2
else:
out.append(pieces[w][i])
i += 1
pieces[w] = out
new = best[0] + best[1]
print(f"merge {step}: {new!r}, seen {pairs[best]} times -> {count_tokens()} tokens")start: 36 tokens
merge 1: 'lo', seen 6 times -> 30 tokens
merge 2: 'low', seen 6 times -> 24 tokens
merge 3: 'lowe', seen 4 times -> 20 tokens
merge 4: ' s', seen 3 times -> 17 tokens
merge 5: ' lowe', seen 2 times -> 15 tokens