See text the way a model does
Type anything. Each colored chip is one token, the unit a language model reads and writes.
Loading the tokenizer vocabulary...
Tokens...
Characters58
Words11
Characters per tokenn/a
Challenges
0 of 3 complete
- Find a text that takes GPT-2 at least twice as many tokens as GPT-4o. (not yet)
- Type a number with seven or more digits that the two tokenizers cut in different places. (not yet)
- Find a word that is one token after a space but splits into pieces at the very start of the text. Type it twice to compare. (not yet)
Try this
- Compare the two tokenizers on Languages. The older one needs far more tokens for Japanese and Arabic.
- Type a long number. How does each tokenizer cut it?
- Paste a paragraph you wrote and read the characters-per-token readout.
Tokenizing with learned mergesPython with the standard library
# A byte-level BPE tokenizer, the kind GPT-2 and GPT-4o use, with a tiny made-up
# merge list. Real ones learned 50,000 to 200,000 merges from huge amounts of
# text and split text into words before merging. To run GPT-4o's real one:
# pip install tiktoken, then tiktoken.get_encoding("o200k_base").encode(text)
merges = [b"at", b" c", b" s", b"th", b"the", b" cat", b" sat", b"Th", b"The"]
rank = {m: i for i, m in enumerate(merges)} # earlier merges win
def encode(text):
parts = [bytes([b]) for b in text.encode("utf-8")] # start from single bytes
while len(parts) > 1:
# Find the neighboring pair whose merge was learned first.
best, i = min((rank.get(parts[i] + parts[i + 1], len(rank)), i)
for i in range(len(parts) - 1))
if best == len(rank):
break # no learned merge applies
parts[i:i + 2] = [parts[i] + parts[i + 1]]
return parts
for text in ["The cat sat", "猫が"]:
tokens = encode(text)
print(f"{text!r}: {len(text)} characters -> {len(tokens)} tokens")
print(" ", tokens)'The cat sat': 11 characters -> 3 tokens
[b'The', b' cat', b' sat']
'猫が': 2 characters -> 6 tokens
[b'\xe7', b'\x8c', b'\xab', b'\xe3', b'\x81', b'\x8c']