Write with a word-counting model
This model learned by counting which words follow which in ten short fables. The bars show what it thinks comes next. Pick one, or let it choose at random.
The
What comes next?
and 34 other words with 60.7% between them
Looking at"the"
Times that context appears in the fables107
Challenges
0 of 3 complete
- With the text at just "The", lower the temperature until "lion" gets over 40% of the chance. At temperature 1 it has 6.5%. (not yet)
- Write until the last 8 words form a run that appears nowhere in the fables. (not yet)
- Switch to 2 words of context and find a spot where the model has only one choice for the next word. (not yet)
Try this
- Click the bars to write a sentence yourself, one word at a time.
- Set temperature to its lowest and press Write 12 words a few times. Then try the highest.
- Switch to 2 words of context and write again. Does it sound more like the fables?
Optional: the same idea in codePython with NumPy
import re
import numpy as np
# One of the ten fables the demo learned from.
text = """The fox and the grapes. One hot day a hungry fox saw some grapes hanging
from a high vine. The grapes were ripe and sweet. The fox jumped for the grapes,
but he could not reach them. He jumped again and again, but the grapes were too
high. At last the fox walked away and said, the grapes are sour, and I did not
want them anyway. It is easy to hate what you cannot have."""
# Split into lowercase words and punctuation marks, as the demo does.
tokens = re.findall(r"[a-z]+(?:'[a-z]+)?|[.,!?;:]", text.lower())
def next_word_chances(word, temperature=1.0):
# "Training" is just counting what came right after this word.
followers = [b for a, b in zip(tokens, tokens[1:]) if a == word]
options, counts = np.unique(followers, return_counts=True)
# Temperature reshapes the counts: below 1 sharpens, above 1 flattens.
weights = counts ** (1 / temperature)
return options, weights / weights.sum()
for t in (1.0, 0.5, 2.0):
options, chances = next_word_chances("the", t)
shown = ", ".join(f"{w} {p:.0%}" for w, p in zip(options, chances))
print(f"After 'the' at temperature {t}: {shown}")
# Writing is a loop: get the chances, pick a word at random by its
# chance, add it, and repeat.
rng = np.random.default_rng(3)
story = ["the"]
for _ in range(12):
options, chances = next_word_chances(story[-1])
story.append(str(rng.choice(options, p=chances)))
print(" ".join(story))After 'the' at temperature 1.0: fox 38%, grapes 62%
After 'the' at temperature 0.5: fox 26%, grapes 74%
After 'the' at temperature 2.0: fox 44%, grapes 56%
the fox and the grapes , but he could not reach them anyway