A cascaded voice assistant is three models in a row: speech recognition turns your question into text, a language model writes an answer, and text to speech reads it aloud. Here they are this site’s Whisper base.en, SmolLM2-135M-Instruct, and Supertonic 2. The lesson explains what the cascade costs and how end-to-end speech models differ.
A voice assistant taken apart
Three separate models in a row: one hears your question, one writes an answer, one speaks it.
Running your own question needs all three downloads below; until then each stage shows what the real models did with the bundled questions.
1. HearWhisper base.en
Loading the recorded results
2. WriteSmolLM2-135M-Instruct
Loading the recorded results
3. SpeakSupertonic 2
Loading the recorded results
4. Check Whisper base.en again
Hears in the spoken answer
Silence before the answern/a
Whisper's word error raten/a
Answer lengthn/a
Round trip error raten/a
Challenges
0 of 3 complete
- Add noise and find the question whose answer changes because Whisper misheard it. (not yet)
- Find an answer that comes back different when Whisper listens to the spoken reply. (not yet)
- Ask something no amount of good hearing or speaking can answer. (not yet)
Try this
- Ask who wrote Pride and Prejudice, then add noise. Follow the mistake from stage to stage.
- Ask how many legs a spider has and compare what stage 3 was given with what stage 4 heard.
- Ask what time it is. Which stage is at fault?
- Look at the time line for each question. Which stage is slowest, and when?
- With all three models downloaded, ask your own question out loud. Nothing leaves your device.
Word error rate and the waits of a cascadePython with NumPy
import numpy as np
def word_error_rate(reference, heard):
"""Edits (substitutions, deletions, insertions) per reference word."""
clean = lambda s: "".join(c if c.isalnum() or c == " " else " " for c in s.lower()).split()
r, h = clean(reference), clean(heard)
d = np.zeros((len(r) + 1, len(h) + 1), dtype=int)
d[:, 0], d[0, :] = range(len(r) + 1), range(len(h) + 1)
for i in range(1, len(r) + 1):
for j in range(1, len(h) + 1):
same = r[i - 1] == h[j - 1]
d[i, j] = min(d[i - 1, j] + 1, d[i, j - 1] + 1, d[i - 1, j - 1] + (not same))
return d[-1, -1] / len(r)
# What Whisper heard in the lab's recording, clean and with loud noise.
asked = "Who wrote Pride and Prejudice?"
for heard in ["Who wrote Pride and Prejudice?", "He wrote, five and prejudice."]:
print(f"heard {heard!r}: word error rate {word_error_rate(asked, heard):.0%}")
# A cascade waits for each stage in turn, so the waits add up.
stages = {"hear": 1.8, "write": 0.2, "speak": 0.5} # seconds, roughly the lab's
print("silence before the answer:", round(sum(stages.values()), 1), "s")
# Starting to speak after the first sentence instead of the whole answer
# saves only what is left to write and to speak.
write_first, speak_first = 0.5, 1.6 # seconds for a two-sentence apology
write_all, speak_all = 0.7, 1.7
print("wait, whole answer first:", round(1.9 + write_all + speak_all, 1), "s")
print("wait, first sentence first:", round(1.9 + write_first + speak_first, 1), "s")heard 'Who wrote Pride and Prejudice?': word error rate 0%
heard 'He wrote, five and prejudice.': word error rate 40%
silence before the answer: 2.5 s
wait, whole answer first: 4.3 s
wait, first sentence first: 4.0 s