A voice assistant taken apart

You have met each part already. Speech to text turned speech into words with Whisper, How a chat assistant works answered messages with an instruction-tuned SmolLM2, and Text to speech read text aloud with Supertonic. Put them in a row and you have a voice assistant. This lesson does exactly that, on your device, and times every step.

Three models in a row

A cascaded assistant is a pipeline:

1. Hear. Whisper base.en turns the recorded question into text.
2. Write. SmolLM2-135M-Instruct reads that text in its chat template, after a short system prompt (“You are a helpful voice assistant. Reply in one short spoken sentence.”), and writes a reply.
3. Speak. Supertonic 2 reads the reply aloud, after the lab strips Markdown such as **bold** that the model sometimes writes.

A fourth step is only for measuring: Whisper listens to the spoken answer, so you can see what the speech stage lost.

Each stage starts when the one before it has finished, and each hands on nothing but text. Those two facts explain most of what you are about to see.

What you saw

The waits add up. On the laptop the recordings were made on, with every model on the CPU, the silence between the end of a question and the start of its answer was about 2.5 to 5 seconds. Hearing took about 1.5 to 3 seconds, writing a short answer a fraction of a second, and speaking from under half a second for a short answer to about 2 seconds for a long one. The small language model was the fastest stage. In a cascade it does not matter much which stage is slow: all of them happen one after another, and you wait for their sum.

Streaming can shave some of it. A real assistant can start speaking the first sentence while the model is still writing the next. For the two-sentence answer to the time question, that saves only a few tenths of a second here, because most of the wait came before any answer existed.

Mistakes travel. In the noise, Whisper heard “Who wrote Pride and Prejudice?” as “He wrote, five and prejudice.” The language model never hears sound, only that text, and it simply repeated it. Clean, the same question got “The novel by Jane Austen.” A cascade is only as good as the text each stage hands on, and no stage knows how sure the one before it was.

Each stage keeps its own weaknesses. Whisper heard “What time is it right now?” as “What time right now?” even without noise; the question was spoken by Supertonic, so the synthetic voice may share the blame. The language model has no clock, and it answered with a two-sentence apology despite being asked for one sentence. Asked for dinner ideas, it started a numbered recipe list in Markdown and was cut off at 48 tokens. The speech stage read “8” as “eight” correctly, and Whisper, listening back, heard “A spider, eight legs.”, dropping “has”.

End-to-end speech models

OpenAI’s description of the Voice Mode that ChatGPT used before May 2024 matches this lab: “a pipeline of three separate models: one simple model transcribes audio to text, GPT-3.5 or GPT-4 takes in text and outputs text, and a third simple model converts that text back to audio”, with average latencies of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4). The text in the middle meant the language model “can’t directly observe tone, multiple speakers, or background noises, and it can’t output laughter, singing, or express emotion” (OpenAI, May 2024).

The alternative is an end-to-end speech modelEnd-to-end speech modelA model that takes speech in and gives speech out with one network, instead of a cascade of speech recognition, a text model, and text to speech. It can hear and produce tone, laughter, or overlapping talk that text would drop.Open in glossary: one network that takes sound in and produces sound out, usually by turning audio into tokens much as Janus-Pro turns pictures into codes.

• GPT-4o (OpenAI, 2024) is “trained end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.” Its system card reports that it “can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds” (OpenAI, 2024).
• Moshi (Kyutai, 2024) generates its own speech as audio tokens from a neural codec at 12.5 frames a second, in a stream parallel to the user’s, so it can listen while it talks and has no fixed turns. Its paper reports a theoretical latencyLatencyThe wait between a request and the response, such as the silence between the end of your question and the start of a voice assistant's answer. In a chain of models the waits add up.Open in glossary of 160 ms and 200 ms in practice, against an average of 230 ms between turns in human conversation (Défossez et al., 2024).
• Qwen2.5-Omni (Alibaba, 2025) splits one model into a “Thinker”, a language model writing text, and a “Talker” that turns the Thinker’s internal states, not just its text, into audio tokens as it goes, trained together end to end (Xu et al., 2025).

A cascade has its own advantages: each part can be swapped, tested, and improved on its own, and the text in the middle can be read, logged, and checked. This lab works only because its three parts are separate models small enough for a laptop. Where multimodal models are heading looks at single models that read and write many kinds of data.