Models that hear and speak
Sound is a long list of numbers. Watch a spectrogram of your own voice, see why no window gives both sharp timing and sharp pitch, name everyday sounds with a model that reads spectrograms as pictures, transcribe speech with Whisper and find where it goes wrong, synthesize speech from text and send it back through the recognizer, and chain three models into a voice assistant. Every model runs on your device.
- Level
- Advanced
- Length
- 5 lessons, 1 hr 28 min
- Assumes
- The transformers track. The first lesson needs only the idea of turning things into numbers from AI from zero.
- Builds on
- Transformers and LLMs
- Entry point
- How computers hear, an intermediate lesson that needs only Turning things into numbers (AI from zero).
- 1How computers hearEntry pointSound reaches a computer as a long list of numbers. Cut it into short windows, find the frequencies in each, and you get the spectrogram that speech models read.16 min
- 2What sound is this?A transformer built for pictures can name sounds too, by reading the spectrogram as an image. Label bells, rain, a cat, or your own recording, and see why the scores add up to more than 100%.18 min
- 3Speech to textWhisper reads a log-mel spectrogram with a transformer encoder and writes the words with a decoder. Run it in your browser and find where it fails.18 min
- 4Text to speechTurn text into a voice in your browser. See how a model predicts the length of the speech, refines noise into a compact code in a few steps, and decodes it into sound.16 min
- 5A voice assistant taken apartChain speech recognition, a chat model, and text to speech, and you have a voice assistant. Run all three on your device, time each stage, follow a mistake through, and compare end-to-end speech models.20 min
Labs in this track
- SpectrogramSee sound as a picture. Change the window length to trade time detail for frequency detail, switch to mel bands, and watch your own voice live.
- What sound is this?Label bells, rain, a train, a cat, or your own recording with an Audio Spectrogram Transformer, mix two sounds, and see why its 527 scores add up to more than 100%.
- Speech to textRun Whisper in your browser on clean speech, five speakers, added noise, silence, or your own voice, and see the spectrogram it reads and when it heard each word.
- Text to speechTurn your own text into speech with Supertonic 2 in your browser, see it as a spectrogram with its pitch, and check what Whisper hears in it.
- A voice assistant taken apartAsk a spoken question and follow it through three models in a row, Whisper hearing it, SmolLM2 answering, Supertonic speaking, with each stage's output, time, and mistakes.