Speech to text

Video captions, transcribed voice messages, and meeting notes all start with the same step: speech recognitionSpeech recognitionTurning recorded speech into text. Modern systems such as Whisper are neural networks that read a spectrogram of the audio and write the transcript one token at a time.Open in glossary, turning sound into text. In September 2022 OpenAI released Whisper, a family of open models trained on 680,000 hours of audio paired with transcripts collected from the internet. Its smaller versions are light enough to run in a browser. This lesson runs one on your device and then goes looking for its mistakes.

From a spectrogram to words

Whisper is an encoder-decoderEncoder-decoderA model in two parts. The encoder reads the whole input (a sentence, or a spectrogram) into a set of vectors; the decoder writes the output one token at a time, looking back at those vectors through cross-attention.Open in glossary transformer, the same design as the original transformer for translation. Its input is not the waveform but the log-mel spectrogram from How computers hear: 80 mel bands, one frame every 10 ms, for a 30-second window.

The encoder starts with two small convolutions; the second takes every other step, so 3,000 frames become 1,500 positions, one per 20 ms. Position encodings are added, and then transformer blocks let every moment of the audio attend to every other moment. Out comes one vector per 20 ms of sound.

The decoder is a language model. It writes the transcript one tokenTokenThe unit of text a language model reads and writes. A token can be a whole word, part of a word, a single character, or punctuation, often including a leading space.Taught in TokensOpen in glossary at a time, using the same tokenizer as GPT-2 for the English-only models, and each new token depends on the tokens before it, as in the transformers track. What makes it a recognizer is cross-attention: in every decoder block, the token being written can look at all 1,500 encoder vectors and pull in whatever part of the sound it needs. The model in this lab, Whisper base.en, has 74 million parameters, with 6 blocks in the encoder and 6 in the decoder.

Tokens that set the task

The decoder’s first tokens are instructions rather than words. Every transcript starts with <|startoftranscript|>. The multilingual models then predict a language token (one per language, 99 in all) and a task, <|transcribe|> or <|translate|> (into English); English-only models such as base.en skip those two. A <|notimestamps|> token asks for plain text. Without it, the decoder writes time tokens between pieces of text: a start time before each segment and an end time after it, rounded to 20 ms, which is the timing you see with Segments in the lab.

Word-level timings, the default in the lab, are not something Whisper was trained to write. They are worked out afterwards: a few cross-attention heads tend to look at the part of the audio being transcribed, so smoothing their attention weights and aligning them with the tokens (dynamic time warping) gives an estimated start and end for each word. Both OpenAI’s code and the Transformers.js library used here do this. The times are estimates, as you will see.

Whisper was trained on 30-second windows and cannot read more at once. Shorter audio is padded with silence; choose Whisper’s 30 seconds in the lab to see the padding. For longer recordings, OpenAI’s own code moves the window forward according to the timestamps it predicted. The library used here instead cuts long audio into 30-second windows that start every 20 seconds, so neighbors overlap by 10 seconds, and stitches the text together where they overlap. Record more than 30 seconds of your own voice to see the windows.

import re
import numpy as np

def words(text):
    return re.findall(r"[a-z0-9']+", text.lower())

def word_error_rate(reference, heard):
    r, h = words(reference), words(heard)
    d = np.zeros((len(r) + 1, len(h) + 1), dtype=int)
    d[:, 0] = np.arange(len(r) + 1)          # drop every reference word
    d[0, :] = np.arange(len(h) + 1)          # insert every heard word
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            swap = d[i - 1, j - 1] + (r[i - 1] != h[j - 1])   # match or substitute
            d[i, j] = min(swap, d[i - 1, j] + 1, d[i, j - 1] + 1)
    return d[-1, -1] / len(r)                # fewest edits per reference word

said = "He was a head shorter than his companion, of almost delicate physique."
print(f"{word_error_rate(said, 'He was head shorter than his companion of almost delicate physics.'):.3f}")
print(f"{word_error_rate(said, 'Here is a headshot of Dennis from Banyo, a far more delicate figure.'):.3f}")

# Background noise at a chosen signal-to-noise ratio (SNR), as the lab adds it.
rng = np.random.default_rng(0)
speech = 0.3 * np.sin(2 * np.pi * 200 * np.arange(16000) / 16000)   # stand-in for a voice
noise = rng.standard_normal(16000)
rms = lambda x: np.sqrt(np.mean(x ** 2))
for snr_db in (20, 0, -10):
    scaled = noise * rms(speech) / rms(noise) / 10 ** (snr_db / 20)
    print(f"SNR {snr_db:3d} dB: noise RMS {rms(scaled):.4f}, speech RMS {rms(speech):.4f}")

0.167
0.917
SNR  20 dB: noise RMS 0.0212, speech RMS 0.2121
SNR   0 dB: noise RMS 0.2121, speech RMS 0.2121
SNR -10 dB: noise RMS 0.6708, speech RMS 0.2121

Scoring a transcript

The standard score is the word error rateWord error rateThe standard score for speech recognition. Count the fewest word substitutions, deletions, and insertions that turn the transcript into the correct text, and divide by the number of correct words. Lower is better.Open in glossary. Line the transcript up with the correct text using the fewest edits, count substitutions $S$, deletions $D$, and insertions $I$, and divide by the number of reference words $N$:

It can pass 100% when the model invents many words. The score depends on how text is normalized first: is “20th” the same as “twentieth”? The lab ignores case and punctuation and expands contractions such as “I’m”, but does not match numbers. Whisper’s paper uses a far more thorough normalizer, so the lab’s numbers cannot be compared with published ones. In the paper, with greedy decoding on the LibriSpeech test-clean benchmark, base.en scored 4.2% and the larger medium.en 3.1%.

Where it fails

Noise. On the clean reading the lab’s model makes no errors. With white noise added, it got at most one word wrong down to 5 dB (speech about three times more powerful than the noise), about one word in five at 0 dB, where the noise is as loud as the speech, and at -5 and -10 dB it wrote fluent sentences that were never said, one beginning “This is the only chance to”. Whisper was unusually robust for its time: in its paper’s noise test, all 14 comparison models, trained largely on the LibriSpeech dataset, did worse than Whisper once the signal-to-noise ratio of added pub noise (background chatter) fell below 10 dB. But every model, Whisper included, degraded as the noise grew.

Silence. Given 6 seconds of exact zeros, the model wrote “you”; given a faint hiss, “and”. Whisper’s authors list “complete hallucination where the model will output a transcript entirely unrelated to the actual audio” among its failure modes, and its model card suggests why: the model combines “trying to predict the next word in audio with trying to transcribe the audio itself”. The word is a hallucinationHallucinationWhen a language model states something false or invented, such as a fake quote or citation, in the same fluent, confident style as true statements.Taught in What AI gets wrongOpen in glossary in the same sense as a chatbot’s. A 2024 study of OpenAI’s Whisper service by Allison Koenecke and colleagues found that roughly 1% of transcriptions contained whole invented phrases or sentences, more often for speakers with long pauses. OpenAI’s own transcription code checks the probability of a no-speech token together with the average confidence of the text to skip silent windows; the plain pipeline in this lab has no such check.

Accents. On the lab’s sentence, the US, Canadian, and Scottish speakers were transcribed perfectly and the Indian English speaker’s had two errors. One sentence is an anecdote, so we ran 20 sentences from each speaker through the same model: about 2 to 4 errors per 100 words for the two US speakers and the Canadian, 5 for the Scottish speaker, and 15 for the Indian English speaker. Each accent here is a single person reading in a quiet room, so speaker and accent cannot be separated. Larger studies point the same way: Whisper’s model card warns of “disparate performance on different accents and dialects”, and a 2020 study of five commercial recognizers found an average word error rate of 35% for Black speakers against 19% for white speakers in recorded interviews.