Speech to text

The picture is the log-mel spectrogram Whisper reads, computed in your browser with the same steps as OpenAI’s code. Whisper base.en runs on your device after you agree to the download; your recordings never leave the page.

Try this

Press a word in the transcript to hear that stretch of audio and see its box on the spectrogram.
Lower Added noise step by step and note where the first error appears.
Go through Five speakers: same sentence, five people.
Choose No speech, then view Whisper’s 30 seconds to see where it placed the words it invented.
Record more than 30 seconds of your own voice to see how the audio is split into windows.

Word error rate and noise in NumPyPython with NumPy

import re
import numpy as np

def words(text):
    return re.findall(r"[a-z0-9']+", text.lower())

def word_error_rate(reference, heard):
    r, h = words(reference), words(heard)
    d = np.zeros((len(r) + 1, len(h) + 1), dtype=int)
    d[:, 0] = np.arange(len(r) + 1)          # drop every reference word
    d[0, :] = np.arange(len(h) + 1)          # insert every heard word
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            swap = d[i - 1, j - 1] + (r[i - 1] != h[j - 1])   # match or substitute
            d[i, j] = min(swap, d[i - 1, j] + 1, d[i, j - 1] + 1)
    return d[-1, -1] / len(r)                # fewest edits per reference word

said = "He was a head shorter than his companion, of almost delicate physique."
print(f"{word_error_rate(said, 'He was head shorter than his companion of almost delicate physics.'):.3f}")
print(f"{word_error_rate(said, 'Here is a headshot of Dennis from Banyo, a far more delicate figure.'):.3f}")

# Background noise at a chosen signal-to-noise ratio (SNR), as the lab adds it.
rng = np.random.default_rng(0)
speech = 0.3 * np.sin(2 * np.pi * 200 * np.arange(16000) / 16000)   # stand-in for a voice
noise = rng.standard_normal(16000)
rms = lambda x: np.sqrt(np.mean(x ** 2))
for snr_db in (20, 0, -10):
    scaled = noise * rms(speech) / rms(noise) / 10 ** (snr_db / 20)
    print(f"SNR {snr_db:3d} dB: noise RMS {rms(scaled):.4f}, speech RMS {rms(speech):.4f}")

0.167
0.917
SNR  20 dB: noise RMS 0.0212, speech RMS 0.2121
SNR   0 dB: noise RMS 0.2121, speech RMS 0.2121
SNR -10 dB: noise RMS 0.6708, speech RMS 0.2121