How computers hear

Say a word to a voice assistant and its microphone hands the computer nothing but a stream of numbers, tens of thousands every second. Nothing in those numbers says “vowel”, “word”, or even “sound starts here”. This lesson turns that stream into a picture in which speech becomes visible: the same picture a speech recognizer such as Whisper reads.

Sound is a list of numbers

Sound is a rapid wobble in air pressure. A microphone turns the wobble into a voltage, and a converter measures that voltage at evenly spaced instants. Each measurement is a sample, and the number of samples per second is the Sampling rateHow many times per second a recording measures the sound wave. CD audio uses 44,100 samples per second; Whisper works at 16,000. A rate can only capture frequencies below half of it.Open in glossary. CD audio uses 44,100 samples per second; Whisper resamples everything it hears to 16,000.

The simplest sound is a pure tone, a sine wave. Its frequency is how many times per second it repeats, measured in hertz (Hz). The A above middle C is 440 Hz. Real sounds, a voice or a violin, are many sine waves at once, and the mix of frequencies is what makes an “ee” differ from an “oo”.

There is a hard limit on what a sampling rate can hold. To pin down a wave you need more than two samples per cycle, so the highest frequency a recording can represent is half its sampling rate, called the Nyquist frequency. At 16,000 samples a second that is 8,000 Hz. A higher tone does not vanish: its samples are identical to those of a lower tone, so it shows up at the wrong frequency. That is AliasingWhen a frequency above half the sampling rate is recorded, its samples look exactly like those of a lower frequency, so it shows up in the wrong place. Recorders filter such frequencies out before sampling.Open in glossary, and it is why recorders filter out high frequencies before they sample.

From a waveform to a spectrogram

A waveform shows pressure over time, which says little about pitch. The Fourier transformA way to rewrite a signal as a sum of sine waves, giving how much of each frequency it contains. Computers use the discrete version on a block of samples, computed quickly with the fast Fourier transform (FFT).Open in glossary rewrites a block of samples as a sum of sine waves and tells you how much of each frequency the block contains. For a block of $N$ samples at sampling rate $f_s$ it measures the frequencies $0, \tfrac{f_s}{N}, \tfrac{2 f_s}{N}, \ldots, \tfrac{f_s}{2}$ , in steps of $f_s / N$ .

Speech changes every few tens of milliseconds, so one transform of a whole sentence would mix every sound together. Instead, cut the recording into short, overlapping windows. Fade each window in and out (a Hann window) so its hard edges do not add false frequencies, transform it, and place the result as one column of a picture. Columns side by side make a SpectrogramA picture of sound with time across, frequency up, and loudness as color, made by taking the frequencies of many short, overlapping windows of the recording and placing them side by side.Open in glossary: time across, frequency up, loudness as color. The method is called the short-time Fourier transform.

Try this

Choose Sweep, a tone gliding from 100 Hz to 7,900 Hz. Predict the shape first, then look: one straight rising line.
Choose Two notes, a semitone apart (440 and 466 Hz), and step the window from 5 ms up to 100 ms. Which windows show two lines?
Choose Two clicks, 12 ms apart, and step the window the other way.
Choose Tone and drag the frequency past 8,000 Hz. Before you do, guess where the line will go.
Choose Speech and tap the spectrogram during a vowel, then during the “s” of “first”. Compare the spectra in the lower chart.

What you saw: time against frequency

A window of $N$ samples gives frequency steps of $f_s / N$ , which is one divided by the window’s length in seconds. A 25 ms window has 40 Hz steps; a 100 ms window has 10 Hz steps. The two notes are 26 Hz apart, so only the 100 ms window separates them. The clicks behave the other way. Both fall inside any window longer than 12 ms, so a long window sees one event and only the 5 ms and 10 ms windows show two.

You cannot have both at once. A window that is short in time is blurry in frequency, and one that is sharp in frequency is long in time. This trade-off is a property of the transform itself, not of the code. Speech systems commonly settle on windows of about 25 ms taken every 10 ms, short enough to follow consonants and long enough to resolve the bands of energy that distinguish vowels. Whisper uses exactly that.

The tone above 8,000 Hz folded back down: at 16,000 samples a second, a 10,000 Hz tone shows up at 6,000 Hz. Nothing in the samples can tell the two apart.

In the speech sample, the vowels show a stack of evenly spaced horizontal lines. They are the harmonics of the voice: the vocal folds buzz at one pitch, and the buzz contains every whole multiple of it. The “s” is different, a spread of energy high up with no lines, because it is turbulent air rather than a buzz.

Pitch is not linear: the mel scale

To most listeners, the step from 200 Hz to 400 Hz sounds like a large jump; the step from 7,000 Hz to 7,200 Hz barely registers. Listening experiments in the late 1930s produced the Mel scaleA pitch scale built from listening experiments, roughly linear below 1,000 Hz and logarithmic above, so equal steps in mels sound like equal steps in pitch. Speech models group frequencies into bands spaced evenly in mels.Open in glossary, a pitch scale on which equal steps sound equally large. In the first (Stanley Smith Stevens, John Volkmann, and Edwin Newman, 1937), listeners set one tone to sound half as high as another; in a 1940 follow-up, Stevens and Volkmann had them divide a range of pitch into equal-sounding parts.

Several formulas approximate it. The one Whisper’s preprocessing uses (Malcolm Slaney’s version, the default in the librosa library) is linear below 1,000 Hz and logarithmic above:

m(f) = \begin{cases} \dfrac{3f}{200} & f < 1000 \text{ Hz} \\[6pt] 15 + \dfrac{27 \ln(f / 1000)}{\ln 6.4} & f \ge 1000 \text{ Hz} \end{cases}

So 1,000 Hz is 15 mels and 8,000 Hz is about 45. A mel filterbank places triangular bands evenly along this scale and adds up the power inside each. With 80 bands up to 8,000 Hz, the bands below 1,000 Hz have centers about 37 Hz apart, while those near 8,000 Hz are several hundred hertz wide. Switch the lab to 80 mel bands to see it: the sweep climbs quickly through the low bands and crawls through the high ones, and the two notes always merge.

Finally, take the logarithm of each band’s power. Loudness is heard roughly logarithmically, and the power in a spectrogram of speech spans many powers of ten, so a log scale keeps both the quiet and the loud parts visible. The result, a log-mel spectrogram, is what most speech models read.

Why the window shape and the FFT matterOptional

Cutting a window out of a longer sound with hard edges is like multiplying it by a rectangle, and the rectangle’s own spectrum smears every tone across many frequencies (spectral leakage). With hard edges, the strongest false peaks sit only 13 dB below the true one. A Hann window tapers to zero at both ends, which pushes them to about 31 dB below and makes them fall off quickly, at the cost of a main peak twice as wide.

The fast Fourier transform, published by James Cooley and John Tukey in 1965, computes the $N$ frequencies of a block in about $N \log_2 N$ steps instead of $N^2$ . For a 400-sample window that is roughly 3,500 steps instead of 160,000, which is why a phone can draw a live spectrogram. Whisper’s 400-point FFT is not a power of two; this site computes it with Bluestein’s method, which rewrites it as a convolution done with power-of-two FFTs.

Key ideas

A recording is a list of samples. At a sampling rate of $f_s$ it can only hold frequencies below $f_s / 2$ ; higher ones alias to the wrong frequency.
The Fourier transform measures how much of each frequency a block of samples contains. A spectrogram does this for short, overlapping windows and lines the results up in time.
Short windows show when things happen; long windows show which frequencies are present. No window does both, so speech systems use about 25 ms.
The mel scale spaces frequency bands the way people hear pitch. A log-mel spectrogram (80 bands, 25 ms windows every 10 ms) is exactly what Whisper reads.