Everything here runs at 16,000 samples per second, the rate Whisper uses, so the picture stops at 8,000 Hz. The microphone is used only if you turn it on; the sound is analyzed in your browser and never recorded or sent anywhere.
Seeing sound: the spectrogram
Cut a sound into short windows, take the frequencies in each, and stack them side by side.
Tap or drag on the spectrogram (or use the arrow keys) to pick a moment. The lower chart is the spectrum of that one window: one column of the picture above.
Challenges
0 of 3 complete
- Choose Two notes, a semitone apart (440 and 466 Hz), and make them show as two separate lines. (not yet)
- Choose Two clicks, 12 ms apart, and make them show as two separate moments. (not yet)
- Raise the tone above 8,000 Hz, half the sampling rate, and find where it shows up. (not yet)
Try this
- Choose Sweep and watch a single rising line. Then switch the frequency axis to 80 mel bands: the same sweep now rises quickly at first and slowly at the top.
- Choose Two notes and step the window from 5 ms to 100 ms. Only the longest window shows two lines.
- Choose Two clicks and do the opposite: only short windows show two moments.
- Choose Tone and drag the frequency past 8,000 Hz. The line turns around and comes back down.
- Choose Microphone, turn it on, and whistle, hum, or say “sss” and “ooo”.
A spectrogram in NumPyPython with NumPy
import numpy as np
sr = 16000 # samples per second, the rate Whisper uses
t = np.arange(2 * sr) / sr # two seconds of sample times
notes = 0.3 * np.sin(2 * np.pi * 440 * t) + 0.3 * np.sin(2 * np.pi * 466.16 * t)
def spectrogram(x, win_ms, hop=64, n_fft=8192):
n = sr * win_ms // 1000 # window length in samples
window = np.hanning(n + 1)[:-1] # periodic Hann window: fades each frame in and out
frames = np.array([x[s:s + n] * window for s in range(0, len(x) - n, hop)])
power = np.abs(np.fft.rfft(frames, n=n_fft)) ** 2 # one FFT per frame (zero-padded)
return power, np.fft.rfftfreq(n_fft, 1 / sr) # frames x frequencies, and the frequencies
for win_ms in (25, 100):
power, freqs = spectrogram(notes, win_ms)
avg = power.mean(axis=0) # average spectrum over time
db = lambda f: 10 * np.log10(avg[np.argmin(np.abs(freqs - f))])
gap = db(453) - min(db(440), db(466.16))
print(f"{win_ms:3d} ms window ({1000 // win_ms} Hz steps): halfway between the notes {gap:+.1f} dB")
def hz_to_mel(f): # Slaney's mel scale, the one Whisper's filters use
f = np.asarray(f, dtype=float)
return np.where(f < 1000, 3 * f / 200, 15 + 27 * np.log(np.maximum(f, 1e-9) / 1000) / np.log(6.4))
print("mels at 500, 1000, 2000, 4000, 8000 Hz:", np.round(hz_to_mel([500, 1000, 2000, 4000, 8000]), 1)) 25 ms window (40 Hz steps): halfway between the notes +0.5 dB
100 ms window (10 Hz steps): halfway between the notes -8.0 dB
mels at 500, 1000, 2000, 4000, 8000 Hz: [ 7.5 15. 25.1 35.2 45.2]The lab simplifies three things. It pads each window with zeros up to a power of two before the FFT, which draws smoother curves but does not sharpen them. Its mel bands are triangles with a peak of 1, while Whisper’s are scaled so each band has the same area; the band edges are the same. And it sets 0 dB at the level of a full-scale sine wave. The speech sample is from the LJ Speech dataset (Keith Ito and Linda Johnson, 2017), which is in the public domain.