What sound is this?

The Audio Spectrogram Transformer reads a sound’s log-mel spectrogram as a picture, in 1,212 overlapping 16 by 16 patches, and scores each of AudioSet’s 527 kinds of sound with its own probability. The lesson explains the model and what its scores mean.

What sound is this?

The Audio Spectrogram Transformer reads a sound's spectrogram in patches and scores 527 kinds of sound, each on its own.

A woman reading a book aloud (LJ Speech, public domain)

Recorded from Audio Spectrogram Transformer

Loading the model's recorded outputs for every bundled sound, alone and mixed with each of the five sounds you can mix in.

About the download

Label your own recordings from the microphone. This uses Audio Spectrogram Transformer, an 87-million-parameter model from MIT (2021) that reads a sound's spectrogram in patches, like a vision transformer reads a photo, and scores 527 kinds of sound from AudioSet.

The first time, your browser downloads about 91 MB of model files from Hugging Face, plus a 14 MB runtime from this site. Both are cached for later visits. Everything runs on your device; nothing you record is sent anywhere.

Manage downloads

Loading the recorded results

What the model reads: 128 mel bands every 10 milliseconds for 10.24 seconds, low pitches at the bottom. Short sounds are padded. The outlined square is one 16 by 16 patch; the dashed one is the next, 10 frames later, so neighbors overlap by 6. There are 12 by 101, 1,212 patches in all.

Top labels, each scored on its own

Labels above 50%n/a

All 527 probabilities add up ton/a

Sound readn/a

Time to labeln/a

Challenges

0 of 3 complete

Mix two sounds so that the model gives each of them its own label above 50% at the same time. (not yet)
With both labels above 50%, switch to shared probabilities and see what happens to them. (not yet)
Get the siren sweep heard as a siren: any siren label above 30%. (not yet)

The model is MIT's Audio Spectrogram Transformer (BSD 3-Clause license), fine-tuned on AudioSet, run here as an 8-bit ONNX export. Recordings are CC0 from Wikimedia Commons (church bells by Mxsmanic, applause by Sandermotions, rain by kvgarlic, cat by freemaster2, train by felix.blume, bird by Tiago Lubiana); the speech is from LJ Speech (public domain); the tone, sweep, noise, and notes were made for this site. Nothing you record leaves your device.

Try this

Play the cat, then look at its labels. Which animal does the model hear?
Mix the plucked notes into the speech. How many labels pass 50%?
Switch the same mix to shared probabilities.
Try the siren sweep alone, then mix in speech.
With the model downloaded, record a sound near you: a kettle, keys, a door, your own voice. Nothing leaves your device.

Patches, padding, and two ways to turn scores into probabilitiesPython with NumPy

import numpy as np

# 1. The input: 128 mel bands every 10 ms, for 1,024 frames (10.24 s).
#    Patches are 16 x 16 and start every 10 frames and every 10 bands.
bands, frames, patch, stride = 128, 1024, 16, 10
rows = (bands - patch) // stride + 1
cols = (frames - patch) // stride + 1
print(f"patches: {rows} x {cols} = {rows * cols}")

# A short sound fills only part of the 1,024 frames; the rest is padding.
seconds = 1.5
used = int(1 + (seconds * 16000 - 400) // 160)
print(f"a {seconds} s sound fills {used} of {frames} frames")

# 2. The scores: one raw score (logit) per label.
labels = ["Speech", "Music", "Piano", "Dog"]
logits = np.array([0.16, 0.04, -1.0, -6.0])

sigmoid = 1 / (1 + np.exp(-logits))         # each label on its own
softmax = np.exp(logits) / np.exp(logits).sum()  # labels share 100%
for name, p, q in zip(labels, sigmoid, softmax):
    print(f"{name:7s} on its own {p:4.0%}   shared {q:4.0%}")
print(f"sums: on their own {sigmoid.sum():.2f}, shared {softmax.sum():.2f}")

patches: 12 x 101 = 1212
a 1.5 s sound fills 148 of 1024 frames
Speech  on its own  54%   shared  45%
Music   on its own  51%   shared  40%
Piano   on its own  27%   shared  14%
Dog     on its own   0%   shared   0%
sums: on their own 1.32, shared 1.00