What sound is this?

In How computers hear a sound became a spectrogram: time across, pitch up, loudness as brightness. A spectrogram is a picture. So a model built to read pictures can read sounds, and that is exactly what the Audio Spectrogram Transformer does. This lesson runs it on your device to answer a question speech recognition does not ask: not what words were said, but what is making the sound. That task is audio classificationAudio classificationNaming the kinds of sound in a recording, such as speech, rain, or a siren, from a fixed list of labels. Unlike speech recognition, it says what is making the sound, not what words are said.Open in glossary.

A picture of sound, in patches

The model, AST, is from MIT (Gong, Chung, and Glass, 2021). Its input is a log-mel spectrogram with 128 bands, computed over 25 millisecond windows every 10 milliseconds, always 1,024 frames long: 10.24 seconds. Shorter sounds are padded, longer ones cut.

Then it does what a vision transformer does with a photo. It cuts the picture into 16 by 16 squares, turns each into a vector of 768 numbers, adds a learned position for each, and lets 12 transformer layers attend across all of them. One difference: the squares overlap. Each starts 10 frames or 10 bands after the last, so neighbors share 6 rows or columns. That gives 12 squares up the pitch axis and 101 along time: 1,212 patches. Two extra tokens, a classification token and a second one inherited from DeiT, gather what the model needs, and a final layer turns their average into a score for every label.

The authors did not start from random weights. They started from DeiT, a vision transformer trained on ImageNet photos, and averaged its patch layer over the three color channels so it would take a one-channel spectrogram. On AudioSet that start beat random weights clearly, from 0.366 to 0.459 in mean average precision on the full training set, the paper reports. Edges, blobs, and textures turn out to be useful in pictures of sound too.

Each label on its own

The labels come from AudioSet, Google’s collection of 2,084,320 ten-second clips from YouTube videos, each marked by people with the sounds they heard, from 527 labels (AudioSet). A clip of a street can be Speech, Car, and Siren at once, and about half of all clips contain Music and about half Speech.

So the last step is not the softmaxSoftmaxA function that turns a list of real-valued scores into probabilities that are positive and sum to 1, by exponentiating each score and dividing by the total.Taught in Predicting the next tokenOpen in glossary you met in Recognizing objects with your camera, where labels compete for one total of 100%. AST gives each label its own sigmoidSigmoidThe S-shaped function 1 / (1 + e^-z), which squashes any number into the range 0 to 1 so it can be read as a probability.Taught in Classification and probabilityOpen in glossary, a separate yes-or-no probability, and was trained with binary cross-entropy on each one. This is multi-label classificationMulti-label classificationClassification where one input can have several labels at once, such as a clip with both speech and music. Each label gets its own yes-or-no probability from a sigmoid, so the probabilities need not add up to 1.Open in glossary: the probabilities need not add up to anything.

What you saw

Clear recordings mostly got a fitting label at the top, though not always above 50%: the bells and the rain stayed under it. The train scored Train at 95%, and three more train labels above 50%; its 527 probabilities add up to about 4. Applause got Applause at 83%; the reading got Speech at 87%.

Mix speech over the plucked notes and both are heard at once: Speech 54%, Music 51%. Switch the same raw scores to a softmax and the two compete instead, falling to 30% and 26%. A softmax can say which label is most likely; it cannot say two things are both there.

The meow is the instructive mistake. The model hears Bleat at 63% and Sheep at 59%, and gives Cat under 5%. The clip is one and a half seconds of a Siamese cat, padded with almost nine seconds of nothing, and a single drawn-out meow does look like a bleat in a spectrogram. AudioSet’s own quality checks warn that its labels are not perfect either: checking ten clips of each, it found that 9 of those labeled Meow, and only 6 of those labeled Sheep, really contained that sound.

The made-up sounds show what the model learned from. The steady tone is a Sine wave at 84%: AudioSet has that label, with a few hundred clips. The siren sweep, a sine wave rising and falling like a wailing siren, gets no siren label at all alone. Mixed with speech, Police car (siren) comes second at 41%. The model never learned a definition of a siren, only the thousands of YouTube clips labeled Siren.