How AI sees, hears, and makes pictures

Point a phone at a flower and an app names it. Say a message out loud and it appears as text. Type a sentence and a picture appears. These are often the first things people try with AI, and they can feel like the most magical. They rest on ideas from this track: everything becomes numbers, a network learns from examples, and a model can run on your own device. In this lesson you use a real model of each kind, running in this page, and catch each one making mistakes.

Three models, three directions

Each model below is a neural networkNeural networkA model built from layers of artificial neurons, where each layer's outputs become the next layer's inputs. Training adjusts all of the weights together.Taught in Networks of neuronsOpen in glossary trained on examples, like the tiny networks in Networks of neurons, only far larger. What sets them apart is what goes in and what comes out.

Seeing: from pixels to a label

In Turning things into numbers you saw that a picture is a grid of numbers, a few for every pixel. A phone photo is millions of them. An image classifier takes those numbers in and gives back one percentage for each label on a fixed list. The task is called classificationClassificationA prediction task where the answer is one of a fixed set of categories, such as spam or not spam, or which digit a drawing shows.Taught in Classification and probabilityOpen in glossary.

The classifier here, DeiT-Tiny, learned from ImageNetImageNetA large collection of labeled photos organized by the WordNet hierarchy. Its best-known subset, 1.28 million training images in 1,000 categories, became the standard benchmark for image classification.Taught in Recognizing objects with your cameraOpen in glossary: 1.28 million photos that people sorted into 1,000 categories. Nobody wrote a rule for what a boxer dog looks like. During training, the network’s numbers were nudged again and again toward the labels people had given (and, for this distilled version, toward the answers of a larger network that served as its teacher), the same way the artificial neuron learned its weights.

Recognizing objects

Copy a linkResetFocus mode

A classifier names the whole picture with one of 1,000 labels; a detector boxes every object it knows from a list of 80.

Model

Name the pictureFind objects

Sample photos

Use your own photoUse the camera

Your photos and camera frames are processed on this device and never uploaded.

Recorded from DeiT-Tiny

Loading the model's recorded outputs for the eight sample photos.

About the download

Name any picture with one of ImageNet's 1,000 categories. This uses DeiT-Tiny, a 6-million-parameter vision transformer from Meta (2020) that sorts a photo into 1,000 ImageNet categories.

The first time, your browser downloads about 24 MB of model files from Hugging Face, plus a 14 MB runtime from this site. Both are cached for later visits. Everything runs on your device; no photo or camera frame is sent anywhere.

Manage downloads

Checking for a saved copy

Loading the recorded results

ModelDeiT-Tiny (24 MB)

Time for this picturen/a

Entropy over all 1,000 labelsn/a

Challenges

0 of 3 complete

1. Name the cat on a motorcycle with the classifier, then use the detector to find both the cat and the motorcycle in it. (not yet)
2. Find a picture whose top label gets less than 30%. (not yet)
3. Find a picture the classifier labels wrongly while giving that label more than 70%, and mark it wrong. (not yet)

What you saw:

• It can only answer from its list. ImageNet’s 1,000 categories include more than 100 breeds of dog, the moped, and the motor scooter, but no motorcycle and no general category for a person. So the motorcycle became a “moped” at 75%. Whatever you show this model, its answer is one of its 1,000 labels.
• The percentage is not the chance that it is right. The corridor is a stone colonnade in a temple, and the classifier calls it “prison” at 78%. The percentages are shared out among its labels and always add up to 100%, whether or not any label fits. So 78% says how strongly the model prefers “prison” over its other labels, not how likely it is to be right. A low top score, like the 25% the breakfast table gets for “plate”, often means that several labels fit a little.
• Naming a picture is not finding things in it. Find objects runs a second model, YOLOS-Tiny (27 MB), for object detectionObject detectionFinding every object of known kinds in an image and reporting each one as a box (its bounding box), a label, and a confidence score, instead of one label for the whole picture.Taught in Recognizing objects with your cameraOpen in glossary: a box around each object it knows, from a shorter list of 80 everyday kinds such as person, car, cup, and cat. It finds both the cat and the motorcycle that the classifier had to squeeze into one label.

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).

Hearing: from sound to a picture to words

Sound is air pressure wobbling very fast. A microphone measures it thousands of times a second, so a recording is a long list of numbers; Whisper, the speech model here, uses 16,000 of them for every second of sound. A list of pressures tells the eye very little, so the first step is to turn it into a picture called a spectrogramSpectrogramA picture of sound with time across, frequency up, and loudness as color, made by taking the frequencies of many short, overlapping windows of the recording and placing them side by side.Taught in How computers hearOpen in glossary: time runs from left to right, pitch from low at the bottom to high at the top, and the stronger the color, the louder that pitch is at that moment.

Turning speech into text is speech recognitionSpeech recognitionTurning recorded speech into text. Modern systems such as Whisper are neural networks that read a spectrogram of the audio and write the transcript one token at a time.Taught in Speech to textOpen in glossary. OpenAI released Whisper in September 2022, trained on 680,000 hours of audio paired with transcripts collected from the internet (Radford and colleagues, 2022). It reads the spectrogram as numbers, then writes the transcript one word or piece of a word at a time, much like the chatbot in How a chatbot writes, except that every choice also depends on the sound.

What you saw:

• Your voice as a picture. Vowels appear as stacks of horizontal stripes: your vocal folds buzz at one pitch, and the buzz contains many multiples of it, one stripe each (they crowd together toward the top, because this picture squeezes high pitches). A sound like “s” is a haze high up, because it is hissing air rather than a buzz. Whisper does not listen the way you do; it reads this picture, as numbers.
• Noise. On the clean reading Whisper made no mistakes. With hiss added, it made at most one mistake down to 5 dB, got about one word in five wrong at 0 dB, garbled the words at louder hiss, and at the loudest wrote a fluent sentence that nobody said, beginning “This is the only chance to”.
• Silence. Given six seconds of silence, it wrote “you”; given a faint hiss, “and”. The part of Whisper that writes is a language model trained on audio that nearly always had speech in it, so it produces likely words even when the sound gives it nothing to go on. This is a hallucinationHallucinationWhen a language model states something false or invented, such as a fake quote or citation, in the same fluent, confident style as true statements.Taught in What AI gets wrongOpen in glossary, the same kind of mistake a chatbot makes in What AI gets wrong, and it happens outside labs too: a study of OpenAI’s Whisper service in 2023 found whole invented phrases or sentences in roughly 1% of transcriptions (Koenecke and colleagues, 2024).
• Speakers. Whisper got the sentence right for four of the five speakers and made two mistakes for the speaker of Indian English. One sentence proves little, so this site ran 20 sentences from each speaker through the same model: about 2 to 4 mistakes per 100 words for the two American speakers and the Canadian, 5 for the Scottish speaker, and 15 for the Indian English speaker. Each accent here is one person, so accent and person cannot be told apart, but Whisper’s own model card warns of uneven accuracy across accents. Errors that fall more often on some groups of people are what What AI gets wrong calls unfair errors; one common cause is training data that covers some people better than others.

Making pictures: from noise to a digit

The first two models turn something real into a label or text. A generator goes the other way: you give it a request, and it gives back new numbers that form a picture. Many image generators are diffusion modelsDiffusion modelA generative model that learns to reverse a gradual noising process, so it can turn pure random noise into new data such as images, step by step.Taught in Diffusion: from noise to dataOpen in glossary, and the idea fits in a few sentences.

Training. Take a real handwritten digit and sprinkle random noise over it, like static on an old television. Show the noisy picture to a network and train it to tell the noise apart from the digit underneath. Do this with every amount of noise, from a light sprinkle to so much that nothing of the digit is left, on tens of thousands of digits.

Drawing. Start from pure noise, which hides no digit at all. Ask the network for its guess of the finished picture underneath, move the picture a small step toward that guess, which removes a little of the noise, and repeat. Tell it which digit you want, and every guess leans toward that digit.

The model below was trained by this site on the 60,000 handwritten digits of a collection called MNISTMNISTA classic dataset of 70,000 handwritten digits, each a 28 x 28 grayscale image: 60,000 for training and 10,000 for testing.Taught in Reading handwritingOpen in glossary. It draws a 28 by 28 pixel digit in 20 steps.

import numpy as np

# A 5 x 5 picture of a plus sign, on the model's scale: 1 is ink, -1 is paper.
plus = -np.ones((5, 5))
plus[2, :] = 1
plus[:, 2] = 1

# How much of a picture survives at each of 1,000 noise levels (DDPM's schedule).
kept = np.sqrt(np.cumprod(1 - np.linspace(1e-4, 0.02, 1000)))


def draw(x):  # dark ink "#", light ink "+", paper "."
    return ["".join("#" if v > 0.5 else "+" if v > 0 else "." for v in row) for row in x]


rng = np.random.default_rng(7)
x = rng.standard_normal((5, 5))  # the starting noise: random numbers, no picture
levels = [999, 749, 499, 249, 0]  # four big steps, from pure noise to almost none

frames = [draw(x)]
for now, nxt in zip(levels, levels[1:]):
    guess = plus  # stand-in: a trained network makes a new guess at every step
    noise = (x - kept[now] * guess) / np.sqrt(1 - kept[now] ** 2)  # the noise x must contain
    x = kept[nxt] * guess + np.sqrt(1 - kept[nxt] ** 2) * noise  # less noise, more guess
    frames.append(draw(x))

for row in zip(*frames):
    print("   ".join(row))

++...   .+...   .++..   ..#..   ..#..
.+#..   .+#..   ..#..   ..#..   ..#..
+++..   #++.+   ##+.+   ###+#   #####
#....   #....   +....   ..+..   ..#..
...++   ...++   .....   .....   ..#..

In the code the guess is always the plus sign, standing in for a perfect network. The real network has to work its guess out from the noisy picture at every step. What you saw in the lab:

• Rough shape first, details last. Early on, so much noise is left that many different drawings could be hiding under it, and the network’s best guess is a blur, an average of them. With each step fewer drawings fit, and the guess sharpens.
• Your request picks the digit; the noise picks the drawing. From one starting noise you can draw every digit, and the digits tend to share a look: measured over 200 starting noises, when one digit drawn from a noise used more ink than is usual for that digit, the others drawn from the same noise tended to as well (Diffusion on images). A new starting noise gives a differently drawn 3, and the same noise with the same request gives exactly the same drawing again. The starting noise comes from a random seedRandom seedA number that sets where a computer's random number generator starts. The same seed gives the same "random" numbers every time, so a result that uses randomness can be repeated exactly.Open in glossary. Diffusion software such as Hugging Face’s diffusers library starts each picture from fresh random noise unless told otherwise, so the same prompt gives a different picture each time. Fixing the seed lets it repeat a picture, though its documentation warns that even an identical seed does not guarantee the same result on different hardware (diffusers documentation).
• With no request. Any digit runs the network without telling it a digit, which it can do because training hid the digit one time in ten. It still draws something that looks like handwriting, since handwritten digits are all it ever learned from, but sometimes a shape between two digits that the recognizer cannot read with confidence. Asking for a digit pushes every step toward that one.
• It does not copy. The network never looks training digits up while drawing. Measured over 1,000 of its drawings, a typical drawing is about as far from the nearest of the 60,000 training digits as new, real handwriting is. Very large image models can memorize, though: researchers pulled near-copies of some training images out of Stable Diffusion, mostly images that appeared many times in its training data (Carlini and colleagues, 2023).

From digits to a sentence

Text-to-image modelsText-to-image modelA generative model that makes a picture from a written description, called a prompt. Many, such as Stable Diffusion, are diffusion models guided by the text; others build the picture piece by piece, the way a chatbot writes.Open in glossary make a picture from a written description instead of a digit. Stable Diffusion runs the same loop on a compressed version of the picture, guided by your prompt at every step, and turns the result back into pixels at the end. The scale is what differs: its first version makes 512 by 512 color pictures, its denoising network has about 860 million learned numbers against about half a million here, and it was trained on images from a subset of LAION-5B, a public dataset of links to billions of web images with their captions (CompVis).

Not every generator removes noise. OpenAI says the image generation built into GPT-4o “is an autoregressive model”, unlike DALL·E, which OpenAI describes as a diffusion model (OpenAI, March 2025). Autoregressive means it builds its output piece by piece, each piece depending on the ones before, the way a chatbot writes.

Everything runs on your device

All three models run inside this page. Your photos, camera frames, and recordings were processed on your device and never sent anywhere. As in Where your AI runs, the price was the downloads (24 MB, 27 MB, and 80 MB for the models you chose to download, and 1.3 MB for the digit model) and your device’s own speed. Apps that name plants or type what you say may run their model on your phone or send your photo or voice to a company’s computers; only the app’s own documentation says which.

Where to go next

This lesson skipped the machinery: how a network finds patterns in pixels, what a transformer is, why the guess starts out blurry. The advanced tracks explain each properly. They build on the neural networks and transformers tracks, so expect some math.

• Seeing. How computers see shows how a network reads a photo in small patches. Recognizing objects with your camera runs this lesson’s lab with confidence and detection thresholds in depth. Where the classifier looks tests what a classifier relies on, and Finding anything you can name runs a detector that looks for any word you type, on photos or your camera.
• Hearing. How computers hear shows how the picture of sound is made, Speech to text goes inside Whisper, and Text to speech runs a model that speaks.
• Making pictures. Diffusion: from noise to data explains why removing noise works, and Diffusion on images runs this lesson’s digit model with all of its settings.