Show a chatbot a photo and ask what is in it, and it answers in a sentence. But a language model reads tokens: whole numbers from a fixed vocabulary, turned into vectors. A photo is millions of pixel values. Something has to turn one into the other. This lesson takes a real Vision-language modelA language model that also reads pictures: a vision encoder turns the picture into vectors, a small projector maps them into the language model's input space, and the model reads them alongside the text to answer in words.Open in glossary apart: Janus-Pro-1B, from DeepSeek, running on your device.
Three parts
A vision-language model has three parts, and each hands the next a list of vectors.
- A Vision encoderThe part of a multimodal model that turns a picture into a sequence of vectors, usually a vision transformer giving one vector per square patch. Janus-Pro's SigLIP encoder gives 576 vectors for a 384 by 384 picture.Open in glossary turns the picture into vectors. Janus-Pro’s is SigLIP, a vision transformer trained, like CLIP in Images and words together, to match pictures with their captions (Zhai et al., 2023). It reads a 384 by 384 picture in 16 by 16 squares: a 24 by 24 grid, 576 squares, one vector of 1,024 numbers each.
- A ProjectorIn a vision-language model, the small network that maps each vector from the vision encoder into the language model's embedding space, so the language model can read picture pieces like word tokens. Also called an adaptor or connector.Open in glossary maps each of those vectors into the language model’s own space. The language model’s token embeddings are 2,048 numbers wide, so a small two-layer network turns 1,024 numbers into 2,048, square by square.
- The language model reads the 576 picture vectors in the places of 576 tokens, between a start marker and an end marker, with ordinary text around them: the system prompt and your question, in the same kind of chat template any chat model uses. Then it writes the answer one token at a time.
- Pictureany sizelong side shrunk to 384, gray padding to a square
- Vision encoderSigLIP-L/16, 24 blocks576 vectors of 1,024: one per 16 by 16 square
- Projectortwo-layer networkeach vector to 2,048 numbers, the language model's width
- Language model24 blocks, about 1.5 billion parametersreads instructions, picture, and question as one sequence
- Answerone token at a timethe most likely token each step, until it ends
You are a helpful language and vision assistant ...<|User|>:<begin_of_image>576 picture vectors<end_of_image>What does the sign say?<|Assistant|>:
Nothing in between is a word. The picture never becomes a caption that the language model then reads; the language model attends to the picture vectors directly, at every step of its answer, as it attends to the words of your question. The Janus-Pro paper describes exactly this: SigLIP’s features “flattened from a 2-D grid into a 1-D sequence”, and “an understanding adaptor … used to map these image features into the input space of the LLM” (Chen et al., 2025).
What your picture becomes
Before the encoder sees anything, the picture is fitted to 384 by 384 pixels. Janus-Pro scales the long side to 384 and pads the short side with gray (red, green, and blue all 127), as its paper says and DeepSeek’s code does. Then the 24 by 24 grid is cut.
So every picture costs 576 image tokens. A photo from a phone, 4,032 by 3,024 pixels, becomes the same 576 vectors as a thumbnail. A tall print loses half its tokens to gray padding. Fine detail that does not survive the shrink is simply not in the input.
How a model reads a picture
Janus-Pro turns the picture into 576 image tokens, puts them in front of your question, and answers one token at a time.
Your photos and camera frames are processed on this device and never uploaded.
Loading the recorded results
Answer
Pick a picture and a question.
Challenges
0 of 3 complete
- Find a picture where at least 4 in 10 of the 576 image tokens see nothing but gray padding. (not yet)
- Find a wrong detail the model stated with more than 50% probability, and tap it. (not yet)
- Ask about the text on a text card and about the writing in a photo, and compare. (not yet)
Try this
- Pick the woodblock print and count the hatched squares. Then pick a photo shaped like your phone’s.
- Ask what is written on the motorcycle. Tap F, U, and JI and read their probabilities and the runners-up.
- Ask the sign card what it says, then ask the street photo. Both are signs.
- Read the receipt line by line, then ask for the whole small-print notice.
- With the model downloaded, photograph something with writing on it and ask what it says.
What you saw
Every picture became 576 image tokens, and the model read about 630 tokens before writing a word: 576 from the picture, two markers, and about 50 of text. On the laptop the recording was made on (an Apple M2, with other work running), turning the picture into vectors took 5 to 11 seconds on the CPU, the language model’s first pass over the whole input about 2 seconds on the GPU, and each token of the answer after that a few hundredths of a second. The picture is most of the input, and most of the work.
The answers to plain questions were right. Two cups of coffee, two croissants, a black and white checkered floor, a Boxer. The sign card came back right, partly in the model’s own words (“FRESH BREAD”, open 7 AM to 3 PM, closed on Mondays), and every line of the receipt word for word.
Then the wrong ones. The motorcycle’s tank says “pulsar” in curly script; the model said the word “FUJI” is written on it. The street sign, Irish and English for “shared space”, came back as “COMPLAISING SHARROD STREET”. The Hiroshige print was “created in the early 20th century”; Hiroshige died in 1858, the Metropolitan Museum of Art’s record notes. A temple corridor in Thanjavur, India, became “the ancient city of Petra in Jordan”. And the chart’s March bar, labeled 50, became 70: the value of the February bar next to it.
Reading text is where the 384 pixels show most. Large, plain letters on a card survive the shrink. The small print came back only in part: the first and last sentences word for word, but the model filled a gap it could not read with “in paperback, urban reader, and hardback” where the page says “in person or by telephone”, turned the late fee into a garbled sentence that it wrote twice, and left out the rule about rare books. The Janus-Pro paper names this limit itself: the input resolution of 384 by 384 “affects its performance in fine-grained tasks such as OCR” (reading text). OCR stands for optical character recognition.
How other models spend tokens
Janus-Pro’s fixed 576 is one choice. LLaVA, which made this recipe widely known, used a CLIP encoder at 224 pixels and a single linear layer as its projector (Liu et al., 2023); LLaVA-1.5 moved to 336 pixels and a two-layer projector, which gives the same 576 tokens from patches of 14 pixels (Liu et al., 2023). Qwen2-VL went the other way: it reads a picture at its own resolution, so a larger picture becomes more tokens, and merges each 2 by 2 block of patch vectors into one, so that a 224 by 224 picture becomes 66 tokens (Wang et al., 2024).
The trade is the one you saw in the lab. More image tokens keep more detail, so text and small objects survive, but every token costs memory and time in every step of the answer, as Generating fast: the KV cache showed.
How the parts learn to fitOptional
The vision encoder and the language model start out trained separately: SigLIP on pictures and captions, the language model on text. The projector is what joins them, so it is trained first. In the Janus recipe, stage one trains only the adaptors (and the image head used for making pictures) with the rest frozen; stage two trains everything except the two image encoders on a mix of text, pictures with text, and text-to-image data; stage three is supervised fine-tuning on conversations about pictures, with the understanding encoder unfrozen too. For Janus-Pro, DeepSeek added about 90 million examples for understanding, including captions, tables, charts, and documents (Chen et al., 2025).
Janus-Pro is unusual in one way: it also makes pictures, and for that it uses a different encoder, an image tokenizer whose codes the model can write one at a time (Pictures, one token at a time). The Janus papers keep the two encoders apart because understanding needs vectors rich in meaning, while making pictures needs codes good for rebuilding pixels. Where multimodal models are heading looks at models that read and write more than one kind of data.
Key ideas
- A vision-language model is a vision encoder, a projector, and a language model. The picture reaches the language model as vectors in the places of tokens, never as words.
- Janus-Pro turns every picture into 576 image tokens: 384 by 384 pixels in 16-pixel squares. Detail that does not survive the shrink is not in the input, and padding wastes tokens on tall or wide pictures.
- The answer is written like any text, one likely token after another. A token’s probability measures how well it continues the text, not whether the picture agrees.
- Reading text works when letters stay large and plain after the shrink, and fails, plausibly and confidently, when they do not.
Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).