How a model reads a picture

Janus-Pro-1B, from DeepSeek, reads a picture by turning it into 576 vectors, one for each 16 by 16 square of a 384 by 384 version of it, and putting them in front of your question as if they were words. The lesson explains each step.

Try this

Pick the woodblock print and turn on the token grid. How many of the 576 tokens see any of the print?
Ask what is written on the motorcycle. Tap each word of the answer and read its probability.
Read the receipt, then the small print. Which one comes back word for word?
Ask how many umbrellas were sold in March, and check the bar yourself.
With the model downloaded, ask your own question, or use your own photo or the camera. Nothing leaves your device.

From a photo to the language model's input, in NumPyPython with NumPy

import math

import numpy as np

SIZE, PATCH = 384, 16  # SigLIP-L/16 at 384 x 384 pixels

# 1. The picture: long side to 384, gray padding to a square, 16 x 16 squares.
def image_tokens(width, height):
    scale = SIZE / max(width, height)
    w, h = int(width * scale), int(height * scale)
    x0, y0 = (SIZE - w) // 2, (SIZE - h) // 2
    grid = SIZE // PATCH
    padding = 0
    for r in range(grid):
        for c in range(grid):
            x, y = c * PATCH, r * PATCH
            if not (x < x0 + w and x + PATCH > x0 and y < y0 + h and y + PATCH > y0):
                padding += 1
    return grid * grid, padding

for name, (w, h) in {"phone photo": (4032, 3024), "square": (500, 500),
                     "tall print": (466, 1024), "huge panorama": (12000, 2000)}.items():
    total, pad = image_tokens(w, h)
    print(f"{name:13s} {w:5d} x {h:4d} -> {total} image tokens, {pad} only padding")

# 2. The vision encoder gives one 1,024-number vector per square; a small
#    two-layer network (the adaptor) maps each into the language model's
#    2,048-number embedding space. Random weights here, real shapes.
rng = np.random.default_rng(0)
features = rng.normal(size=(576, 1024))
W1, W2 = rng.normal(size=(1024, 2048)) * 0.03, rng.normal(size=(2048, 2048)) * 0.02
erf = np.vectorize(math.erf)
gelu = lambda x: 0.5 * x * (1 + erf(x / math.sqrt(2)))
image_embeds = gelu(features @ W1) @ W2
print("image embeddings:", image_embeds.shape)

# 3. The language model's input: text tokens, the picture's 576 vectors
#    between two marker tokens, then the question, all in one sequence.
system, question = 46, 8  # token counts (typical for the lab)
sequence = ["text"] * system + ["<begin_of_image>"] + ["image"] * 576 \
    + ["<end_of_image>"] + ["text"] * question
print("tokens read before answering:", len(sequence))
print("share that comes from the picture:", f"{578 / len(sequence):.0%}")

phone photo    4032 x 3024 -> 576 image tokens, 144 only padding
square          500 x  500 -> 576 image tokens, 0 only padding
tall print      466 x 1024 -> 576 image tokens, 288 only padding
huge panorama 12000 x 2000 -> 576 image tokens, 480 only padding
image embeddings: (576, 2048)
tokens read before answering: 632
share that comes from the picture: 91%

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).