Janus-Pro-1B, from DeepSeek, makes a 384 by 384 picture the way a language model writes a sentence: one token at a time. Each token is a code from a learned codebook of 16,384, standing for a 16 by 16 square of the picture. The model writes 576 of them, left to right and top to bottom, and a decoder turns the finished grid into pixels. The lesson explains each step and compares it with diffusion.
Text to image, one token at a time
Janus-Pro writes a picture as 576 codes, left to right and top to bottom, then a decoder turns the codes into pixels.
Loading the recorded results
The picture appears here.
Janus-Pro-1B on WebGPU, at temperature 1 as in DeepSeek's example code. The same seed gives the same picture on the same device; another GPU or browser can round numbers differently, pick a different code somewhere, and from there draw a different picture.
Recorded: each example with seeds 0 to 2 at guidance 5; the red apple and the lighthouse also at every guidance with seed 0.
Challenges
0 of 4 complete
- Make two pictures from the same prompt and guidance with different seeds. (not yet)
- Find the guidance setting at which the picture ignores the prompt. (not yet)
- Scrub the finished picture back to the code where exactly the top half is decided. (not yet)
- Ask for an exact number of things, then count what you get. (not yet)
Try this
- Keep the prompt and press Next seed a few times. How different are the pictures?
- Set guidance to 0, then 1, then 5, with the same seed.
- Pick the five ducks and count them. Then pick the sign and read it.
- When a picture is finished, drag Show codes up to back to the middle. Which part of the picture had the model decided by then?
- Compare the red apple and the green apple with the same seed.
Codes, reading order, guidance, and seeds in NumPyPython with NumPy
import numpy as np
# 1. A vector-quantized (VQ) tokenizer turns each 16 x 16 square of a picture
# into one code: the number of the nearest entry in a learned codebook.
# Janus-Pro's codebook has 16,384 entries of 8 numbers; this one has 4 of 2.
codebook = np.array([[0.0, 0.0], [1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
squares = np.array([[0.9, 0.2], [0.1, 0.8], [0.6, 0.7]]) # the encoder's output
dist = ((squares[:, None, :] - codebook[None, :, :]) ** 2).sum(axis=-1)
print("codes:", dist.argmin(axis=1))
# 2. The model writes codes in reading order: code i lands in row i // 24,
# column i % 24 of the 24 x 24 grid, so 576 codes make a 384 x 384 picture.
for i in (0, 23, 24, 287, 575):
print(f"code {i:3d} -> row {i // 24:2d}, column {i % 24:2d}")
# 3. For each code, guidance mixes the logits computed with the prompt (cond)
# and with the prompt replaced by pad tokens (uncond).
cond = np.array([1.2, 0.9, 0.3, 0.0])
uncond = np.array([0.6, 0.6, 0.5, 0.5])
def probs(s):
logits = uncond + s * (cond - uncond)
p = np.exp(logits - logits.max())
return p / p.sum()
for s in (0, 1, 5):
p = probs(s)
print(f"guidance {s}: p = {np.round(p, 3)}, spread {-(p * np.log2(p)).sum():.2f} bits")
# 4. Sampling picks each code at random from p; a seed fixes the random
# numbers, so the same seed picks the same codes again.
for seed in (0, 0, 1):
rng = np.random.default_rng(seed)
print(f"seed {seed}: codes", rng.choice(4, size=10, p=probs(1)))codes: [1 2 3]
code 0 -> row 0, column 0
code 23 -> row 0, column 23
code 24 -> row 1, column 0
code 287 -> row 11, column 23
code 575 -> row 23, column 23
guidance 0: p = [0.262 0.262 0.238 0.238], spread 2.00 bits
guidance 1: p = [0.408 0.303 0.166 0.123], spread 1.85 bits
guidance 5: p = [0.804 0.179 0.013 0.003], spread 0.81 bits
seed 0: codes [1 0 0 0 2 3 1 2 1 3]
seed 0: codes [1 0 0 0 2 3 1 2 1 3]
seed 1: codes [1 3 0 3 0 1 2 1 1 0]