The models in this site’s labs each do one or two things: Janus-Pro reads and makes pictures, Whisper hears, Supertonic speaks, and in A voice assistant taken apart three of them were chained. Much recent work aims at a single Multimodal modelA model that works with more than one kind of data, such as text, pictures, and sound. Some only read other kinds of data and answer in text; others can also produce pictures or speech.Open in glossary that reads and writes several kinds of data at once. This lesson sorts the designs, using only what their papers and model cards report.
Four designs
- Encoder and projectorIn A separate vision encoder gives vectors; a projector maps them into the language modelOut Text onlyLLaVA; Janus-Pro reading pictures
- Everything as tokensIn Pictures quantized into codes from a codebook, read like wordsOut Text and picture codes, one token at a timeChameleon
- Two encoders, one modelIn A meaning-rich encoder for reading picturesOut Picture codes from a separate tokenizer for making themJanus, Janus-Pro
- Text, pictures, and sound togetherIn Text, pictures, and sound into one networkOut Text, and pictures or speech from the same networkGemini, GPT-4o, Qwen2.5-Omni; Moshi for speech alone
Encoder and projector. A separately trained vision encoder turns a picture into vectors, and a small projector maps them into a language model’s input. LLaVA made this recipe common (Liu et al., 2023); Janus-Pro reads pictures this way. It is cheap, because both big parts start from existing models, but the model only reads pictures; it answers in text.
Everything as tokens. Chameleon, from Meta, turns each 512 by 512 picture into 1,024 tokens from a codebook of 8,192, adds them to its vocabulary, and trains one transformer from scratch “on an interleaved mixture of all modalities, i.e., images, text, and code”, so the same model reads and writes pictures token by token (Chameleon Team, 2024). Its authors call this early fusion. They also report that it made training unstable, and that the 34-billion-parameter model needed changes such as normalizing queries and keys and reordering its layer norms to train stably.
Two encoders, one model. Janus keeps one transformer but gives it two ways in: SigLIP features for reading pictures and VQ codes for writing them, because, its papers argue, understanding needs meaning while generation needs detail (Chen et al., 2025).
Text, pictures, and sound together. Google describes Gemini as “natively multimodal, as they are trained jointly across text, image, audio, and video”, taking audio as 16 kHz features from its Universal Speech Model and able to “natively output images using discrete image tokens” (Gemini Team, 2023). OpenAI describes GPT-4o as “an autoregressive omni model, which accepts as input any combination of text, audio, image, and video and generates any combination of text, audio, and image outputs”, “trained end-to-end” so that “all inputs and outputs are processed by the same neural network” (OpenAI, 2024). Open models do the same for speech: Moshi listens and talks at once in parallel audio streams (Défossez et al., 2024), and Qwen2.5-Omni pairs a text-writing “Thinker” with a “Talker” that turns the Thinker’s hidden states into speech (Xu et al., 2025).
Not every design turns pictures into discrete tokens. Transfusion, also from Meta, trains one transformer with next-token prediction on text and a diffusion loss on continuous image patches in the same sequence, and its authors report that this “scales significantly better than quantizing images and training a language model over discrete image tokens” (Zhou et al., 2024). The diffusion lessons explain that loss.
What the papers say is gained
One network can use what it learns from one kind of data on another. The Gemini report says its natively multimodal models set “a new state of the art across a wide range of text, image, audio, and video benchmarks”, and Chameleon’s authors report that their 34B model matched or beat existing language models on text-only tests while also handling pictures.
Sound shows the clearest difference. A cascade passes only text between its parts; OpenAI wrote that its earlier three-model Voice Mode could not “directly observe tone, multiple speakers, or background noises”, while GPT-4o’s system card reports answers to audio in 320 milliseconds on average.
What is still hard
Tokens are expensive. Every picture or second of sound becomes many tokens, and every token costs memory and time at every step, as Long context in practice showed for text. Janus-Pro spends 576 tokens on a 384 by 384 picture; Chameleon 1,024 on a 512 by 512 one. Moshi’s codec runs at 12.5 frames a second with several codebooks per frame.
Resolution limits what can be read. Janus-Pro’s paper names its 384 by 384 input as the reason it is weaker at fine-grained tasks such as reading text, which the lab showed on a street sign.
Training is harder. Chameleon’s paper is largely about keeping mixed-modal training stable at scale.
Confident mistakes come along. Every design here writes its text answers by next-token prediction, so a wrong detail can arrive at a high probability, in a description of a picture as in any other text.
Key ideas
- Multimodal models range from a language model with an image encoder attached, which only reads pictures, to single networks that read and write text, pictures, and sound.
- Early fusion turns pictures into tokens that one transformer reads and writes; Janus uses separate encoders for reading and writing; Transfusion mixes next-token prediction with diffusion.
- “Natively multimodal” and “end to end” mean trained on several kinds of data from the start, with one network handling inputs and outputs.
- Costs remain: many tokens per picture or second of sound, limited resolution, harder training, and confident mistakes.