One space for images and text

Type “dog on a beach” into a modern photo app and it finds your pictures, even though nobody tagged them with that phrase. Text-to-image generators need the same skill in the other direction: they must know which pictures fit a prompt. Both often rely on a model that maps images and sentences into one shared embeddingEmbeddingA list of numbers (a vector) that represents a token, word, sentence, or image, learned so that similar things end up with similar vectors.Taught in Word embeddingsOpen in glossary space, where a picture and a good description of it land close together.

The best-known model of this kind is CLIP, introduced by OpenAI (Radford et al., 2021). This lesson trains a miniature version in your browser; Images and words together runs the real model on your own photos.

Two encoders, one space

CLIP has two separate networks. An image encoder (the paper tried ResNets and Vision Transformers) turns a picture into a vector. A text encoder (a transformer) turns a caption into a vector of the same length. Neither network knows anything about the other’s input. The only thing connecting them is how they are trained.

Both vectors are scaled to length 1, so the comparison between an image $i$ and a caption $j$ is their cosine similarityCosine similarityA measure of how closely two vectors point in the same direction, from -1 (opposite) through 0 (perpendicular) to 1 (same direction), ignoring their lengths.Taught in Vectors and similarityOpen in glossary $s_{ij}$, a number from $-1$ (opposite directions) to $1$ (same direction).

Learning from pairs instead of labels

CLIP was trained on 400 million image and text pairs collected from publicly available sources on the internet, each image paired with text that appeared alongside it. Nobody labeled them with categories. The training signal is simply which caption came with which image.

Take a batch of $N$ pairs and compute every image against every caption: an $N \times N$ matrix of similarities whose diagonal holds the true pairs. Contrastive learningContrastive learningTraining that pulls the embeddings of matching pairs together and pushes non-matching pairs apart, so the model learns a useful space without category labels.Open in glossary asks each image to pick out its own caption from the whole row, and each caption to pick out its own image from the whole column. As a formula, with temperature $\tau$:

Each bracket is an ordinary softmaxSoftmaxA function that turns a list of real-valued scores into probabilities that are positive and sum to 1, by exponentiating each score and dividing by the total.Taught in Predicting the next tokenOpen in glossary cross-entropy, once along a row and once along a column. This is often called the InfoNCE loss. Every other item in the batch acts as a negative example, which is one reason CLIP used very large batches of 32,768 pairs.

Below, nine toy pictures (three colors times three shapes) are paired with nine captions. Both encoders start random. Press Train and watch both views.

import numpy as np

def unit(v):
    return v / np.linalg.norm(v, axis=1, keepdims=True)

def infonce(sim, tau):
    # Symmetric InfoNCE: each image must pick its own caption from its row,
    # and each caption its own image from its column.
    logits = sim / tau
    rows = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))
    cols = logits - np.log(np.exp(logits).sum(axis=0, keepdims=True))
    return -(np.trace(rows) + np.trace(cols)) / (2 * len(sim))

# Three images and their three captions, embedded in 2 dimensions.
images = unit(np.array([[1.0, 0.2], [-0.3, 1.0], [-0.7, -0.8]]))
captions = unit(np.array([[0.9, -0.1], [0.1, 1.0], [-0.9, -0.5]]))
sim = images @ captions.T  # cosine similarity: image i (row), caption j (column)
for row in sim:
    print("  ".join(f"{v:5.2f}" for v in row))
for tau in (1.0, 0.3):
    print(f"loss at temperature {tau}: {infonce(sim, tau):.3f}")

# Nine pairs, each image exactly on its caption, spread evenly around the
# circle. This is about where the demo's training settles.
angles = 2 * np.pi * np.arange(9) / 9
pairs = np.stack([np.cos(angles), np.sin(angles)], axis=1)
for tau in (1.0, 0.3, 0.05):
    print(f"nine perfect pairs at temperature {tau}: loss {infonce(pairs @ pairs.T, tau):.3f}")

0.95   0.29  -0.95
-0.39   0.92  -0.21
-0.57  -0.81   0.94
loss at temperature 1.0: 0.435
loss at temperature 0.3: 0.051
nine perfect pairs at temperature 1.0: loss 1.433
nine perfect pairs at temperature 0.3: loss 0.723
nine perfect pairs at temperature 0.05: loss 0.018

What the miniature leaves out

The demo is honest about the method but small in every other way:

• Its “image features” are five hand-made numbers per picture (how red, green, and blue it is, how round, how many corners). CLIP’s image encoder reads raw pixels.
• Its “text features” count words from a six-word vocabulary. CLIP’s text encoder is a transformer that reads tokens in order.
• Its encoders are single linear maps into two dimensions, so you can see the space. CLIP’s embeddings have hundreds of dimensions.
• Its temperature is a slider. CLIP learns the temperature as a parameter during training, starting from 0.07.

Why one space is so useful

Once images and text live in the same space, many tasks become lookups.

Zero-shot classification. To sort photos into categories, write each category as a caption (“a photo of a dog”, “a photo of a cat”) and pick the caption closest to each image. No category-specific training is needed, which is what zero-shotZero-shotDoing a task without any examples of that specific task during training or in the prompt, for example classifying images into categories the model was never trained to label.Open in glossary means here. The CLIP paper reports that this matched the accuracy of the original ResNet-50 on ImageNet without using any of ImageNet’s 1.28 million labeled training images.

Search in both directions. Embed every photo once, embed the query text, and return the nearest photos. Or start from an image and find captions or similar images.

Steering other models. Many image generators use a CLIP-style text encoder to turn the prompt into the vectors that condition the denoising network, and many vision-language models connect a CLIP-style image encoder to a language model. Both uses have moved since 2022. Stable Diffusion v1 read prompts with CLIP alone; by 2025 and 2026, open generators such as Qwen-Image and FLUX.2 read them with a vision-language model instead (Diffusion on images). On the reading side, CLIP-style encoders are still common: Janus-Pro reads pictures with SigLIP and ByteDance’s BAGEL with SigLIP 2, both described below.

Shared spaces in 2026

The recipe above is from 2021. Two changes since then show where the idea has gone, as of October 2026.

Rate each pair on its own. The softmax in CLIP’s loss compares every caption in a row with every other, so the whole batch has to be gathered in one place. SigLIP (Zhai et al., 2023) from Google replaces the softmax with a sigmoidSigmoidThe S-shaped function 1 / (1 + e^-z), which squashes any number into the range 0 to 1 so it can be read as a probability.Taught in Classification and probabilityOpen in glossary and makes each image and caption pair, matching or not, its own yes or no question,

where $z_{ij}$ is $1$ for a true pair and $-1$ otherwise, $t$ is a learned scale (the inverse of a temperature), and $b$ is a learned bias. A batch of $N$ pairs holds $N$ matching pairs and $N^2 - N$ mismatched ones, so the authors start $b$ at $-10$: the model begins by answering “no” to every pair, which is right for almost all of them, instead of taking large early steps to learn that. They found the sigmoid loss “performs significantly better than the softmax loss when the batch size is smaller than 16 k”, with the gap closing at larger batches. SigLIP 2 (Tschannen et al., February 2025) keeps this loss and adds training on captions, self-supervised losses, and many languages; its encoders, from 86 million to 1 billion parameters, read pictures inside other models, such as BAGEL (Deng et al., 2025).

import numpy as np

def unit(v):
    return v / np.linalg.norm(v, axis=1, keepdims=True)

def sigmoid_loss(sim, t, b):
    # SigLIP: every image and caption pair is its own yes or no question.
    z = 2 * np.eye(len(sim)) - 1  # 1 for the true pairs, -1 for the rest
    per_pair = np.log1p(np.exp(-z * (t * sim + b)))  # -log sigmoid(z (t s + b))
    return per_pair, per_pair.sum() / len(sim)

# The same three images and captions as in the contrastive loss code.
images = unit(np.array([[1.0, 0.2], [-0.3, 1.0], [-0.7, -0.8]]))
captions = unit(np.array([[0.9, -0.1], [0.1, 1.0], [-0.9, -0.5]]))
sim = images @ captions.T

per_pair, loss = sigmoid_loss(sim, t=10.0, b=-10.0)  # SigLIP's starting values
print("loss at the start:", round(loss, 3))
for row in per_pair:
    print("  ".join(f"{v:6.3f}" for v in row))

# Move caption 3 next to image 1. Only the three terms that involve caption 3
# change. In the softmax loss every image's term would change, because
# caption 3 is in every row's sum.
captions[2] = unit(np.array([[1.0, 0.1]]))[0]
changed, _ = sigmoid_loss(images @ captions.T, t=10.0, b=-10.0)
rows, cols = np.nonzero(~np.isclose(changed, per_pair))
print("terms that changed (image, caption):", [(int(i) + 1, int(j) + 1) for i, j in zip(rows, cols)])

loss at the start: 1.042
 0.956   0.001   0.000
 0.000   1.140   0.000
 0.000   0.000   1.030
terms that changed (image, caption): [(1, 3), (2, 3), (3, 3)]

One model for every kind of input. CLIP has two separate encoders. Newer embedding models start from one large model that already reads several kinds of data and train it with the contrastive loss of this lesson. Alibaba’s Qwen3-VL-Embedding (Li et al., January 2026) starts from the Qwen3-VL vision-language model; a text, a picture, a screenshot, a video, or a mix of them goes in, and the model’s final hidden state at a token added to the end becomes the vector. It is trained with InfoNCE using the other items in the batch, plus deliberately chosen hard negatives. Google’s Gemini Embedding 2 (March 2026) “maps text, images, videos, audio and documents into a single, unified embedding space”, and its report says it uses “the multimodal capabilities of Gemini” and “large-scale contrastive learning” (Shanbhogue et al., May 2026). Both let you shorten the vectors to save storage: Gemini Embedding 2 from its default of 3,072 numbers down to 128, and Qwen3-VL-Embedding because its training also scores shortened prefixes of each vector. What has not changed is the training signal: which item came with which.