Reading an embedding

In the last two lessons, a sentence embedding model found the right documents for a question and fed them to a language model. It works, but it is easy to treat the vector in the middle as a black box. What is actually in those 384 numbers? Can you read one?

Not directly. But you can ask good questions of it, and every question you can ask has a blind spot worth knowing.

384 numbers with no labels

The model in this lab, all-MiniLM-L6-v2, reads your text, produces one vector per token, and averages them into a single vector. That last step is called mean poolingMean poolingTurning a model's per-token vectors into one vector for the whole text by averaging them, number by number. Many sentence embedding models, including all-MiniLM-L6-v2, work this way.Open in glossary. The average is then scaled to length 1, so every text, from one letter to a paragraph, ends up as 384 numbers of the same overall size.

Each number is small: almost all fall between -0.2 and +0.2. None of them means “money” or “sadness”. Nobody chose what the positions mean, and training never asked any single position to mean anything. Ideas are spread across many numbers at once, and each number takes part in many ideas, which is called a distributed representationDistributed representationRepresenting each idea as a pattern spread across many numbers, with every number taking part in many ideas, instead of giving each idea a number of its own.Open in glossary.

So instead of reading the numbers, you compare the vector with vectors you do understand. Embed a list of plain words, such as “joy”, “market”, and “farmer”, and measure the cosine similarityCosine similarityA measure of how closely two vectors point in the same direction, from -1 (opposite) through 0 (perpendicular) to 1 (same direction), ignoring their lengths.Taught in Vectors and similarityOpen in glossary between your vector and each of them. That list is a probe: a set of reference points you chose. It is not a list of concepts stored inside the model.

The workbench

The workbench below embeds 403 reference words in ten categories, then gives you five tools. Encode shows the raw numbers. Decode ranks the reference words by similarity. Arithmetic adds and subtracts vectors. Diff compares two vectors. Map flattens your vectors and some reference words onto a page.

import numpy as np

# Made-up 4-number vectors for six concepts. Real ones have 384 numbers
# that no one chose, but the operations are exactly the same.
words = ["king", "queen", "man", "woman", "palace", "dog"]
V = np.array([[0.9, 0.8, 0.1, 0.4],
              [0.9, 0.1, 0.8, 0.4],
              [0.1, 0.9, 0.1, 0.6],
              [0.1, 0.1, 0.9, 0.6],
              [0.8, 0.3, 0.3, 0.0],
              [0.0, 0.4, 0.3, 0.9]])
V = V / np.linalg.norm(V, axis=1, keepdims=True)  # length 1, like MiniLM

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

def decode(v, k=3, leave_out=()):
    sims = V @ (v / np.linalg.norm(v))  # cosine with every concept
    order = [i for i in np.argsort(-sims) if words[i] not in leave_out]
    return [(words[i], round(float(sims[i]), 3)) for i in order[:k]]

vec = dict(zip(words, V))
print("cos(king, queen):", round(float(cosine(vec["king"], vec["queen"])), 3))

result = vec["king"] - vec["man"] + vec["woman"]
print("king - man + woman:", decode(result))
print("  leaving out inputs:", decode(result, leave_out={"king", "man", "woman"}))
print("king alone:", decode(vec["king"], leave_out={"king", "man", "woman"}))

# Diff: which concepts favor king over queen?
delta = V @ vec["king"] - V @ vec["queen"]
print("favors king:", words[int(np.argmax(delta))], "| favors queen:", words[int(np.argmin(delta))])

cos(king, queen): 0.698
king - man + woman: [('queen', 0.977), ('woman', 0.797), ('palace', 0.763)]
  leaving out inputs: [('queen', 0.977), ('palace', 0.763), ('dog', 0.418)]
king alone: [('palace', 0.859), ('queen', 0.698), ('dog', 0.542)]
favors king: man | favors queen: woman

What decoding shows

“I can’t stop smiling today” lands nearest to happiness (0.49) and joy (0.44). No word in the sentence is a feeling word; the model placed it there from the situation. That is the point of a sentence embedding: it encodes what a text is about, not just which words it contains.

Read every score against a baseline. Similarities between this model’s vectors are almost always positive, so 0.3 is not “30 percent related”. For “The stock market crashed today”, the median over all 403 reference words is only about 0.07, which is why market (0.40) and panic (0.36) stand out. The thin mark on each bar in Decode shows that median.

Arithmetic, tested fairly

The word embeddings lesson showed that king − man + woman lands near queen with GloVe vectors. Sentence embeddings are trained differently, to pull together texts that mean the same thing, so there is less reason to expect clean directions between related words. Check before you believe.

With the inputs left out, vector arithmeticVector arithmeticAdding, subtracting, and scaling embeddings, as in king - man + woman, to test whether a relationship between meanings shows up as a consistent direction.Taught in Word embeddingsOpen in glossary gives king − man + woman ≈ queen (0.58). Turn the exclusion off and two of the inputs, king and woman (0.64 each), outrank queen. And king alone, inputs left out, is already nearest to kingdom and then queen. The arithmetic nudged queen up one place; it did not discover it. In 2016 Tal Linzen showed the same pattern in standard analogy tests for word vectors: often the answer is already a near neighbor of one of the words.

Some examples hold up better. Doctor alone is nearest to medicine, but doctor − hospital + school gives teacher (0.60): swapping the workplace really did swap the job. Chef − restaurant + farm gives farmer, and breakfast − morning + evening gives dinner.

Paris − france + japan looks like a failure: the nearest reference word is just “city”. But Tokyo is not on the list, so it could never appear. Save the result and compare it with “tokyo” in Diff: 0.84, closer than “japan” itself is to “tokyo”. The arithmetic worked; the probe could not show it. A real failure is the famous example done with whole sentences: “The king ruled the country” − “The man walked home” + “The woman walked home” stays about countries and kings.

What the vectors blur

Diff shows how close two vectors are and which reference words favor each side. Try a sentence and its negation:

One word reverses the meaning, and the vectors barely move. Swapping who bit whom moves them even less. Averaging token vectors means two sentences built from the same words start with much in common, and the training pairs taught the model which texts are about the same thing; a sentence and its negation are. This is a real limit for search: a query can retrieve passages that say the opposite of what it asks.

The Map uses principal component analysisPrincipal component analysisA method that finds the directions along which data varies the most, often used to project many-dimensional data down to two or three dimensions for viewing.Taught in Seeing high dimensionsOpen in glossary from Seeing high dimensions. With a few categories on it, the flat picture usually keeps well under a fifth of the variation, so treat it as a sketch and confirm any pair with Diff.

Asking a language model to name the theme

Under Decode you can download a second, optional model: SmolLM2-135M, the small base modelBase modelA language model after pretraining only. It continues text in whatever style the input suggests, rather than following instructions or answering like an assistant.Taught in Inside a real language modelOpen in glossary from the transformers track. It gets a prompt with five worked examples, each a list of words followed by the theme they share, and then your nearest words. This is few-shot promptingFew-shot promptingPutting a few worked examples of a task in a language model's prompt, so it continues the pattern for a new case without any further training.Open in glossary. The workbench shows the model’s probabilities for the first token of its answer and the answer it gives by always taking the most likely token, which is called greedy decodingGreedy decodingGenerating text by always choosing the single most probable next token. Equivalent to sampling at temperature 0.Taught in Predicting the next tokenOpen in glossary.

We tried the prompt on 100 lists of nearest words. About 9 times in 10 its answer was a sensible theme for the list, and more than half the time it simply repeated one of the listed words. Note what it works from: only the list. It never sees your vector or your text, so it can give a fine theme for an odd list and still miss what the sentence was about. A much larger model would name themes more gracefully, but the same caution holds: it would be telling a story about the probe’s output, not reading the embedding.