Look inside a 100-dimensional space
Word vectors flattened to their three most informative directions. Drag to turn it. Tap a word to compare its true neighbors with what you see.
Loading 1.1 MB of word vectors...
Variance kept (PC1, PC2, PC3)...
Challenges
0 of 3 complete
- Turn the view by hand until all three of tiger's true neighbors are also its nearest on screen (3 of 3 match). (not yet)
- Find a word whose closest true neighbor belongs to a different group. (not yet)
- Show exactly two groups and find a pair where the first direction, PC1, keeps more than 35% of the variance on its own. (not yet)
Try this
- Select a word and turn the view. Do its on-screen neighbors match its true neighbors?
- Add groups one at a time and watch the share of variance kept.
Principal components in a few lines of codePython with NumPy
import numpy as np
# Six made-up word vectors in 4 dimensions (the demo's GloVe vectors have 100).
words = ["cat", "dog", "tiger", "bread", "rice", "soup"]
X = np.array([
[0.9, 0.1, 0.3, 0.0],
[0.9, 0.1, -0.3, 0.0],
[0.7, 0.0, 0.3, 0.0],
[0.1, 0.9, 0.1, 0.6],
[0.0, 0.8, 0.1, -0.6],
[0.1, 0.9, 0.1, 0.0],
])
X = X / np.linalg.norm(X, axis=1, keepdims=True) # unit length, so dot = cosine
# PCA: center the cloud; the right singular vectors are the principal directions.
centered = X - X.mean(axis=0)
_, s, directions = np.linalg.svd(centered, full_matrices=False)
share = s**2 / np.sum(s**2)
print("variance per direction:", np.round(share, 2))
print(f"a 2D map keeps {share[:2].sum():.0%}")
flat = centered @ directions[:2].T # each word's position on the 2D map
cos = X @ X[0] # cosine of every word with "cat"
on_map = np.linalg.norm(flat - flat[0], axis=1)
cos[0], on_map[0] = -np.inf, np.inf # skip cat itself
print("nearest to cat in 4D:", words[np.argmax(cos)])
print("nearest to cat on the map:", words[np.argmin(on_map)])variance per direction: [0.68 0.21 0.1 0.01]
a 2D map keeps 89%
nearest to cat in 4D: tiger
nearest to cat on the map: dog