Sinusoidal position encodings
Each row is the vector added to the token at that position. Each column is a wave; columns to the right oscillate more slowly.
How similar is position 12 to every other position? (dot product, scaled so a position with itself scores 1)
Challenges
0 of 2 complete
- At d_model 64, find the first dimension whose wave takes more than 1,000 positions to repeat. (not yet)
- Find a d_model at which dimension 15 repeats only every 10,000 positions or more. (not yet)
Try this
- Tap anywhere on the heatmap to pick a position and a dimension at once.
- Compare the similarity curve at d_model 16 and 128.
Rotary position embedding (RoPE)
RoPE rotates each pair of query and key dimensions by an angle proportional to the token's position.
Dashed arrows are this pair of dimensions before rotation. The query turns by m × θ = 4.00 radians and the key by n × θ = 1.00. The angle between them changes by (m − n) × θ, so the dot product depends only on how far apart the two tokens are.
Press Both +1 or Both +5 to move both tokens later in the text. Both arrows turn, but neither dot product changes. Pair 0 spins fastest (one radian per position); higher pairs turn so slowly that they encode long distances.
Challenges
0 of 2 complete
- Move the query to position 20 or later and get exactly the same full q · k as at the start. (not yet)
- With the query at position 40, find the lowest dimension pair that turns it by less than one radian in total. (not yet)
Try this
- Set the query and key to the same position. The dot product is the same as with no rotation at all.
- Find a pair where shifting both positions by 5 visibly turns the arrows, and confirm the dot product stays put.
Sinusoidal encodings and RoPE in a few linesPython with NumPy
import numpy as np
def sinusoidal(pos, d_model):
i = np.arange(d_model // 2)
angle = pos / 10000 ** (2 * i / d_model)
pe = np.empty(d_model)
pe[0::2], pe[1::2] = np.sin(angle), np.cos(angle) # one sine and cosine per pair
return pe
def rope(x, pos, base=10000):
d = len(x)
theta = base ** (-2 * np.arange(d // 2) / d)
c, s = np.cos(pos * theta), np.sin(pos * theta)
out = np.empty(d)
out[0::2] = x[0::2] * c - x[1::2] * s # rotate each pair of dimensions
out[1::2] = x[0::2] * s + x[1::2] * c
return out
# Sinusoidal: similarity (dot product / 32, as in the demo) depends only on distance.
for p, q in [(12, 13), (30, 31), (12, 20)]:
print(f"similarity of {p} and {q}: {sinusoidal(p, 64) @ sinusoidal(q, 64) / 32:.3f}")
# RoPE: rotate a query and a key by their positions, then take the dot product.
q = np.array([1.0, 0.5, -0.3, 0.8])
k = np.array([0.2, 1.0, 0.7, -0.4])
for m, n in [(4, 1), (24, 21), (5, 1)]:
print(f"query at {m}, key at {n}: score {rope(q, m) @ rope(k, n):.3f}")similarity of 12 and 13: 0.966
similarity of 30 and 31: 0.966
similarity of 12 and 20: 0.700
query at 4, key at 1: score -1.109
query at 24, key at 21: score -1.109
query at 5, key at 1: score -1.686