How positional encoding puts order back in

Part eight ended on attention being permutation equivariant. Shuffle the inputs and the outputs shuffle identically; not one value changes. Attention has no “before” and no “after”.

In a sentence, order is meaning. So order has to be supplied separately. This part is about how.

Why the naive way fails

The first idea is to write the position number in as one more dimension: append i to token i’s vector.

It breaks immediately. An attention logit is a dot product, and a dot product carries the i x j term straight through.

positions 0-5     content spread 3.30   position spread  11.18   max prob 0.626
positions 100-105 content spread 3.30   position spread 458.39   max prob 1.000

Content moves the logits by 3.30 while position moves them by 458, and the ratio only widens further into the sentence. Part eight showed that large logits saturate the softmax; here the largest probability is 1.000. Attention stops looking at content at all.

That gives the requirement. Position information has to differ from place to place without growing as the place gets later.

Sinusoids

The transformer’s answer is periodic functions. Dimensions are taken in pairs, each pair carrying a sine and cosine at its own frequency.

def PE(L, d):
    pos = np.arange(L)[:, None]
    i = np.arange(0, d, 2)[None, :]
    w = 1.0 / (10000 ** (i / d))          # frequency, different per dimension
    P = np.zeros((L, d))
    P[:, 0::2] = np.sin(pos * w)
    P[:, 1::2] = np.cos(pos * w)
    return P

Take the guarantees one at a time. Everything below uses d=64 and 512 positions.

First, every position has the same size. Each frequency contributes sin² + cos² = 1, so the norm is sqrt(d/2) no matter the position. Measured, that is 5.6569, varying by 8.9e-16 across all 512 positions. This is exactly the condition the naive method broke.

Second, the dot product of two positions depends only on their distance. PE(i)·PE(j) does not see i and j separately, only i-j. Measuring the spread of values along each diagonal gives 1.8e-13, floating-point noise. The angle-addition identity explains it in one line.

PE(i)·PE(j) = Σ [sin(i·w) sin(j·w) + cos(i·w) cos(j·w)] = Σ cos((i-j)·w)

Comparing against Σ cos(distance · w) directly, the largest difference is 7.1e-15. Absolute positions go in and relative position comes out. That is the real reason for choosing sinusoids.

Similarity does not decay with distance

The usual next step is one claim too far: “so the dot product shrinks with distance, and nearby tokens end up more alike”. Measured, it does not.

At short range it holds - 32.00, 30.92, 28.30, 25.59, 23.93, 23.50. Then at distance 6 it climbs back to 23.56. Of the 511 steps, 244 go up rather than down. The minimum is 2.56 at distance 406, and by distance 511 the value has risen again to 6.42.

Which is what a sum of periodic functions does. It oscillates; it is not a decaying function. What sinusoidal encoding buys is not “closer means more alike” but the fact of being a function of distance at all. What that function looks like against distance is a separate question, and as you can see, it is not pretty.

The real payoff: shifting is a linear map

The value of choosing periodic functions lies elsewhere. Fix any gap k and there is a single fixed matrix M_k, the same one for every position, with PE(pos + k) = M_k · PE(pos). Not a different matrix per position - one.

Each frequency pair is [sin(pos·w), cos(pos·w)], so shifting by k is a rotation through the angle k·w, and a rotation matrix does not depend on pos.

Check it by measurement. Fitting 512 positions with 64 dimensions leaves eight times more equations than unknowns, so this cannot come out right by accident.

   k     sinusoidal    random embedding
   1        4.9e-14              0.504
   2        5.0e-14              0.489
   5        4.8e-14              0.548
  17        3.9e-14              0.501
  50        1.6e-13              0.549

The sinusoidal residual is floating-point noise; a random embedding of the same size sits around 0.5 - a relative error of 0.898, which is to say it fits nothing at all.

The difference is practical. For attention to key on a relation like “three places back”, the W that builds Q and K has to be able to express it, and what a W does is a linear transform. With sinusoids that relation already exists as a linear map, so it only has to be found. Handing each position its own arbitrary vector leaves no such map to find.

None of this makes learned positional embeddings unusable. They are widely used and they learn a vector per position from data. The difference is that this structure has to be learned rather than had for free.

A bonus: beyond the trained length

A sinusoid is a formula, so it has a value at any position. Train on 512 and position 4000 still has norm 5.6569, and the dot product of 4000 with 4001 is 30.917 - exactly the value for positions 0 and 1. A learned embedding has nothing to hand back for a row that was never in its table.

Having a value and working well there are different things, of course. Models do degrade past their trained length, which is why current ones reach for other schemes such as rotary encodings. What sinusoids guarantee stops at “defined”.

It costs something

Not free. Position is added, and what it adds it also blurs. Adding positional encoding to part eight’s six tokens changes the first row like this.

              t0     t1     t2     t3     t4     t5
no position  0.510  0.030  0.015  0.389  0.039  0.016
with         0.610  0.213  0.020  0.140  0.012  0.004

t0’s topic partner is t3, and its share drops from 0.389 to 0.140, while the immediate neighbour t1 climbs from 0.030 to 0.213. Position competes with content.

Position won here, and what settles that contest is the ratio between the embedding’s scale and the encoding’s. Real models scale the embedding by sqrt(d) before adding, which tunes exactly that ratio.

The goal is met all the same. Measure permutation equivariance again and it is broken.

Z  = X + PE(6, d)          # position added in the original order
Zp = X[perm] + PE(6, d)    # tokens shuffled, positions left where they are

np.allclose(att(X[perm]), att(X)[perm])    # no position:   True
np.allclose(att(Zp),      att(Z)[perm])    # with position: False

The point is that PE is not shuffled when building Zp. Shuffle the positions along with the tokens and att(Z[perm]) is equivariant again - carry the seat number around with the passenger and nobody has changed seats.

So

• To give attention order, each place must differ without later places growing larger. Writing positions in as raw integers makes their logit contribution 458 against content’s 3.30 near position 100, burying the content and pinning the largest probability at 1.000
• Sinusoids hold every position’s norm at 5.6569 and make the dot product of two positions a function of distance alone, Σ cos(distance·w). Absolute positions go in, relative position comes out
• That function does not decay monotonically with distance. 244 of 511 steps go up, and the minimum sits at distance 406
• The real payoff is that shifting is a fixed linear map: residual 5e-14 against 0.5 for a random embedding. It leaves relative position in a form a W merely has to find
• What is added also blurs. The topic partner’s share falls from 0.389 to 0.140

Next time the head gets split in several. If one attention produces one weighted average, the next question is what several of them look at separately.