1 · Architectures — common core, different bets · 20/38
RoPE, RMSNorm, SwiGLU
The 2024–26 decoder defaults.
playrelative distance d = j − i+3Play, then watch the right panel: both tokens spin, but the gap — and the score — never move.
relative distance d—
the only thing the dot product can see
attention score ⟨q·k⟩—
a function of d alone — absolute position i, j cancel
Position is a rotation of Q and K, not a vector you add.