1 · Architectures — common core, different bets · 17/38

Attention

Tokens mix. QKᵀV. Causal mask.

play
Follow one word

We’ll watch how one word decides what to pay attention to.

softmax(QKᵀ / √d) V

The only place tokens talk. Everything else is per-token.

chapter →Attention & the Transformerwalk →Attention from the paper