Attention, visualized
straight from the paper

00 Watch it happen first

Before a single formula, watch the whole thing move. Below, one word — the verb “chased” — works out which earlier words matter to it. Press play and read the line under the stage; each scene is one small step. Scrub back and forth, or click a chapter dot to jump. Nothing here needs to make sense yet — the point is to see the shape of the computation before we name its parts.

That's the entire mechanism. Said in one sentence: attention is a soft dictionary lookup. Each token emits a query — “what am I looking for?” — and every token also advertises a key— “here's what I am.” Match every query against every key, turn the matches into weights that sum to 1, and use those weights to average the tokens' values. A token's new representation is a blend of the tokens it found relevant. Everything below is that one sentence, drawn at deeper and deeper zoom — exactly the two figures the 2017 paper is famous for.

01 Scaled dot-product attention — Figure 2, left

The paper writes the whole operation as one line:

Attention(Q, K, V) = softmax( Q·Kᵀ / √dₖ ) · V

Read it right to left through the pipeline and nothing is mysterious:

• Q·Kᵀ — every query dotted with every key. For T tokens that's a T×T grid of raw match scores. High score = “this key answers my query.”
• ÷ √dₖ — the scaled part. Dot products of dₖ-dimensional vectors grow like √dₖ; left unscaled they'd push softmax into a corner where its gradient vanishes. Dividing by √dₖ keeps the scores in a sane range. (With dₖ = 64, that's ÷8.)
• mask — in a decoder (GPT), a token may not see the future. Positions ahead are set to −∞ so softmax gives them exactly zero weight.
• softmax — each row becomes a probability distribution: how much this query attends to each key, summing to 1.
• · V — those weights average the value vectors. The output for each token is a weighted blend of the values it attended to.

The animation up top drove itself. This one you drive — same operation, new sentence, now a sandbox. Pick a word on the left to make it the query. Switch heads. Toggle the causal mask. Presssweep to watch every token take its turn. The connection lines, the matrix, and the arithmetic strip are all the same numbers, shown three ways:

Two things are worth pausing on. First, the matrix is lower-triangular once the mask is on — that wedge of zeros in the top-right is causality, the reason a language model can't cheat by peeking ahead. Second, on Head 2 the query “it” puts almost all its weight on “cat”: the model has, in this head, learned to resolve the pronoun. Different heads, different jobs — which is the whole point of the next level.

02 Multi-head — Figure 2, right

One attention is one point of view. If a single head has to track grammar and coreference andlocal phrasing all at once, those signals collide. So the Transformer runs h heads in parallel, each with its own small learned projections of Q, K and V into a d/h-dimensional subspace:

MultiHead(Q,K,V) = Concat(head₁, …, head_h) · Wᴼ
        headᵢ = Attention(Q·Wᵢᵠ, K·Wᵢᴷ, V·Wᵢⱽ)

Each headᵢ is exactly the scaled dot-product attention you just used — just on its own slice of the space, free to specialise. In the widget above, “local” and “reference” are two such heads: one wires each word to its predecessor, the other wires “it” to “cat.” Their outputs are concatenatedback to width d and passed through one more linear layer Wᴼ. Same cost as one big head, many more relationships captured.

03 The block, and the stack — Figure 1

Attention is one sub-layer. Wrap it the way the paper does — a residual add and aLayerNorm around it, then a position-wise feed-forward network wrapped the same way — and you have one Transformer block. Stack N = 12 (or 96) of them, put an embedding at the bottom and a softmax head at the top, and that's the model. Click through it:

The vertical line is the residual stream — the paper's quiet hero. Each sub-layer computes a small update and adds it; it never overwrites. That's why 12, 96, 175 layers deep still trains: the gradient always has a clean path straight down the skip connections, and every layer only has to learn anudge. Attention moves information between tokens; the feed-forward network then does the per-token processing. Alternating those two is the rhythm of the whole architecture.

04 Where the endgame lives