Modern architecture
the 2017 block, upgraded

00 Same core, new everything

The attention operation at the heart of a Transformer is exactly what Vaswani et al. published in 2017 — query · key, softmax, weighted sum of values. That part didn’t change. What changed is everything wrapped around it. A 2025 frontier block keeps the same skeleton (attention → add → norm → feed-forward → add → norm) but swaps in a better part at nearly every slot, and the very largest models replace the single feed-forward with a sparse mixture of experts.

Here is the whole diff. Each row is one upgrade the frontier made — the old part struck through, the new part named. Click any row for what changed, and for the point that ties this entire layer together:almost every one of these is a memory or bandwidth win, not an arithmetic one.

Of the five upgrades, RoPE is the one that’s clearest in motion. On the static sketch above it’s just two dials; play it and the whole idea lands — watch position become rotation, and watchrelative position fall out of the dot product on its own:

Two of these are big enough to deserve their own instrument, because they’re where the plasmient thesis lives most sharply: grouped-query attention (a pure inference-memory win) and mixture-of-experts(where “GPU utilization” quietly loses its meaning). The next two sections let you poke each one.

01 The tensor that fills the GPU: the KV cache

At generation time a model doesn’t recompute attention over the whole sequence each step — itcaches the key and value vectors of every past token and reuses them. That KV cache is, physically, what fills the GPU’s memory during inference: one key and one value, per token, per layer, per KV head. It grows with context length and batch size, and when it’s full, you’re done — no more concurrent requests, no more context.

This is why grouped-query and latent attention matter so much. They don’t touch the math of attention; they shrink the cache. Switch the scheme below and watch the bytes-per-token — and the context you can afford in a fixed 40 GB budget — move:

The dramatic part is the context readout. Cutting the cache 4× doesn’t make attention 4× faster — it lets you fit 4× more tokens or 4× more concurrent users on the same card. Decode ismemory-bound: the GPU spends its time streaming that cache in and out, not doing arithmetic. Shrink the cache and you turn a half-starved serving node into a full one. This is the utilization gap, made of one tensor.

02 Where “utilization” stops meaning anything: MoE

The other big change is structural. Instead of one large feed-forward that every token passes through, a mixture-of-experts model has many expert feed-forwards plus a router that sends each token to only its top-k. Total parameters explode — DeepSeek-V3 has 671 billion — while the compute per token stays tiny, because only k of N experts ever fire. You get a giant model’s knowledge at a small model’s FLOPs.

Watch a stream of tokens route. Turn the expert count up to 256 and keep top-k at 2, and look at the two gauges: the GPU is genuinely busy on the experts it touches — so nvidia-smi reads nearly full — while the fraction of the model actually doing work is a rounding error:

That gap is not a bug — sparsity is the point, and it’s why open models caught the frontier. But itbreaks the metric. If a token touches 2 of 256 experts, what is “GPU utilization”? An expert that barely fires is capacity you paid for and left idle; a hot, overloaded expert is a bottleneck throttling the batch. A single averaged percentage can’t see either. Only a per-expert, per-token view tells the truth — and nothing on the market draws it.

03 The modern block, assembled

Put every upgrade back in place and the frontier decoder block looks like this — the same 2017 skeleton, every slot re-fitted, the feed-forward optionally gone sparse:

          token + RoPE-rotated positions
                        │
      ┌─────────────────▼──────────────────┐
      │  RMSNorm                            │   ← was LayerNorm
      │  Grouped-Query / Latent Attention   │   ← was full multi-head  (shrinks the KV cache)
      └─────────────────┬──────────────────┘
                        ▼   ⊕ residual
      ┌─────────────────▼──────────────────┐
      │  RMSNorm                            │
      │  ┌───────── router (top-k) ───────┐ │   ← dense FFN became…
      │  │ SwiGLU  SwiGLU  SwiGLU   …×N   │ │   … a sparse Mixture-of-Experts
      │  └────────────────────────────────┘ │      (only k of N fire per token)
      └─────────────────┬──────────────────┘
                        ▼   ⊕ residual
                   next block ( ×  ~ 60–120 )

DeepSeek-V3 is the open model that ships all of it at frontier scale — MLA for the attention, fine-grained MoE (256 experts, 8 active) for the feed-forward, RoPE, RMSNorm, SwiGLU throughout — and its technical report writes the recipe down. Llama 3 and Qwen 2.5 are the dense reference implementations. None of it is secret; the gap to the frontier is data and compute, not architecture.

04 Why an observability company cares