Scaling
one model across a thousand GPUs

00 Four ways to split a model

A frontier model overflows a single GPU in every dimension: the parameters don’t fit, theactivations for a full batch don’t fit, and even if they did, one GPU would take years. So the model is cut apart along four independent axes, usually all at once. Split the batch across replicas (data), split each layer’s matrices (tensor), split the stack of layers into stages (pipeline), or split an MoE’s experts (expert). Each axis solves a different “doesn’t fit”, and each one adds a different kind of network traffic. Click through them:

The pattern to carry forward: every axis trades compute capacity for communication. Data parallelism is cheapest to reason about but all-reduces the entire gradient each step; tensor parallelism has the heaviest traffic and must stay inside a node over NVLink; expert parallelism’s all-to-all depends on a router that’s never quite balanced. None of that traffic does any floating-point work — and all of it registers as a busy GPU.

Data parallelism is the one every run uses, and its entire cost is a single collective: theall-reduce that sums every replica’s gradient before the next step. It’s worth seeing what that collective actually is, because the name hides the mechanics. The standard ring all-reduce — what NCCL runs under the hood — does it in two halves, each N−1 rounds of send-to-your-neighbour. Firstscatter-reduce: every GPU keeps passing a chunk clockwise and adding it in, until each GPU owns one fully-summed chunk. Then all-gather: those finished chunks walk back around the ring until every GPU has all of them. It’s clever and bandwidth-optimal — and it is pure communication.

01 The purest bubble: pipeline parallelism

Pipeline parallelism gives the cleanest picture of the whole problem. Put the first layers on GPU 0, the next on GPU 1, and so on: an assembly line. Feed one batch in and only one GPU works at a time — the rest wait their turn. The fix is to chop the batch into microbatches so several are in flight at once, like parts moving down the line. But the line still has to fill at the start anddrain at the end, and during those the early and late stages sit idle. That idle wedge is thebubble. Change the microbatch count and the stage count and watch it breathe:

This is the thesis with no room to hide. A GPU stalled in the bubble is doing literally zerofloating-point work, yet the process holds the device, so utilization reads high. The efficiency gauge — the honest one — falls straight out of the geometry: with few microbatches or many stages, a third or more of your cluster-time can be bubble. Real schedules (1F1B, interleaved / virtual pipelining) pack the forward and backward passes tighter to shrink it, but they never reach zero, and this widget only shows the forward half — the real picture doubles it.

02 The one number that survives: MFU

Stack it all up — all-reduce for data parallelism, mid-layer collectives for tensor parallelism, the pipeline bubble, all-to-all for experts, plus kernels that were already memory-bound before any of this — and you get the real efficiency of a training run. The field has a name for the honest version:MFU, model FLOPs utilization — the actual useful FLOPs done, divided by what the hardware could do at peak. It’s the same idea Karpathy’s estimate_mfu()computes for a single GPU, lifted to the whole cluster.

  what the dashboard shows            what MFU shows
  ┌──────────────────────────┐        ┌──────────────────────────┐
  │ 16,384 GPUs   ~100% busy  │        │ useful FLOPs / peak       │
  │ ████████████████████████  │   vs   │ ██████████░░░░░░░░░░░░░░░  │
  └──────────────────────────┘        └──────────────────────────┘
     one green number                    PaLM: 46% · GPT-class: 30–50%
     hides the collectives,              the collectives, the bubble,
     the bubble, the stalls              and the memory walls, counted

Published frontier runs report MFU in the 30–50% range — PaLM’s own paper states 46.2% across 6144 chips, and it was celebrated as excellent. Read that plainly: at the frontier of engineering, half the most expensive compute humanity has ever assembled is going into the tax, and the dashboard everyone watches shows 100%.

03 Why an observability company cares