Code Walk F · layer 04deep dive · post-training

Alignment
the RL loop that runs two GPUs at once

Layer 04 of the Frontier Stack, gone deep. A pretrained model can finish your sentence; it can’t follow your instructions. Alignment is the path from raw next-token predictor to an assistant that’s helpful and safe — SFT, a learned reward model, then reinforcement learning against it (or DPO, the shortcut that skips the loop). It’s also home to the single sharpest illustration of the whole company thesis: RL training oscillates between a memory-bound generation phase and a compute-bound update phase, spends most of its clock in the memory-bound one, and reports a flat 100% utilization the entire time. Two interactive pieces.

the goalnext-token predictor → helpful assistant
the pathSFT → reward model → RLHF · or DPO
the trapRL = generate (memory) + update (compute)
the tellone flat util line over two machines

00 From predictor to assistant

Pretraining leaves you with a model that knows language but not manners: prompt it with a question and it might continue with more questions, because “what token comes next on the web” is not “answer this helpfully.” Alignment (post-training) closes that gap in stages — first imitate good answers (SFT), then learn what humans prefer (a reward model), then optimize against that preference with reinforcement learning. DPO is the newer shortcut that collapses the last two into one. Click through the path — watch the GPU-character tag on each stage:

The post-training pipeline, from the pretrained base to an aligned assistant. Each stage’s tag is its GPU character: SFT and reward modelling are ordinary compute-bound backprop, but RLHF/PPO is split — it generates and it updates. DPO’s whole appeal is that it deletes the generation loop and stays compute-bound, which is why it’s become the default for open models.

Notice the tags line up into a story. Everything except the RL stage is a plain compute-bound training loop — the well-behaved kind from Walk 02. RLHF is the odd one out, taggedgen + update, because it does two fundamentally different things in alternation. That split is the most GPU-revealing thing in all of post-training, and it’s worth watching move.

01 The trap: one run, two opposite machines

Here is the sharpest version of the company thesis on the whole site. An RL fine-tuning step is a loop: the policy model generates a batch of responses (autoregressive decode — one token at a time,memory-bound, tensor cores mostly idle), a reward model scores them, then an optimizerupdates the policy (a big backward pass — compute-bound, cores saturated). Generation is slow and dominates the wall-clock. So the run spends most of its time memory-bound — andnvidia-smi reads ~100% through every phase. Change the rollout-to-update ratio and watch the achieved line crater:

One RL iteration, left to right: a long memory-bound generate phase (~13% of peak), a short reward-model score, then a compute-bound update (~68% of peak). The dashed red line is nvidia-smi — flat at ~97% across all three. The grey line is the cycle-average achieved, dragged down toward generation because generation owns most of the clock. Push the ratio to 20:1, as reasoning-RL runs often do, and the average collapses further.

rollout : update time

nvidia-smi says—

busy in every phase — generation and update look identical

achieved now · generating—

cycle-average achieved — generation dominates the clock

This is the utilization gap you can’t argue with. Two phases, opposite bottlenecks, and the one number the dashboard reports is flat across both — it literally cannot tell you that the GPU spent 80% of the run streaming memory with its tensor cores asleep. It’s exactly the memory-bound decode oflayer 05, now buried inside a training loop where nobody’s watching for it. This is why modern RL stacks (OpenRLHF, veRL) put so much engineering into the rollout: they hand generation to a fast serving engine like vLLM precisely because that phase is where the GPUs are starved.

02 Why an observability company cares

the plasmient thread

Alignment is where the thesis stops being a slogan and becomes a line item. RLHF-style training is how frontier labs turn base models into products, and it burns enormous compute in a loop whose dominant phase — generation — is the most memory-bound, most-starved workload there is. A single averaged utilization number doesn’t just hide the gap here; it hides that the run is two different workloads that need to be measured, and optimized, separately. You cannot fix what you’ve blended into one green bar.

Splitting that bar is Plasmient’s core move: report the generate phase and the update phase as what they are — one memory-bound, one compute-bound — each placed onthe roofline for the card it ran on. The model being aligned islayer 02; the generation phase islayer 05 wearing a training hat; the whole thing scales across the cluster of layer 03. Alignment is where all of it converges — and where measuring the gap is worth the most. Back to the Frontier Stack.

↗ Full references — go to the source

InstructGPT is the canonical recipe; Christiano 2017 is where learning a reward from preferences began; DPO and GRPO are the modern simplifications. TRL and OpenRLHF are the code — note how OpenRLHF and veRL are explicitly architected around the generate/update split this page is about.