Prefill vs decode

Compute-bound then HBM-bound. Same weights, two machines.

play

op · intensityPick an op. Low intensity starves the cores; high intensity leaves the memory lane with slack.

CPU CRUX: prefill I≫295 (compute). Decode I≈1 (HBM). nvidia-smi 100% can still be waiting on HBM.