Inference & serving
where the GPU lies loudest

00 Two phases, two very different machines

Generating from an LLM is really two workloads wearing one costume. First prefill: the whole prompt goes through in a single big pass — a fat matmul that saturates the tensor cores. It’scompute-bound, exactly the work a GPU is built for. Then decode: one token at a time, each step reading the entire model’s weights and the whole KV cache to produce a single token. That’smemory-bound — the tensor cores sit mostly idle while the chip streams gigabytes in and out.

This split is the root of everything on this page. Decode is where models spend most of their time, and decode is where the silicon is starved. Every serving technique below exists to claw back the utilization that memory-bound decode leaves on the floor — and none of it shows up on the one number the dashboards report.

01 Keeping the slots full: continuous batching

A single decode step barely uses the GPU, so you run many requests at once — a batch. The catch: requests are different lengths and finish at different times. With old static batching the whole batch waits for its slowest member; short requests finish early and their slots sit idle. The GPU still reads busy. Continuous batching (a.k.a. iteration-level / in-flight batching) refills a slot the instant its request finishes. Flip between them and watch the goodput gauge:

That is the thesis in one widget. The left gauge — what a monitoring dashboard shows — barely moves between the two modes. The right gauge — what’s actually being accomplished — is the entire difference between a serving node that pays for itself and one that burns half its capacity waiting. Continuous batching (from Orca, now standard in vLLM and every serious server) is the single biggest throughput win in inference, and it’s invisible to utilization.

02 Fitting more requests: PagedAttention

Batching only helps if the requests fit. The limit is the KV cache (the tensor you shrank with grouped-query attention over in layer 02). Classic serving reserved one big contiguous block of memory per request, sized for the longest output it mightproduce — so most of it went unused, and fragmentation wasted the rest. Real GPUs ran at a fraction of their possible batch size.

PagedAttention (the idea that launched vLLM) borrows the oldest trick in operating systems:virtual memory. Store the KV cache in small fixed-size pages, allocated on demand and addressed through a page table. No giant reservation, almost no fragmentation — so you fit far more concurrent sequences on the same card, which makes the batching above actually pay off.

  reserved (old way)                     paged (PagedAttention)
  ┌───────────────────────────┐          ┌──┬──┬──┬──┬──┬──┬──┬──┐
  │ req A ████░░░░░░░░░░░░░░░  │          │A0│A1│B0│C0│A2│B1│C1│░░│   pages handed out
  │ req B ██░░░░░░░░░░░░░░░░░  │   →      └──┴──┴──┴──┴──┴──┴──┴──┘   as tokens arrive
  │ req C █░░░░░░░░░░░░░░░░░░  │          page table maps each seq → its pages
  └───────────────────────────┘          almost no waste, no fragmentation
     reserved-for-max = wasted              → many more requests fit

The diagram above sketches the idea; here it is live. Flip between reserved and paged and watch how many requests fit on the same 64-page pool — and where the memory actually goes.

The payoff compounds with §01: paging lets more requests be resident, continuous batching keeps their slots full, and together they turn the memory-bound decode phase from a capacity disaster into a packed, efficient pipeline. Chunked prefill (Sarathi-Serve) adds one more fix — slicing long prompts so a big prefill doesn’t stall everyone else’s decode.

03 Spending the idle headroom: speculative decoding

Here’s the clever one. Since decode is memory-bound, the target model reads all its weights to produce a single token — with the tensor cores mostly idle. What if we gave them something to do? A small, cheapdraft model proposes several tokens; the big target model then verifies all of them in one pass. If the draft was right, you commit several tokens for the price of one expensive pass. Drag the draft-acceptance slider:

This is the thesis flipped into a feature. Speculative decoding wins precisely because decode under-uses the tensor cores: verifying k candidate tokens costs almost the same as generating one, since both are dominated by reading the weights once. It converts the utilization gap directly into throughput — typically 2–3×. Medusa, EAGLE and n-gram drafting are variations on the same insight.

04 Why an observability company cares