Code Walk F · the stackthe map · open-source frontier

The Frontier Stack
a Claude-class system, from open parts

You have read the seed: micrograd, nanoGPT, one attention block. This is the whole tree. Eight layers stand between that toy GPT and a Claude / Cursor / Grok-class product — and every one of them can now be assembled from open models and open tools. Here is the map, and where each layer hides the gap the company is about.

the jumpnanoGPT → a Claude-class system
assembled fromopen models + open tools
layers8, one map
ties tothe observability endgame

00 The seed, and the tree

The Code Walks so far were the seed: 154 lines of autograd, a small GPT, one attention block, one CUDA kernel. Real, but toy-sized. The question this page answers is the one you actually care about —what stands between that and a system like Claude, Cursor, or Grok?

The honest answer, in 2026, is: eight layers, and none of them are secret anymore. Open models (DeepSeek-V3, Llama 3.1, Qwen 2.5, Mixtral) and open tooling (vLLM, Megatron, TRL, Qdrant) now cover every layer end to end. The gap to the frontier is data, compute, and taste — not hidden algorithms. Click through the stack. Each layer tells you what it is, the open pieces you'd build it from, what the frontier labs do there, and — the part only this site cares about — where the GPU gap hides.

The eight layers between nanoGPT and a frontier product. Click any layer for its open-source build-kit and its thesis tie; press “trace a request” to watch a prompt fall from the agent loop down to the silicon.

Click any layer to open it. Or trace how one prompt flows through the stack.

Read it top to bottom and it's a product falling toward physics: an agent loop (Cursor) calls a retrieval brain (RAG), which calls a serving engine (vLLM), which runs a model (a sparse MoE Transformer), which is ultimately just kernels on silicon. Read it bottom to top and it's how youbuild one: get the architecture right, pretrain it, align it, serve it, give it memory, wrap it in a loop. Either direction, the bottom layer — observability — is the one that can see all the others, and the one nobody has built.

01 The frontier, and its open twin

For almost every closed frontier model there is now an open model within striking distance. This is what makes the whole exercise real: you can hold the actual weights of a near-frontier model and run every layer above yourself.

closed / frontier	open twin	shape	the open report tells you
GPT-4 / GPT-4o (OpenAI)	DeepSeek-V3	671B MoE, 37B active	MoE + MLA + FP8 training, in full detail
Claude (Anthropic)	Llama 3.1 405B	405B dense	the pretraining run, data, and scaling choices
Gemini (Google)	Qwen 2.5 72B	dense + MoE variants	a strong, permissively-licensed full family
o1 / o3 reasoning	DeepSeek-R1	RL-trained reasoner	how test-time reasoning is trained with RL
Grok (xAI)	Mixtral 8×22B	sparse MoE	the open MoE that proved the pattern

“Shape” is the honest column. The frontier's edge is increasingly sparsity done well (mixture-of-experts: hundreds of billions of parameters, only a fraction active per token) plus enormous, clean data — not a mystery architecture. DeepSeek-V3's report is the clearest proof: it reads like a build manual.

02 Three products, built from open parts

The same eight layers, recombined, are three different companies. Here is what each famous productis, in terms of the stack — and the open pieces you'd reach for to build your own.

build “Claude” — a helpful, aligned assistantBase = Llama 3.1 or Qwen 2.5. Post-train with TRL / OpenRLHF: SFT on instructions, then DPO or PPO on preferences, with a constitution-style rubric. Serve on vLLM behind an API. The model is 20% of it; the alignment layer is where the personality and the safety live.

build “Cursor” — a coding agent that knows your repoAny strong open model + a retrieval brain: embed the codebase (bge / e5) intoQdrant / pgvector, retrieve on every query, rerank. Wrap it in an agent loop (tool calls to read/edit files, run tests) over MCP. The weights are commodity; the RAG + agent layers are the product.

build “xAI servers” — the cluster underneathPretrain with Megatron-LM / TorchTitan across a GPU fabric (NVLink inside a node, InfiniBand between nodes), scheduled by Slurm or Kubernetes, checkpointing constantly. This is xAI's Colossus (100k+ H100s) in open form. The bottleneck isn't code — it's keeping every GPU fed, which is exactly what nobody can measure well.

Want to see layers 07 and 05 for real, not as a diagram? Therequest-lifecycle example animates exactly what crosses the wire when a coding agent (Claude Code) runs a turn — the stateless resend, prompt caching, and the client-side tool loop that people mistake for server-side RAG.

03 What changed recently — the research that matters

“Attention Is All You Need” is 2017. The frontier moved. The upgrades below are the difference between nanoGPT and a 2026 model — and most arrived in the last two years, in the open.

upgrade	replaces	why it won	in
RoPE	learned position embeddings	relative positions, extends to longer context	Llama, Qwen, DeepSeek
RMSNorm	LayerNorm	cheaper, no mean-centering, just as stable	~every open model
SwiGLU	ReLU/GELU FFN	a gated FFN that simply learns better	Llama, PaLM, DeepSeek
GQA / MLA	full multi-head attention	shrinks the KV cache → long context gets affordable	Llama 3, DeepSeek-V3
Mixture-of-Experts	dense FFN	10× params, ~same FLOPs per token	Mixtral, DeepSeek-V3, Grok
FlashAttention-3	naive attention	IO-aware; more FLOPs, far less memory traffic	every serious kernel
Speculative decoding	one-token-at-a-time	a small model drafts, the big one verifies → 2–3× faster	vLLM, TensorRT-LLM
RL reasoning	plain next-token	test-time “thinking” trained by RL (o1 / R1)	DeepSeek-R1
SSMs (Mamba)	attention itself	linear-time sequence mixing; the main challenger	Mamba, hybrids

Notice the pattern in the “why it won” column: almost every upgrade is about memory and utilization, not raw math. GQA/MLA exist to shrink a cache. FlashAttention exists to stop touching slow memory. MoE exists to spend parameters without spending FLOPs. The whole frontier is quietly a systems story — which is the door the company walks through.

The first five rows are layer 02 of the stack, and they now have their own deep-dive with three interactive pieces — the block diff, the KV cache that GQA shrinks, and the MoE router where “utilization” breaks: Modern architecture — the 2017 block, upgraded →The next rows — FlashAttention and speculative decoding — are the serving layer, and it has its own deep-dive too, built around the goodput-vs-utilization gap that is the company in one gauge:Inference & serving — where the GPU lies loudest →

04 Why this whole map ends at the bottom layer

the plasmient thread

Walk back up the stack and count where GPUs burn: pretraining (thousands of them, for weeks), RL rollouts, embedding a codebase, serving millions of requests. Every one of those runs on silicon, and every one of them is measured today by the same misleading number — nvidia-smi says 100% while the achieved fraction of peak FLOPs is often half that or less.

The MoE layer makes it worse, not better: “utilization” of an expert that fires on 3% of tokens is a meaningless number. Speculative decoding, paged attention, RL's alternating phases — every modern trick widens the gap between “busy” and “useful,” and none of the tools in the stack above can see across the layers to find it.

That is the whole map's punchline. You can now build every layer from open parts — except the one that tells you the truth about all of them. That layer isPlasmient, and it now has its own deep-dive — the vertical trace that joins kernel, op, model, and cluster on one clock: Observability — the layer that measures the other seven → Or start where it started: the attention deep-diveand the FlashAttention walk, the two operations where the gap is widest.

↗ Full references — go to the source

The frontier is more open than it looks. These are the papers and repos that let you build each layer yourself — start with DeepSeek-V3, the report that reads like a recipe.