Code Walk F · exampleworked example · agents + serving

Inside a coding agent
the request lifecycle

A worked example for the top of the Frontier Stack: what actually happens when you ask a coding agent to “fix a bug.” No server-side session, no vector database of your repo, no secret router — just a stateless API, a content-addressed cache, and a tool loop that runs on your own machine. Two animations make it concrete.

the surprisethe API is stateless
cachingcontent-addressed prefix, ~5-min TTL
“retrieval”client-side tool loop, not server RAG
routingnone — model is an explicit field

00 The mental model that’s wrong

Almost everyone pictures it like this: “there’s a session on Anthropic’s server; it remembers my conversation; each message continues it.” That is not how it works, and the real picture explains a surprising amount — why long sessions still work, why they get cheaper on repeated turns, why editing an early message can suddenly cost more.

The truth: the Messages API (POST /v1/messages) is stateless. There is no session object the server looks up. Every call — including the 50th message of a long Claude Code session — carries the entire conversation in the request body: every prior message, every tool call, every result. The client keeps the transcript and resends the whole thing each time. Watch it:

One turn of a coding agent, end to end. The left stack is the transcript — it only grows, and the whole thing is re-sent every turn. Notice the grep runs on your machine and never crosses the wire. Press play, then scrub.

The request lifecycle

What actually crosses the wire when you talk to a coding agent.

So the mental model isn’t “the server finds my session and continues it.” It’s “the client keeps a growing transcript locally and re-submits all of it, every call.” The model has no memory between requests; the appearance of memory is entirely the client resending context.

01 Then what is prompt caching?

If the whole transcript is re-sent every turn, wouldn’t that be slow and expensive? It would — except for caching. But caching is not a session lookup either. It’s a content-addressed prefix cache:

The client marks a point in the request — after the system prompt, tools, and earlier turns — with a cache_control breakpoint.
Anthropic hashes the exact, byte-for-byte content up to that point. If a later request has an identical prefix, the cached attention state is reused instead of recomputed.
It’s ephemeral — a ~5-minute TTL (refreshed on each hit; a 1-hour option exists). No ID, no database row — just “have I seen these exact leading tokens recently.”
You can watch it in the response: cache_read_input_tokens (cheap, ~10% of normal) vs cache_creation_input_tokens (a little above normal, since it’s writing the cache).

Poke it. Take a normal turn, then edit an early message, then let it go idle:

A content-addressed prefix cache, not a session store. “Next turn” hits the cache for the whole prefix. “Edit an early message” changes the hash → the entire prefix misses (there’s no fuzzy match). “Wait 5 minutes” evicts it. Numbers are illustrative; the shape is real.

cache: cold

what you paid this turn

vs. re-computing the whole prefix at full price

Press “next turn” to send another message and watch the prefix hit the cache.

This is exactly why a Claude Code session feels fast and cheap on repeated turns: your system prompt, CLAUDE.md, and tool schemas are identical every call, so that big fixed prefix keeps hitting the cache — only the newest turn is fresh. And it’s why touching an early message hurts: change one byte up top and the whole prefix stops matching.

02 Two things people assume — that aren’t there

There is no RAG on Anthropic’s side. No vector database of your repo lives on the server. “Retrieval,” in a coding agent, is entirely the client-side tool loop you saw in the first animation — the model asks for a grep, your machine runs it, the output is pasted back as text:

  You: "fix the bug in auth.py"
          │
          ▼
  Claude Code (local) ── prompt + tool defs ─────▶ Anthropic API
          │                                             │
          │◀──────────── "call Grep('auth')" ───────────┘
          ▼
  Claude Code runs grep LOCALLY on your disk
          │
          ▼
  Claude Code ── grep output as tool_result ─────▶ Anthropic API
          │                                             │
          │◀──────────── next response / more ──────────┘

Every Read, Grep, Glob, Bash in the transcript is your machine executing that command and stuffing the output into the next request as atool_result. The model never has independent access to your filesystem — it only ever “sees” what gets round-tripped back as text. That’s also why huge repos work: nothing is indexed up front; the agent issues targeted greps and reads as it goes, exactly as you would.

And there’s no automatic model router. The model is an explicit field —model: "claude-sonnet-5" — set by the client on every request. Nothing on the server inspects your question and silently swaps you to a cheaper model. The two things that look like routing —Fast Mode and the Advisor pairing — are both explicit settings you choose, not the API deciding for you.

This is one provider. The other three — OpenAI, Gemini, xAI — converged on the same shape but split on who holds state and who runs the tools. Widen the lens in the model-API landscape, which also maps every piece of it to open source.

03 The one caveat — a genuinely stateful surface

where server-side sessions do existAnthropic also offers a managed Agents / Sessions product (a separate, opt-in beta) where the server does persist session state, run the agent loop itself, and host a container for tool execution. That surface really does have server-side session IDs you resume. But it’s a distinct product — plain Claude Code is built on the stateless Messages API described here, resending full context every call, with caching as a performance optimization rather than a memory mechanism.

04 Why an observability company cares

the plasmient thread

Every arrow in that first animation that lands on the API side is GPU work — a forward pass over thousands of tokens of context. Caching is not just a billing trick: a cache_read means the GPUs skipped recomputing attention over that prefix. The gap between a warm cache and a cold one is, quite literally, the difference between silicon doing useful new work and silicon re-deriving what it already knew.

That is the thesis at the API layer. “Utilization” can read high whether the tokens are fresh or cache-reads — but the useful work is wildly different. The wholeFrontier Stack is made of arrows like these, and the job ofPlasmient is to tell you, for each one, how much of that GPU time was real. Start at the operation where the gap is widest: the attention deep-dive.

↗ Full references — go to the source

This is documented behaviour, not reverse-engineering. These are the primary sources — start with the Messages API reference to see the statelessness for yourself.